Friday, April 15, 2016

LDAP server setup and client authentication

We recently bought at work a CloudBees Jenkins Enterprise license and I wanted to tie the user accounts to a directory service. I first tried to set up Jenkins authentication via the AWS Directory Service, hoping it will be pretty much like talking to an Active Directory server. That proved to be impossible to set up, at least for me. I also tried to have an LDAP proxy server talking to the AWS Directory Service and have Jenkins authenticate against the LDAP proxy. No dice. I ended up setting up a good old-fashioned LDAP server and managed to get Jenkins working with it. Here are some of my notes.

OpenLDAP server setup


I followed this excellent guide from Digital Ocean. The server was an Ubuntu 14.04 EC2 instance in my case. What follows in terms of the server setup is taken almost verbatim from the DO guide.

Set the hostname



# hostnamectl set-hostname my-ldap-server


Edit /etc/hosts and make sure this entry exists:

LOCAL_IP_ADDRESS my-ldap-server.mycompany.com my-ldap-server

(it makes a difference that the FQDN is the first entry in the line above!)

Make sure the following types of names are returned when you run hostname with different options:


# hostname
my-ldap-server

# hostname -f
my-ldap-server.mycompany.com

# hostname -d
mycompany.com

Install slapd



# apt-get install slapd ldap-utils
# dpkg-reconfigure slapd

(here you specify the LDAP admin password)

Install the SSL Components



# apt-get install gnutls-bin ssl-cert

Create the CA Template


# mkdir /etc/ssl/templates
# vi /etc/ssl/templates/ca_server.conf
# cat /etc/ssl/templates/ca_server.conf
cn = LDAP Server CA
ca
cert_signing_key


Create the LDAP Service Template



# vi /etc/ssl/templates/ldap_server.conf
# cat /etc/ssl/templates/ldap_server.conf
organization = "My Company"
cn = my-ldap-server.mycompany.com
tls_www_server
encryption_key
signing_key
expiration_days = 3650


Create the CA Key and Certificate



# certtool -p --outfile /etc/ssl/private/ca_server.key
# certtool -s --load-privkey /etc/ssl/private/ca_server.key --template /etc/ssl/templates/ca_server.conf --outfile /etc/ssl/certs/ca_server.pem

Create the LDAP Service Key and Certificate



# certtool -p --sec-param high --outfile /etc/ssl/private/ldap_server.key
# certtool -c --load-privkey /etc/ssl/private/ldap_server.key --load-ca-certificate /etc/ssl/certs/ca_server.pem --load-ca-privkey /etc/ssl/private/ca_server.key --template /etc/ssl/templates/ldap_server.conf --outfile /etc/ssl/certs/ldap_server.pem


Give OpenLDAP Access to the LDAP Server Key



# usermod -aG ssl-cert openldap
# chown :ssl-cert /etc/ssl/private/ldap_server.key
# chmod 640 /etc/ssl/private/ldap_server.key


Configure OpenLDAP to Use the Certificate and Keys


IMPORTANT NOTE: in modern versions of slapd, configuring the server is not done via slapd.conf anymore. Instead, you put together ldif files and run LDAP client utilities such as ldapmodify against the local server. The Distinguished Name of the entity you want to modify in terms of configuration is generally dn: cn=config but it can also be the LDAP database dn: olcDatabase={1}hdb,cn=config.

# vi addcerts.ldif
# cat addcerts.ldif
dn: cn=config
changetype: modify
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/ssl/certs/ca_server.pem
-
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/ssl/certs/ldap_server.pem
-
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/ssl/private/ldap_server.key


# ldapmodify -H ldapi:// -Y EXTERNAL -f addcerts.ldif
# service slapd force-reload
# cp /etc/ssl/certs/ca_server.pem /etc/ldap/ca_certs.pem
# vi /etc/ldap/ldap.conf

* set TLS_CACERT to following:
TLS_CACERT /etc/ldap/ca_certs.pem

# ldapwhoami -H ldap:// -x -ZZ
Anonymous


Force Connections to Use TLS


Change olcSecurity attribute to include 'tls=1':

# vi forcetls.ldif
# cat forcetls.ldif
dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcSecurity
olcSecurity: tls=1


# ldapmodify -H ldapi:// -Y EXTERNAL -f forcetls.ldif
# service slapd force-reload
# ldapsearch -H ldap:// -x -b "dc=mycompany,dc=com" -LLL dn
(shouldn’t work)

# ldapsearch -H ldap:// -x -b "dc=mycompany,dc=com" -LLL -Z dn
(should work)


Disallow anonymous bind


Create user binduser to be used for LDAP searches:


# vi binduser.ldif
# cat binduser.ldif
dn: cn=binduser,dc=mycompany,dc=com
objectClass: top
objectClass: account
objectClass: posixAccount
objectClass: shadowAccount
cn: binduser
uid: binduser
uidNumber: 2000
gidNumber: 200
homeDirectory: /home/binduser
loginShell: /bin/bash
gecos: suser
userPassword: {crypt}x
shadowLastChange: -1
shadowMax: -1
shadowWarning: -1


# ldapadd -x -W -D "cn=admin,dc=mycompany,dc=com" -Z -f binduser.ldif
Enter LDAP Password:
adding new entry "cn=binduser,dc=mycompany,dc=com"

Change olcDissalows attribute to include bind_anon:


# vi disallow_anon_bind.ldif
# cat disallow_anon_bind.ldif
dn: cn=config
changetype: modify
add: olcDisallows
olcDisallows: bind_anon


# ldapmodify -H ldapi:// -Y EXTERNAL -ZZ -f disallow_anon_bind.ldif
# service slapd force-reload

Also disable anonymous access to frontend:

# vi disable_anon_frontend.ldif
# cat disable_anon_frontend.ldif
dn: olcDatabase={-1}frontend,cn=config
changetype: modify
add: olcRequires
olcRequires: authc


# ldapmodify -H ldapi:// -Y EXTERNAL -f disable_anon_frontend.ldif
# service slapd force-reload


Create organizational units and users


Create helper scripts:

# cat add_ldap_ldif.sh
#!/bin/bash


LDIF=$1


ldapadd -x -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -Z -f $LDIF


# cat modify_ldap_ldif.sh
#!/bin/bash


LDIF=$1


ldapmodify -x -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -Z -f $LDIF


# cat set_ldap_pass.sh
#!/bin/bash


USER=$1
PASS=$2


ldappasswd -s $PASS -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -x "uid=$USER,ou=users,dc=mycompany,dc=com" -Z

Create ‘mypeople’ organizational unit:


# cat add_ou_mypeople.ldif
dn: ou=mypeople,dc=mycompany,dc=com
objectclass: organizationalunit
ou: users
description: all users

# ./add_ldap_ldif.sh add_ou_mypeople.ldif

Create 'groups' organizational unit:


# cat add_ou_groups.ldif
dn: ou=groups,dc=mycompany,dc=com
objectclass: organizationalunit
ou: groups
description: all groups


# ./add_ldap_ldif.sh add_ou_groups.ldif

Create users (note the shadow attributes set to -1, which means they will be ignored):


# cat add_user_myuser.ldif
dn: uid=myuser,ou=mypeople,dc=mycompany,dc=com
objectClass: top
objectClass: account
objectClass: posixAccount
objectClass: shadowAccount
cn: myuser
uid: myuser
uidNumber: 2001
gidNumber: 201
homeDirectory: /home/myuser
loginShell: /bin/bash
gecos: myuser
userPassword: {crypt}x
shadowLastChange: -1
shadowMax: -1
shadowWarning: -1

# ./add_ldap_ldif.sh add_user_myuser.ldif
# ./set_ldap_pass.sh myuser MYPASS


Enable LDAPS


In /etc/default/slapd set:

SLAPD_SERVICES="ldap:/// ldaps:/// ldapi:///"


Enable debugging


This was a life saver when it came to troubleshooting connection issues from clients such as Jenkins or other Linux boxes. To enable full debug output, set olcLogLevel to -1:

# cat enable_debugging.ldif
dn: cn=config
changetype: modify
add: olcLogLevel
olcLogLevel: -1

# ldapadd -H ldapi:// -Y EXTERNAL -f enable_debugging.ldif
# service slapd force-reload


Configuring Jenkins LDAP authentication


Verify LDAPS connectivity from Jenkins to LDAP server


In my case, the Jenkins server is in the same VPC and subnet as the LDAP server, so I added an /etc/hosts entry on the Jenkins box pointing to the FQDN of the LDAP server so it can hit its internal IP address:

IP_ADDRESS_OF_LDAP_SERVER my-ldap-server.mycompany.com

I verified that port 636 (used by LDAPS) on the LDAP server is reachable from the Jenkins server:

# telnet my-ldap-server.mycompany.com 636
Trying IP_ADDRESS_OF_LDAP_SERVER...
Connected to my-ldap-server.mycompany.com.
Escape character is '^]'.

Set up LDAPS client on Jenkins server (StartTLSdoes not work w/ Jenkins LDAP plugin!)


# apt-get install ldap-utils

IMPORTANT: Copy over /etc/ssl/certs/ca_server.pem from LDAP server as /etc/ldap/ca_certs.pem on Jenkins server and then:

# vi /etc/ldap/ldap.conf
set:
TLS_CACERT /etc/ldap/ca_certs.pem

Add LDAP certificates to Java keystore used by Jenkins


As user jenkins:
$ mkdir .keystore
$ cp /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/security/cacerts .keystore/
(you may need to customize the above line in terms of the path to the cacerts file -- it is the one under your JAVA_HOME)

$ keytool --keystore /var/lib/jenkins/.keystore/cacerts --import --alias my-ldap-server.mycompany.com:636 --file /etc/ldap/ca_certs.pem
Enter keystore password: changeit
Owner: CN=LDAP Server CA
Issuer: CN=LDAP Server CA
Serial number: 570bddb0
Valid from: Mon Apr 11 17:24:00 UTC 2016 until: Tue Apr 11 17:24:00 UTC 2017
Certificate fingerprints:
....
Extensions:
....

Trust this certificate? [no]:  yes
Certificate was added to keystore

In /etc/default/jenkins, set JAVA_ARGS to:
JAVA_ARGS="-Djava.awt.headless=true -Djavax.net.ssl.trustStore=/var/lib/jenkins/.keystore/cacerts -Djavax.net.ssl.trustStorePassword=changeit"  

As root, restart jenkins:

# service jenkins restart

Jenkins settings for LDAP plugin


This took me a while to get right. The trick was to set the rootDN to dc=mycompany, dc=com and the userSearchBase to ou=mypeople (or to whatever name you gave to your users' organizational unit). I also tried to get LDAP groups to work but wasn't very successful.

Here is the LDAP section in /var/lib/jenkins/config.xml:
 <securityRealm class="hudson.security.LDAPSecurityRealm" plugin="ldap@1.11">
   <server>ldaps://my-ldap-server.mycompany.com:636</server>
   <rootDN>dc=mycompany,dc=com</rootDN>
   <inhibitInferRootDN>true</inhibitInferRootDN>
   <userSearchBase>ou=mypeople</userSearchBase>
   <userSearch>uid={0}</userSearch>
<groupSearchBase>ou=groups</groupSearchBase> <groupMembershipStrategy class="jenkins.security.plugins.ldap.FromGroupSearchLDAPGroupMembershipStrategy"> <filter>member={0}</filter> </groupMembershipStrategy>
   <managerDN>cn=binduser,dc=mycompany,dc=com</managerDN>
   <managerPasswordSecret>JGeIGFZwjipl6hJNefTzCwClRcLqYWEUNmnXlC3AOXI=</managerPasswordSecret>
   <disableMailAddressResolver>false</disableMailAddressResolver>
   <displayNameAttributeName>displayname</displayNameAttributeName>
   <mailAddressAttributeName>mail</mailAddressAttributeName>
   <userIdStrategy class="jenkins.model.IdStrategy$CaseInsensitive"/>
   <groupIdStrategy class="jenkins.model.IdStrategy$CaseInsensitive"/>

 </securityRealm>


At this point, I was able to create users on the LDAP server and have them log in to Jenkins. With CloudBees Jenkins Enterprise, I was also able to use the Role-Based Access Control and Folder plugins in order to create project-specific folders and folder-specific groups specifying various roles. For example, a folder MyProjectNumber1 would have a Developers group defined inside it, as well as an Administrators group and a Readers group. These groups would be associated with fine-grained roles that only allow certain Jenkins operations for each group.

I tried to have these groups read by Jenkins from the LDAP server, but was unsuccessful. Instead, I had to populate the folder-specific groups in Jenkins with user names that were at least still defined in LDAP.  So that was half a win. Still waiting to see if I can define the groups in LDAP, but for now this is a workaround that works for me.

Allowing users to change their LDAP password


This was again a seemingly easy task but turned out to be pretty complicated. I set up another small EC2 instance to act as a jumpbox for users who want to change their LDAP password.

The jumpbox is in the same VPC and subnet as the LDAP server, so I added an /etc/hosts entry on the jumpbox pointing to the FQDN of the LDAP server so it can hit its internal IP address:

IP_ADDRESS_OF_LDAP_SERVER my-ldap-server.mycompany.com

I verified that port 636 (used by LDAPS) on the LDAP server is reachable from the jumpbox:

# telnet my-ldap-server.mycompany.com 636
Trying IP_ADDRESS_OF_LDAP_SERVER...
Connected to my-ldap-server.mycompany.com.
Escape character is '^]'.

# apt-get install ldap-utils

IMPORTANT: Copy over /etc/ssl/certs/ca_server.pem from LDAP server as /etc/ldap/ca_certs.pem on the jumpbox and then:

# vi /etc/ldap/ldap.conf
set:
TLS_CACERT /etc/ldap/ca_certs.pem

Next, I followed this LDAP Client Authentication guide from the Ubuntu documentation.

# apt-get install ldap-auth-client nscd

Here I had to answer the setup questions on LDAP server FQDN, admin DN and password, and bind user DN and password. 

# auth-client-config -t nss -p lac_ldap

I edited /etc/auth-client-config/profile.d/ldap-auth-config and set:

[lac_ldap]
nss_passwd=passwd: ldap files
nss_group=group: ldap files
nss_shadow=shadow: ldap files
nss_netgroup=netgroup: nis

I edited /etc/ldap.conf and made sure the following entries were there:

base dc=mycompany,dc=com
uri ldaps://my-ldap-server.mycompany.com
binddn cn=binduser,mycompany,dc=com
bindpw BINDUSERPASS
rootbinddn cn=admin,mycompany,dc=com
port 636
ssl on
tls_cacertfile /etc/ldap/ca_certs.pem
tls_cacertdir /etc/ssl/certs

I allowed password-based ssh logins to the jumpbox by editing /etc/ssh/sshd_config and setting:

PasswordAuthentication yes

# service ssh restart


IMPORTANT: On the LDAP server, I had to allow users to change their own password by adding this ACL:

# cat set_userpassword_acl.ldif

dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcAccess
olcAccess: {0}to attrs=userpassword by dn="cn=admin,dc=mycompany,dc=com" write by self write by anonymous auth by users none

Then:

# ldapmodify -H ldapi:// -Y EXTERNAL -f set_userpassword_acl.ldif


At this point, users were able to log in via ssh to the jumpbox using a pre-set LDAP password, and change their LDAP password by using the regular Unix 'passwd' command.

I am still fine-tuning the LDAP setup on all fronts: LDAP server, LDAP client jumpbox and Jenkis server. The setup I have so far allows me to have a single sign-on account for users to log in to Jenkins. Some of my next steps is to use the same user LDAP accounts  for authentication and access control into MySQL and other services.

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...