Using puppet in UEC/EC2: Improving performance with Phusion Passenger

Now that we have an efficient process to start instances within UEC/EC2 and get them configured for their task by puppet we'll dive into improving the performance of the puppetmaster with Phusion Passenger.

Why?

The default configuration used by puppetmasterd is based on webrick which doesn't really scale well. One popular choice to improve puppetmasterd performance is to use mod passenger from the libapache2-mod-passenger package.

Apache2 setup

The configuration is based on the Puppet passenger documentation. It is available from the bzr branch as we'll use puppet to actually configure the instance running puppetmasterd.

The puppet module has been updated to make sure the apache2 and libapache2-mod-passenger packages are installed. It also creates the relevant files and directories required to run puppetmasterd as a rack application.

Passenger and SSL modules are enabled in the apache2 configuration. All of their configuration is done inside a virtual host definition. Note that the SSL options related to certificates and private keys files points directly to /var/lib/puppet/ssl/.

Apache2 is also configured to only listen on the default puppetmaster port by replacing apache2 default ports.conf and disabling the default virtual site.

Finally the configuration of puppetmasterd has been updated so that it can correctly process the certificate clients while being run under passenger.

Note that puppetmasterd needs to be run once in order to be able to generate its ssl configuration. This happens automatically when the puppetmaster package is installed since puppetmasterd is started during the package installation.

Deploying an improved puppetmaster

Log on the puppetmaster instance and update the puppet configuration using the bzr branch:

bzr pull --remember lp:~mathiaz/+junk/uec-ec2-puppet-config-passenger /etc/puppet/

Update the configuration:

sudo puppet --node_terminus=plain /etc/puppet/manifests/puppetmaster.pp

On the Cloud Conductor start a new instance with start_instance.py. If you're starting from scratch remember to update the start_instance.yaml
file with the puppetmaster CA and internal IP:

./start_instance.py -c start_instance.yaml AMI_NUMBER

Following /var/log/syslog on the puppetmaster you should see the new instance requesting a certificate:

Apr 8 00:40:08 ip-10-195-93-129 puppetmasterd[3353]: Starting Puppet server version 0.25.4
Apr 8 00:40:08 ip-10-195-93-129 puppetmasterd[3353]: 7d6b61a7-3772-4c41-a23d-471b417d9c47 has a waiting certificate request

Now that the puppetmasterd process is run by apache2 and mod-passenger you can check in /var/log/apache2/other_vhosts_access.logs.log the http requests made by the puppet client to get its certificate signed:

ip-10-195-93-129.ec2.internal:8140 10.195.94.224 - - [08/Apr/2010:00:40:06 +0000] "GET /production/certificate/7d6b61a7-3772-4c41-a23d-471b417d9c47 HTTP/1.1" 404 2178 "-" "-"
ip-10-195-93-129.ec2.internal:8140 10.195.94.224 - - [08/Apr/2010:00:40:08 +0000] "GET /production/certificate_request/7d6b61a7-3772-4c41-a23d-471b417d9c47 HTTP/1.1" 404 2178 "-" "-"
ip-10-195-93-129.ec2.internal:8140 10.195.94.224 - - [08/Apr/2010:00:40:08 +0000] "PUT /production/certificate_request/7d6b61a7-3772-4c41-a23d-471b417d9c47 HTTP/1.1" 200 2082 "-" "-"
ip-10-195-93-129.ec2.internal:8140 10.195.94.224 - - [08/Apr/2010:00:40:08 +0000] "GET /production/certificate/7d6b61a7-3772-4c41-a23d-471b417d9c47 HTTP/1.1" 404 2178 "-" "-"
ip-10-195-93-129.ec2.internal:8140 10.195.94.224 - - [08/Apr/2010:00:40:08 +0000] "GET /production/certificate/7d6b61a7-3772-4c41-a23d-471b417d9c47 HTTP/1.1" 404 2178 "-" "-"

Once check_csr is run by cron the certificate will be signed and the puppet client is able to retrieve its certificate:

ip-10-195-93-129.ec2.internal:8140 10.195.94.224 - - [08/Apr/2010:00:42:08 +0000] "GET /production/certificate/7d6b61a7-3772-4c41-a23d-471b417d9c47 HTTP/1.1" 200 2962 "-" "-"
ip-10-195-93-129.ec2.internal:8140 10.195.94.224 - - [08/Apr/2010:00:42:08 +0000] "GET /production/certificate_revocation_list/ca HTTP/1.1" 200 2450 "-" "-"

The puppet client ends up requesting its manifest:

ip-10-195-93-129.ec2.internal:8140 10.195.94.224 - - [08/Apr/2010:00:42:09 +0000] "GET /production/catalog/7d6b61a7-3772-4c41-a23d-471b417d9c47?facts_format=b64_zlib_yaml&facts=eNp [....] HTTP/1.1" 200 2354 "-" "-"

Conclusion

I've just outlined how to configure mod passeenger to run puppetmasterd which is a much more efficient setup than using the default webrick server. Most of the configuration is detailed in the files available in the bzr branch.