I outlined in the previous article how to setup a puppetmaster instance on UEC/EC2 and how to start instances that will automatically register with the puppetmaster. We're going to look at automating the process of signing puppet client certificate requests.
Our puppet infrastructure on the cloud can be broken down into three components:
- The Cloud conductor responsible for starting new instances in our cloud.
- A Puppetmaster responsible for configuring all the instances running in our cloud.
- Instances acting as puppet clients asking to be setup correctly.
The idea is to have the Cloud conductor start instances and notify the puppetmaster that these new instances are coming up. The puppetmaster can then automatically sign their certificate requests.
We'll use S3 as the way to communicate between the Cloud conductor and the puppetmaster. The Cloud conductor will also assign a random certificate to each instance it starts.
The Cloud conductor will be located on a sysadmin workstation while the puppetmaster and instances will be running in the cloud. The bzr branch contains all the scripts necessary to setup such a solution.
The Cloud conductor: start_instance.py
Get the tutorial2 bzr on the Cloud conductor (an admin workstation):
bzr branch lp:~mathiaz/+junk/uec-ec2-puppet-config-tut2
In the scripts/ directory start_instance.py plays the role of the Cloud conductor. It creates new instances and stores their certname in S3. The start_instance.yaml configuration file provides almost the same information as the user-data.yaml file we used in the previous article.
Edit the start_instance.yaml file and update each setting:
- Choose a unique S3 bucket name.
- Use the private DNS hostname of the instance running the puppetmaster.
- Add the puppetmaster ca certificate found on the puppetmaster.
Make sure your AWS/UEC credentials are available in the environment. The start_instance.py uses these to access EC2 to start new instances and S3 to store the instance certificate names.
Start a new instance of the Lucid Beta1 AMI:
./start_instance.py -c ./start_instance.yaml ami-ad09e6c4
start_instance.py starts a new instance using the AMI specified on the command line. The instance user data holds a random UUID for the puppet client certificate name. start_instance.py also creates a new file in its S3 bucket named after the puppet client certificate name.
On the puppetmater looking at the puppetmaster log you should see a certificate request show up after some time:
Mar 19 19:09:33 ip-10-245-197-226 puppetmasterd: a83b0057-ab8d-426e-b2ab-175729742adb has a waiting certificate request
Automating the signing process on the puppetmaster
It's time to setup the puppetmaster to check if there are any certificate requests waiting and signs only the ones started by the Cloud conductor. We'll use the check_csr.py cron job that will get the list of waiting certificate requests via puppetca --list and checks whether there is a corresponding file in the S3 bucket.
On the puppetmaster get the tutorial2 bzr branch:
bzr pull --remember lp:~mathiaz/+junk/uec-ec2-config/tut2 /etc/puppet/
The puppetmaster.pp manifest has been updated to setup the check_csr.py cron job to run every 2 minutes. You need to update the cron job command line in /etc/puppet/manifests/puppetmaster.pp with your own S3 bucket name.
Update the puppetmaster configuration:
sudo puppet /etc/puppet/manifests/puppetmaster.pp
Watching /var/log/syslog you should see check_csr being run by cron every other minute:
Mar 19 19:10:01 ip-10-245-197-226 CRON: (root) CMD (/usr/local/bin/check_csr --log-level=debug https://mathiaz-puppet-nodes-1.s3.amazonaws.com)
check_csr gets the list of waiting certificate requests and checks if there is a corresponding file in its S3 bucket:
Mar 19 19:10:03 ip-10-245-197-226 check_csr: DEBUG: List of waiting csr: a83b0057-ab8d-426e-b2ab-175729742adb
Mar 19 19:10:03 ip-10-245-197-226 check_csr: DEBUG: Checking a83b0057-ab8d-426e-b2ab-175729742adb
Mar 19 19:10:03 ip-10-245-197-226 check_csr: DEBUG: Checking url https://mathiaz-puppet-nodes-1.s3.amazonaws.com/a83b0057-ab8d-426e-b2ab-175729742adb
If so it will sign the certificate request:
Mar 19 19:10:03 ip-10-245-197-226 check_csr: INFO: Signing request: a83b0057-ab8d-426e-b2ab-175729742adb
S3 bucket ACL
For now the S3 bucket ACL is set so that anyone can get the list files available in the bucket. However only authenticated requests can create new files in the bucket. Given that the filename are just random UUID this is not a big issue.
Using SQS instead of S3
Another implementation of the same idea is to use SQS to handle the notification of the puppetmaster by the Cloud conductor about new instances. While SQS would seem to be the best tool to provide that functionality it is not available in UEC in Lucid.
We end up with a puppet infrastructure where legitimate instances are automatically accepted. Now that instances can easily show up and be automatically enrolled what should these be configured as? We'll dive into this issue in the next article.