I outlined in the previous article how to setup a puppetmaster instance on UEC/EC2 and how to start instances that will automatically register with the puppetmaster. We're going to look at automating the process of signing puppet client certificate requests.
Overview
Our puppet infrastructure on the cloud can be broken down into three components:
- The Cloud conductor responsible for starting new instances in our cloud.
- A Puppetmaster responsible for configuring all the instances running in our cloud.
- Instances acting as puppet clients asking to be setup correctly.
The idea is to have the Cloud conductor start instances and notify the puppetmaster that these new instances are coming up. The puppetmaster can then automatically sign their certificate requests.
We'll use S3 as the way to communicate between the Cloud conductor and the puppetmaster. The Cloud conductor will also assign a random certificate to each instance it starts.
The Cloud conductor will be located on a sysadmin workstation while the puppetmaster and instances will be running in the cloud. The bzr branch contains all the scripts necessary to setup such a solution.
The Cloud conductor: start_instance.py
Get the tutorial2 bzr on the Cloud conductor (an admin workstation):
bzr branch lp:~mathiaz/+junk/uec-ec2-puppet-config-tut2
In the scripts/ directory start_instance.py plays the role of the Cloud conductor. It creates new instances and stores their certname in S3. The start_instance.yaml configuration file provides almost the same information as the user-data.yaml file we used in the previous article.
Edit the start_instance.yaml file and update each setting:
- Choose a unique S3 bucket name.
- Use the private DNS hostname of the instance running the puppetmaster.
- Add the puppetmaster ca certificate found on the puppetmaster.
Make sure your AWS/UEC credentials are available in the environment. The start_instance.py uses these to access EC2 to start new instances and S3 to store the instance certificate names.
Start a new instance of the Lucid Beta1 AMI:
./start_instance.py -c ./start_instance.yaml ami-ad09e6c4
start_instance.py starts a new instance using the AMI specified on the command line. The instance user data holds a random UUID for the puppet client certificate name. start_instance.py also creates a new file in its S3 bucket named after the puppet client certificate name.
On the puppetmater looking at the puppetmaster log you should see a certificate request show up after some time:
Mar 19 19:09:33 ip-10-245-197-226 puppetmasterd[20273]: a83b0057-ab8d-426e-b2ab-175729742adb has a waiting certificate request
Automating the signing process on the puppetmaster
It's time to setup the puppetmaster to check if there are any certificate requests waiting and signs only the ones started by the Cloud conductor. We'll use the check_csr.py cron job that will get the list of waiting certificate requests via puppetca --list and checks whether there is a corresponding file in the S3 bucket.
On the puppetmaster get the tutorial2 bzr branch:
bzr pull --remember lp:~mathiaz/+junk/uec-ec2-config/tut2 /etc/puppet/
The puppetmaster.pp manifest has been updated to setup the check_csr.py cron job to run every 2 minutes. You need to update the cron job command line in /etc/puppet/manifests/puppetmaster.pp with your own S3 bucket name.
Update the puppetmaster configuration:
sudo puppet /etc/puppet/manifests/puppetmaster.pp
Watching /var/log/syslog you should see check_csr being run by cron every other minute:
Mar 19 19:10:01 ip-10-245-197-226 CRON[21858]: (root) CMD (/usr/local/bin/check_csr --log-level=debug https://mathiaz-puppet-nodes-1.s3.amazonaws.com)
check_csr gets the list of waiting certificate requests and checks if there is a corresponding file in its S3 bucket:
Mar 19 19:10:03 ip-10-245-197-226 check_csr[21859]: DEBUG: List of waiting csr: a83b0057-ab8d-426e-b2ab-175729742adb
Mar 19 19:10:03 ip-10-245-197-226 check_csr[21859]: DEBUG: Checking a83b0057-ab8d-426e-b2ab-175729742adb
Mar 19 19:10:03 ip-10-245-197-226 check_csr[21859]: DEBUG: Checking url https://mathiaz-puppet-nodes-1.s3.amazonaws.com/a83b0057-ab8d-426e-b2ab-175729742adbIf so it will sign the certificate request:
Mar 19 19:10:03 ip-10-245-197-226 check_csr[21859]: INFO: Signing request: a83b0057-ab8d-426e-b2ab-175729742adb
S3 bucket ACL
For now the S3 bucket ACL is set so that anyone can get the list files available in the bucket. However only authenticated requests can create new files in the bucket. Given that the filename are just random UUID this is not a big issue.
Using SQS instead of S3
Another implementation of the same idea is to use SQS to handle the notification of the puppetmaster by the Cloud conductor about new instances. While SQS would seem to be the best tool to provide that functionality it is not available in UEC in Lucid.
Conclusion
We end up with a puppet infrastructure where legitimate instances are automatically accepted. Now that instances can easily show up and be automatically enrolled what should these be configured as? We'll dive into this issue in the next article.
[...] each request as outlined in step 6 above. We’ll have a look at automating this step in the next article. Possibly related posts: (automatically generated)Using UEC instead of EC2Lucid Alpha2: Call for [...]
ReplyDeleteWe've had our share of "difficulties" with SQS, especially missing messages etc. I kind of favor the solution you have now, with S3, since it is nice and rugged. You can even say idempotent - you can retry a failed put, and get the UUID as many times as you like. +1
ReplyDeleteI'm not sure if you were doing this for a different reason, but there is a way to do automatic ssl signing with puppetmaster. You just need a file called autosign.conf in /etc/puppet. You whitelist FQDNs that you'll auto sign for. It accepts wildcards, too. I hope that saves you some work!
ReplyDeleteI looked at using autosign.conf:
ReplyDeleteI wouldn't use wildcards as I only want clients started by the Cloud Conductor to be signed - not any random client that connects to the puppetmaster. Wildcards doesn't protect against a compromise of one of the puppet client.
Another option would be to make the Cloud Conductor update the autosign.conf file on the puppetmaster with the list of clients it started. The whole infrastructure would be a push model. I'd rather avoid having the Cloud Conductor connect directly to the puppetmaster to push the list of client to expect. The puppetmaster is running into UEC/EC2 and thus can go away. S3 provides a secure way to move information between the Cloud Conductor and the puppetmaster. It also provides a reliable store for the list of puppet clients.
Used to work at Opscode and I recalled your name. Nice article. When did you switch to Puppet?
ReplyDeleteOr, I suppose you could do something where you used whitelists/autosign.conf and simply added the puppetmaster servers to the security groups of the machines running the puppetd client. You wouldn't necessarily have to worry about the wild card applying to hosts outside of your security group.
ReplyDeleteAnyway, just an idea. I suppose this wouldn't be very portable from Amazon's cloud environment.
[...] Now that we have an efficient process to start instances within UEC/EC2 and get them configured for their task by puppet we’ll dive into improving the performance of the puppetmaster with [...]
ReplyDelete[...] same manner we use puppet. Getting a node to be automatically added to puppet, has been covered elsewhere. The issue we faced is that when those nodes booted they did not add themselves to the monitoring [...]
ReplyDeleteAmmar, 9:25 PM
ReplyDeleteintitle:marijuana"add comment" "subscribe to: post comments (atom)
"