Wednesday, April 7, 2010

Using puppet in UEC/EC2: Node classification

In a previous article I discussed how to set up an automated registration process for puppet instances. We'll now have a look at how we can tell these instances what they should be doing.

Going back to the overall architecture the Cloud conductor is the component responsible for starting new instances. Of all the three components it's him that has the most knowledge about what an instance should be: it is the one responsible for starting a new instance after all.

Using S3 to store node definitions

We'll use the puppet external node feature to connect the Cloud conductor with the puppetmaster. The external node script - will be responsible for telling which classes each instance is supposed to have. Whenever a puppet client connects to the master the script is called with the certificate name. It is responsible for providing a description of the classes, environments and parameters for the client on its standard output in a yaml format.

Given that the Cloud conductor creates a file with the certificate name for each instance it spawns we'll extend the script to store the node classification in the content of the file created in the S3 bucket.

You may have noticed that instances started by don't have an ssh public key associated with them. So we're going to create a login-allowed class that will install the authorized key for the ubuntu user.

Setup the puppetmaster to use the node classifier

We'll use the Ubuntu Lucid Beta2 image as the base image on which to build our Puppet infrastructure.

Start an instance of the Lucid Beta2 AMI using an ssh key. Once it's running write down its public and private DNS addresses. The public DNS address will be used to setup the puppetmaster via ssh. The private DNS address will be used as the puppetmaster hostname given out to puppet clients.

Log on the started instance via ssh to install and setup the puppet master:

  1. Update apt files:

    sudo apt-get update

  2. Install the puppet and bzr packages:

    sudo apt-get install puppet bzr

  3. Change the ownership of the puppet directory so that the ubuntu user can directly edit the puppet configuration files:

    sudo chown -R ubuntu:ubuntu /etc/puppet/

  4. On the puppetmaster check out the tutorial3 bzr branch:

    bzr branch --use-existing-dir lp:~mathiaz/+junk/uec-ec2-puppet-config-tut3 /etc/puppet/

    You'll get a conflict for the puppet.conf file. You can ignore the conflict as the puppet.conf file from the branch is the one that supports an external node classifier:
    bzr resolve /etc/puppet/puppet.conf

Edit the node classifier script scripts/ to set the correct location of your S3 bucket.

Note that the script is set to return 1 if the certificate name doesn't have a corresponding file in the S3 bucket. You may want to change the return code to 0 if you want to use the normal nodes definition. See the puppet external node documentation for more information.

The puppetmaster configuration in puppet.conf has been updated to use the external node script.

There is also the login-allowed class defined in the manifests/site.pp file. It sets the authorized key file for the ubuntu user.

On the puppetmaster edit manifests/site.pp to update the public key with your EC2 public key. You can get the public key from ~ubuntu/.ssh/authorized_key on the puppetmaster.

To bootstrap the new puppetmaster configuration run the puppet client:
sudo puppet --node_terminus=plain /etc/puppet/manifests/puppetmaster.pp

Note that you'll have to set the node_terminus to plain to avoid calling the node classifier script when configuring the puppetmaster itself. Otherwise the puppet run would fail since the puppetmaster certificate name (which defaults the to fqdn of the instance) doesn't have a corresponding file in the S3 bucket.

We have now our puppetmaster configured to look up the node classification for each puppet client.

Update to provide a node definition

It's time to update the Cloud conductor to provide the relevant node classification information whenever it starts a new instance.

Update the bzr branch on the Cloud conductor system:
bzr pull --remember lp:~mathiaz/uec-puppet-config-tut3

The script has been updated to write the node classification information when it creates the instance file in the S3 bucket. That information is actually set in the start_instance.yaml file under the node key. All of the node classification information expected by the puppetmaster from the external node classifier script is set under the node key in start_instance.yaml. See the puppet external node documentation for more information on the information that can be provided by the external node script.

Review the start_instance.yaml file to make sure the S3 bucket name, the puppetmaster server IP and CA certificate are still valid for your own setup.

Start an instance:
./ -c start_instance.yaml AMI_NUMBER

Following /var/log/syslog you should see something similar to this:
Apr 7 19:15:37 domU-12-31-39-07-D6-52 puppetmasterd[1644]: 77ad2a3c-5d52-4ca7-9fea-b99b767b09d0 has a waiting certificate request

The instance has booted and registered with the puppetmaster.
Apr 7 19:16:01 domU-12-31-39-07-D6-52 CRON[2188]: (root) CMD (/usr/local/bin/check_csr --log-level=debug
Apr 7 19:16:02 domU-12-31-39-07-D6-52 check_csr[2189]: DEBUG: List of waiting csr: 77ad2a3c-5d52-4ca7-9fea-b99b767b09d0
Apr 7 19:16:02 domU-12-31-39-07-D6-52 check_csr[2189]: DEBUG: Checking 77ad2a3c-5d52-4ca7-9fea-b99b767b09d0
Apr 7 19:16:02 domU-12-31-39-07-D6-52 check_csr[2189]: DEBUG: Checking url
Apr 7 19:16:03 domU-12-31-39-07-D6-52 check_csr[2189]: INFO: Signing request: 77ad2a3c-5d52-4ca7-9fea-b99b767b09d0

The puppetmaster checked if the client request is expected and signs it.
Apr 7 19:17:39 domU-12-31-39-07-D6-52 node_classifier[2240]: DEBUG: Checking url
Apr 7 19:17:39 domU-12-31-39-07-D6-52 node_classifier[2240]: INFO: Getting node configuration: 77ad2a3c-5d52-4ca7-9fea-b99b767b09d0
Apr 7 19:17:39 domU-12-31-39-07-D6-52 node_classifier[2240]: DEBUG: Node configuration (77ad2a3c-5d52-4ca7-9fea-b99b767b09d0): classes: [login-allowed]
Apr 7 19:17:39 domU-12-31-39-07-D6-52 puppetmasterd[1644]: Compiled catalog for 77ad2a3c-5d52-4ca7-9fea-b99b767b09d0 in 0.01 seconds

The puppetmaster compiled a manifest for the client according to the information provided by the node classifier script.

Make sure that the instance that has been started doesn't have any ssh key associated with it:

Make a note of the instance ID and its public DNS name.

Login into the instance:

  1. Run euca-get-console-output instance_ID to get the ssh fingerprint. You may need to scroll back to get the fingerprints.

  2. Login into the instances using your EC2 public key:

    ssh -i ~/.ssh/ec2_key ubuntu@public_dns


The script is currently very simple and should be considered as a proof of concept.

Storing the node classification information into an S3 bucket makes it also easy to edit the content of the file. It also provides an easy way to get a list of the nodes that have been started by the Cloud Conductor as well as their classification.

If you look at the script you'll notice that the ACL on the S3 bucket is 'public-read'. That means anyone can read the list of your nodes as well as the list of classes and other node classification information for each of them. You may wanna use S3 private url instead.

We now have a puppet infrastructure where instances are started by a Cloud conductor in order to achieve a specific task. These instances automatically connect to the puppetmaster to get configured automatically for the task they've been created for. All of the instances configuration is stored in a reliable and scalable system: S3.

With instances being created on demand our puppet infrastructure can grow quickly. The puppetmaster can easily be responsible for managing hundreds of instances. Next we'll have a look at how improving the performance of the puppetmaster.


  1. This is a great new feature, and an excellent write-up. Well done!

  2. Hi, thanks for this series, it's good to see someone else solving the CM/cloud issue.

    This post raises a couple of questions - I'm not quite clear what the external node layout does to help in this scenario.
    Can you explain how editing start_instance.yaml is better than just editing nodes.pp on the puppetmaster?

  3. Good point. The Cloud Conductor is a very simple script for now. Down the line I'd like to investigate how the Cloud Conductor could start new instances automatically in order to leverage the dynamism of Cloud computing.

    Once instances can be automatically started maintaining nodes.pp on the puppetmaster doesn't work well. This is where an external node classifier is really useful.

  4. [...] Now that we have an efficient process to start instances within UEC/EC2 and get them configured for their task by puppet we’ll dive into improving the performance of the puppetmaster with Phusion [...]

  5. These provided information was really so nice,thanks for giving that post and the more skills to develop after refer that post. Your articles really impressed for me,because of all information so nice.

    SAP training in Chennai