Are configuration management tools still needed in the cloud?

Cloud is the buzz word of the year and with the Ubuntu Enterprise Cloud available in Ubuntu 9.04 everyone will be able to build its own private cloud to experiment. As a cloud base infrastructure provides more flexibility and dynamism in the computing infrastructure it seems that configuration management tools will become more and more important in the future.

What's the purpose of a configuration management tool ?

According to wikipedia:

Configuration management (CM) is a field of management that focuses on establishing and maintaining consistency of a system's or product's performance and its functional and physical attributes with its requirements, design, and operational information throughout its life. For information assurance, CM can be defined as the management of security features and assurances through control of changes made to hardware, software, firmware, documentation, test, test fixtures, and test documentation throughout the life cycle of an information system

While the definition above is quite generic system administrators are using configuration management tools such as puppet or cfengine in order to automate system deployments and making sure that every instance providing a specific service has the same configuration. Another service provided by these tools is to automatically distribute configuration changes to all running systems.

How does this apply to a cloud infrastructure?

The cloud model as implemented by the Ubuntu Enterprise Cloud is based on the golden image principle. Each system is based on a static image. The cloud infrastructure is then used to spawn new instances of a specific image. This is one of the characteristics of such an infrastructure: deploying new systems is easier, faster and cheaper. Potential resources are much larger than before.

However one of the issue with the golden image model is that over time there is a drift between running systems and the golden image. When a configuration update is made to the service the offline golden image also needs to be updated. Moreover a configuration management system is needed to push the changes to running systems.

Let's take the example of a web hosting infrastructure running 20 instances of an apache server. How would a new virtual host be defined?

With a configuration management system a new virtual host is defined in the central repository and the tool deploys the new virtual host definition to all running systems.

Applying the combination of the golden image feature with the ease of deployment provided by a cloud infrastructure would lead to defining the new virtual host in one running system, updating the golden image, spawning 20 new instances and swapping them with the old ones  in the web infrastructure.

It seems strange at first to re-bundle a new image and redeploy all of your servers just for a one-line configuration change. One reason may be that system re-installation has always been seen as a last resort option in a traditional infrastructure. This assumption is no longer true in the cloud with its fast and easy provisioning feature.

What are the advantages of the golden image pattern?

Rolling back a configuration change is much faster as both revisions of the service are running at the same time. System administrator don't need to learn another tool and can just use their standard ways of administrating a single server.

However some issues remain:

How about applying a configuration change to different golden images? A dozen of images still need to be booted and the change made everywhere. We're back to square one.  Configuration management tools have the concept of classes and each system will apply specific configurations according to the defined classes. This is done to avoid redundancy in configuration definition. Having just a set of golden images creates redundant configuration. However the amount of images to change is much smaller than dealing with hundreds of instances.

How about tracking changes between configuration? Most of the configuration management tools suggest to keep the central repository under revision control so that changes made to the environment can easily be tracked. With golden images we're lacking the tools to store multiple versions of golden images and perform image diffs: what is the difference between the running system and its base offline image, between two revisions of the same golden image, between two running systems? Having access to such tools would be very useful to system administrators to help the debugging process.

In conclusion configuration management tools have been used for some time by groups running big infrastructures with lots and lots of systems to manage. The dynamism of the cloud brings the same problems to its users even if they are only using a couple of instances to run their infrastructure. Configuration management tools should probably be considered as an essential tool when moving into the cloud.

March, 12th 2009: The Thursday samba bugs were exterminated

I call for Ubuntu Bug warriors to unite on the 12th day of the month of March of the year of 2009 and march all together to squash bugs related to the samba package. Instructions for first timers will be provided in a wiki page as well as a list of prime targets. Veterans are also encouraged to join and focus on the most complex issues while providing support for the rest of the troops in the #ubuntu-bugs IRC channel on Freenode.

Join us in the battle for improving the robustness of the three daemons smbd, nmbd and winbindd and turn the next Ubuntu Bug day into a victory for all of the Samba users in Ubuntu.

Automate Ubuntu Server iso testing

For each milestone the ubuntu-server isos need to be tested. There are two of them and ten test cases defined. That makes twenty installations to be performed and checked.

The first one is fun and you got to discover all the new options. During the second one you're still amazed by all the shiny new features. The excitement fades away one installation after the other. By the time you've reached the thirteenth it's been four hours and you're getting sick of the blue and red colors. Your mind starts to wander "if I could automate all of this, I could watch the latest episode of [name your favorite TV show here] instead of triggering an epileptic fit due to the red and blue stroboscopic effect of the installer"...

Virtualization, templating and scripting come to the rescue ! Here is an overview of the workflow I've developed and refined to conduct ubuntu-server iso testing at each milestone.

Generate one preseed for each test case

Although each test case has its own preseed file most of the content is identical in all of them. Only the package installation and the partition setup can differ. The mako template engine is used to generate all of the presseed files.

Each test case template inherits from a base template which has all of the content. The base template defines two functions - pkg_install and partition - that will be called in each test case template to define the correct content of the preseed file.

Remaster the iso for each test case

Once the preseed has been created the original server iso is remastered. The preseed is added to the iso and the isolinux configuration file is modified to use it. The installer is also booted with a debconf level of critical. All necessary debconf questions have preseeded answers either from the kernel command line (via the isolinux configuration line) or via the generated preseed file. The goal is to have the installation fully automated without any interaction required.

Create a guest for each test case

The next step is to define a virtual guest for each test case that is set to boot from the remastered iso. A qcow2 file is created to be used as the root partition. A template file is used to create the libvirt configuration file for the guest. It has a custom network address and correct paths to the root disk and remastered iso file.

Automate the whole setup phase

The three operations outlined above are automated for each ubuntu-server isos to be tested. In the end there are twenty guests ready to be started.

The directory looks like this:

intrepid-server-amd64-default/
intrepid-server-amd64-dns-server/
intrepid-server-amd64-lamp/
[...]
intrepid-server-i386-default/
intrepid-server-i386-dns-server/
[...]

Each intrepid-server-* directory has the following structure:

preseed
remaster.iso
vm/libvirt.xml
vm/root.qcow2
vm/root.qcow2.orig


Run the installation phase of each test case

Once the guests are setup they are booted one after the other. However a guest installation requires IO on the host. Once installed the guest vm reboots and then idles. In order to not bring down the server, guests are only booted if the load is below a certain threshold.

Perform the test procedure and report the result

After a couple of hours all of the guests are installed, rebooted and ready to be tested. This part of the process boils down to login via ssh on each guest and follow the test case procedure from the ServerInstall wiki page. The outcome is then reported to the iso testing tracker.

This part is still manual. The next step is to automate it: run the test procedure according to the installation that has been done and report the result to a central location. Once implemented whenever a new ubuntu-server iso is available it can be automatically downloaded, configured, installed and tested. Continuous integration testing is just around the corner !