Velocity 2010: Fast by default - Thursday

Thursday was the last day of the conference and followed the same format as Wednesday: keynotes in the morning, three parallel tracks in the afternoon.

Creating Cultural Change

John Rauser from Amazon shared a few experiences about creating cultural changes inside and outside organizations.

Here are some key takeaways:

• Try something new


• Seek group identity


• Welcome newcomers


• Be relentless happy

Theses ideas actually reminded me of how the Ubuntu community is been built up.

In the Belly of the Whale Operations at Twitter

John Adams of Twitter presented a few insights on how operations are run at Twitter.

He outlined several principles to keep in mind when building their infrastructure:

• Nothing works the first time. Plan to rebuild everything more than once.


• Deploy faster and more often as less code will change.


• Detect problem as early as possible - to recover fast.


• Disable/enable features in production aka Feature darkmode.

To support these guiding principles he listed some of the tools that are used:

• configuration management done with puppet and svn.


• Reviewboard to review changes made to the infrastructure


• Ganglia to take care of monitoring


• Scribe to collect and aggregate all logs into Hadoop HDFS using LZO compression.


• Murder to deploy their code to all of their systems via bittorrent.


• Google analytics to track errors pages while Whale Watcher to track errors in logs.


• Unicorn to powers their rails stack.

Lightning talks

Thursdays lightning talks covered another round of useful tools in helping optimizing page loads:

• httpwatch: a commercial tool that loads web pages and analyses it


• pagetest


• speedtracer: chrome browser extension that provides an insight on what the browser is doing when a loading a page


• fiddler2

Moving Fast

Robert Johnson of Facebook gave a talk about the culture of moving fast at Facebook. Here are a few short sentence to summarize his points:

• How to scale? Have a team that reacts fast.


• The release cycle: Make changes every day as frequent small changes makes it easier to figure out what went wrong.


• Control and responsibility to one person.

He finished with a few lessons that were learned:

• New code is slow.


• Give developers room to try things.


• Nobody's job is to say no.

Practice of Continuous Deployment

Throughout the conference I heard multiple times the idea of continuous deployment. With continuous integration being pushed on on the developer side, its pendant on the ops side is continuous deployment: tests, build, deploy. Deploy multiple times a day with a good monitoring system to identify quickly when things go wrong. When things go wrong it's easier to identify what changed as the number of changes is rather low. All the big shops have a deployment dashboard to review what went live, when and by whom.

The launchpad team is already following this idea: Launchpad edge has a daily update of the code running against the production database. Releases (with DB schema changes) are conducted on a monthly basis. And Ubuntu is providing something similar as the development version is always available for installation - and releases are cut every 6 months.