Lessons learned from ChefConf 2012

"YOU are actually the best person in the world at solving your problem. There is no one better than you at solving the problems you have in your daily life." Every situation is unique. There's no single system that can be universally applied. But given the right toolset and the right mindset, YOU can handle the biggest problems.

The speaker is Adam Jacob, one of the co-founders of Opscode, and arguably the most talkative and energetic sys admin I've ever run into. Adam describes Opscode and Chef as the result of a year of consulting with 12 different companies, setting up their infrastructure, and keeping the re-usable code for the next project. Even startups don't tend to shift infrastructure that rapidly, so he makes a good point that exposure to 12 different infrastructures in a year can give a good sense of perspective on infrastructure automation.

This is no ordinary motivational speaking. It's about managing thousands of servers with a few commands. It's about putting more responsibility on (and at the same time empowering) developers to see their online creations through to the finish line, rather than just throwing things over the fence and expecting the folks in operations to somehow make it all work. It's infrastructure as code. Primitives over ontologies. Make no mistake, this is deep tech voodoo we're talking about.

So what did he find?

Ontologies don't work. No matter how similar the technologies are among companies, they will all set them up in different ways, sometimes for arbitrary reasons, but sometimes for very important ones. To try and fit all of those into one ordered system would border on the quixotic. So Chef took a very Unix-like approach to the problem. The Chef framework isn't so much a system as a collection of primitives, and for the most part, you can put those primitives together in whatever particular way suits you and your organization best.

Taking system administration one step further, he tells the story of a workshop with a customer where one of the attendees said that he'd automated their entire Windows deployment with Chef in 2 days — all in a single 6000 line Ruby block. Adam managed to boil it down to 80 lines of Chef code after looking at it, but the most important thing, in his opinion, was that initial step to move the infrastructure into code. In other words, the initial monolithic block of code was the proper use of Chef, even if it wasn't the most concise and elegant way the problem could be solved. As Adam puts it, even though he's had a ton of experience with automation, "If you had called me up that morning and been like, 'Come automate this Windows deployment infrastructure in this major company in two days, I would have failed.'"

I had come to ChefConf 2012 hoping to learn how to “properly” use Chef after hacking away at it for several months. The documentation was good, but I was looking for the core philosophy, principles I could use to help guide me away from dead ends and towards the strengths of Chef. Now I was being told to do whatever I wanted.

As wild west as this attitude seems, it turns out to be a very good way to work with Chef and to write better automation code. Once you get your recipes into the system, Chef makes it easy to improve upon them. Focusing on the primitive operations keeps you from wasting time looking for "the right way to install MySQL with Chef" and instead makes you look for "the best way to install a package", "the best way to write a configuration file", "the best way to find a specific type of server in a cluster", etc. These pieces, in turn, are reusable with many other types of infrastructure you may need to automate.

I highly recommend listening to the whole talk if you're working with or plan on doing any work with Chef:

This was Opscode's first ChefConf, and you could tell they were excited about it. I'm still trying to track down the over-the-top faux Samuel L. Jackson video they introduced both Wednesday and Thursday's sessions with, stating that: "The revolution will not be televised — it will be coded in Chef!" While there were a few dry spots for me, the overall conference was well-run and provided a wealth of information that's hard to get just digging through websites (especially when Googling with queries like "chef knife usage" and "best chef recipes" lead more often than not to food) and documentation. Even an all-out power outage in the hotel was corrected with wifi access back on within 10 minutes.

As far as I can tell, Opscode has released all of the talks on their YouTube channel:

Of the sessions I went to, I highly recommend:

  • All of Adam's talks: What can I say? I'm a fan. And he has good taste in music.
  • Ron Vidal's Incident Management: We often talk about putting out fires in the tech industry, but Ron has put out real fires, and offers a very different perspective on, as he puts it, "the bad day business".
  • Fidelity IT's Journey to Cloud and Click2Comput: for anyone in a big organization who might have difficulty introducing something like Chef, this is a great talk about focusing on corporate culture and pushing the envelope.
  • How Opscode Migrated Hosted Chef API to Erlang & Changed Databases Without Breaking: A good reminder about how sometimes hot new technologies aren't the right fit for your needs. Opscode found out that MySQL worked better for them than CouchDB. NoSQL databases can solve a lot of scaling issues by throwing away unnecessary relational DB baggage, but they're not a silver bullet.
  • Last, but not least, Abusing Databags for Fun and Profit: This talk opened my mind to many things that I'd never thought of (and perhaps shouldn't think of!) doing with Chef. It's a great example of the hacker ethic that's alive and well in the Chef community.