Madison Ruby 2013 – The Eight-fingered Chef by Jason Garber

I’m a software engineer and co-founder at prompt works in Philadelphia but happy to be in Madison today it’s I was enjoy coming to madison prompted works is a ruby and javascript consultancy in philadelphia the three principles we’ve been working with rails since about two thousand five and one of the things that we do for our clients among other things you know building rails applications and other ruby applications is infrastructure automation how many people know what I mean when I say infrastructure automation see a show of hands yeah a lot of people now how many people have automated infrastructure yeah about the same people that’s great well so maybe you won’t get a lot out of this talk or maybe you will find new ways to apply the stuff that you already know but for those of you don’t know you know many of us in the room or developers often we write web applications and we finish a web application and we don’t hand it to the client as a zip file you know they need it put somewhere then you know maybe you have the web app but it has a database behind it maybe it has a search engine behind it maybe we need some caching or a queue that does background worker processes or maybe we’re not just deploying a single web app but we’re deploying you know something that has service-oriented architecture so it actually has all these working pieces all this is infrastructure behind a web application and when you have infrastructure you need some way of managing it and so of course we are web developers or developers in general we like to automate all the things and so automating your infrastructure is just that gone are the days when you would rack up a server and you take a couple days to install things and then you’d you know perform upgrades periodically and you dutifully back it up in case something happened now of course we have cloud computing resources are virtualized you can spin up 100 servers in an instant you can tear them all down servers can be disposable you can just treat them as a commodity so if we’re treating servers as disposable we sure better not be setting them up by hand because that’ll just be a big waste of time so what we want is for our entire infrastructure to be represented as code so that we can check it into our repository we can fork it branch it merge it all the things we do to code blame you can get blame your your infrastructure find out who did something so we want our infrastructure to be represented as code and chef is one of the many tools that lets you do that and then chef you you declare what you want on your node you don’t describe how to to get there necessarily but you say this is I want to serve and I want to have all these things all these services running and files in place and then you run these recipes over and over and over so that the node converges into the state you want and there are many other tools besides chef I’m talking about Chef this morning but there’s there’s puppet there’s ansible a number of others doctor is looking really interesting and promising I’ll be interested to see how that plays out through this fall so I don’t really care it’s not that as important what you’re using so long as you are automating your infrastructure so that your entire business can be represented in your repository maybe a backup of your data and of course your bare metal servers when your infrastructure is code it’s portable you can change platforms and providers spin up VMs on your laptop you can run tests against them which is wonderful and you save a lot of time and worry now chef is a huge topic it’s really big and complex and I’ve long been intimidated by it but what I want to present this morning is the tiny part of it that you need to know if you just have you know one or two or five servers and you don’t need some centralized management you don’t need a lot of overhead or orchestration

so that’s what I want to present first I’ll go through some terminology and kind of how chef works because you need to know that as background chef recipes describe resources packages you want installed users files all these things you described with this Ruby dsl those recipes are organized into cookbooks that the cooking theme carries you through out so you can have multiple recipes in a cookbook cookbook scan or don’t have to be in roles and then you can apply roles to nodes so this is my database server this is my web server so let’s say we have multiple environments QA CI production and they have different numbers of servers in them nodes and some are web app nodes and some our database nodes the web app nodes they get recipes from the engine X cookbook let’s say and from a ruby cookbook just as an example and in the database role you’d have maybe a postgresql cookbook and and if you’re familiar with the backup gem there’s a cookbook for that and then these get applied to various servers so to manage all this you’re typically going to need a chef server so you have your workstation has your cookbooks on it and it has a tool called knife and so typically you write those cookbooks on your computer and then you use knife to push those cookbooks to the chef server you use knife to tell the nodes to update themselves they pull the cookbooks from the chef server and then they run themselves they converge those nodes and that’s how working with a chef server works and that’s a lot and it’s and it’s really intimidating if especially if you don’t really need all that overhead so thankfully there’s chef solo which does away with chef server and all it does is run the cookbooks on the nodes where they are so you need to get chef so on those nodes and get the cookbooks on those knows so you probably like yeah SC peeing right no automate all the things friend of mine Matt Shafer whom you probably know wrote knife solo and all it does to plug into knife and it pushes the cookbooks to the the nodes and gets them all set up with Chef solo and that makes the process much simpler so if you have a server or two that’s all you really need to reap the benefits of infrastructure automation knife solo chef solo it keeps things pretty pretty simple so last week I was working at one of our clients and they were discussing at length whether they should use chef or puppet and while they were discussing I just went ahead and did what they wanted in chef and it took you know not too long so what they needed was a Jenkins server so let’s go out to hopefully see there we go so you know the first thing you probably do is look in an OPS code community Opscode is the company that wrote chef and open sourced it so we go and have a look at see if there is a cookbook for Jenkins and indeed there is and let’s see if it’s fairly recent februari first 2013 that’s great there’s a link to it on github we can have a look at the readme says it’s works on Ubuntu gives us the attributes attributes are little Flags you can set or their attributes you pass to the recipe that’s well there on the and the recipes can look at those attributes and choose to do things based on those and then it describes the recipes that this cookbook makes available to us we’re going to use the default the server recipe actually there’s a metadata file you can think of this like a gem spec file describes you know the name of the cookbook and what version it’s on and all the things it depends on so that’s great there’s a recipes folder that contains all the recipes and then there’s a folder that has all the default attributes for those recipes and they’re just you know simple all this is just Ruby so you can do anything you can do in in Ruby in chef so this cook will collect looks good

let’s let’s use it switch over to that one alright so the first thing I’m going to do is just make a empty directory CD over to the right spot and check my vagrant version so let me talk a bit about vagrant vagrant is a tool for managing virtual machines on your computer from the command line you’ll need something like virtual box or any other back-end vm provider so you go out you download vagrant and you have it on your command line and there’s a couple plugins that I want to install for vagrant the first one is the vb guests plugin don’t worry too much about what it does VMs maybe need some guest software sometime this just keeps it up to date I find it works best when I have that managing the guest software for me and then I’m also going to do a little plug in for doing snapshots so that can easily just take a snapshot at various points so you know chef recipes are supposed to be idempotent but as you’re developing them you want to jump back to a previous known good state so snapshotting is fantastic okay so now I’m ready to make my directory CD into Madison Jenkins and the first thing I need to do is to initialize my vm with fragrant so vagrant knit and I’ve got a i’m specifying what base box i want to use found of vanilla box on the internet that doesn’t contain some of the stuff that vagrant boxes often have we want this to be as close to the bare metal as possible similar to what you’d get from amazon or rec space or something like that right so i ran vagrant knit that created a vagrant file right there and now i’m going to open up my vagrant file take a look at it this is just an API for specifying how the how the box should come up first thing I’m going to do is give it a hostname Madison Jenkins vagrant okay oops and then I’m going to change there’s port mapping that’s going to happen Jenkins always runs on port 8080 so I’m going to make sure that Oh 8080 is mapped to a DAT my local system yep and then down at the bottom of this vagrant file is some provisioning for chef solo I don’t actually want to use it in this case because we’re going to use knife solo but it’s there it would run the chef’s all this stuff automatically if I were using that all right great so next thing is to bring up the vm so I’m just going to do vagrant up and once that vm is booted it’s doing that vagrant guest additions installer getting that all up to date and now now can just SSH in with vagrant SSH and I’m in the box and it’s just a plain old going to 1204 box what I want to be able to do not have to use the vagrant SSH command so I’m going to look at the SH config provide a hostname here too so SH config if you’re not familiar with it it can specify the hostname the user identity file all that kind of thing now if i append this to dot SH config then all I have to do is do SSH vagrant and I’m in ok and then my next step is going to be adding some gems to this project so I open my gemfile the gems i want the first one I’m going to have is brooke shelf bookshelf is kind of like bundler but for cookbooks the next thing is going to be knife sola which we already talked about and now let’s run bundler of course to generate bundle and then let’s look at what knife so offers us here the commands for knife solo I’m going to run the in it and I’m going to let it know that I want to use brook shelf so now it created a bunch of empty directories and it also created a Brooks

file for us let’s look at that Burke’s file right now it’s just empty but we’re going to add the cookbooks that we want first the apt cookbook also the Jenkins cookbook and that 1 i’m going to pull straight from github they haven’t they haven’t pushed the latest you know i want to use head from from github and not the one from the opscode community site so made our books file and then just like running bundle we run Burke’s install and that pulls down the cookbooks to our local machine and gets the right versions then we can I slowly prepare vagrants and what knife solo prepare does is it copies well it puts chef solo on the machine and that’s that’s it it also generates a node config for us in the nodes directory so I now have a JSON file that tells tells it what what to run all right so and then I’m going to snapshot this because after I’ve prepared it it’s it’s a spot that I want to jump back to again so I quick took a snapshot if I opened up VirtualBox there’d be a snapshot in the list what’s next it’s time to look at that node file i generated so it’s an empty run list so if I were to cook this node right now it wouldn’t do anything so I want to add first the apt recipe and then the Jenkins server recipe tell those to run and so you know the berks file tells it what to what to have in its in its library but then telling which recipes to run is the job of the node file in this case and so now we’re ready to we’re ready to cook so I’m going to knife solo cook this vagrant node and it’s going to copy those cookbooks up to the the node and then it’s going to converge that node basically going to run all the recipes we specified it’s Jenkins so Java it’s got to install the JDK it takes a little while and the Jenkins package wait for it to finish up and it gives you some nice output lets you know what’s going on all right it’s done so now if we hop over to web browser if i go to localhost 8080 we should cross her fingers yep we see Jenkins so I’ve got Jenkins up and running on a virtual machine that was really easy of course we could have just you know SSH into a machine and done apt-get install Jenkins so maybe not that impressive yet but but at least we we have it somewhere and we could just didn’t read me file say run you know knife solo cook and you’ve got another Jenkins node now let’s do something a little bit harder let’s say we have a rails app that we want to deploy to a server so I need a new kitchen for that app Madison sample app let’s say and now see into that directory and same steps as we did before vagrant in it going to use my vanilla box again and work on the gem of the vagrant file put in the host name doesn’t need a host name but yeah hostname Madison sample yeah okay and then I also want to forward the port this time I’m going to use 80 81 Ford port 80 to 80 81 on my local so vagrant file now we run fake it up to bring our virtual machine up it’s mapping port 80 to 80 81 doing all that good stuff once it finishes bundle knits editor gem file now let’s see we want Brooke shelf again we want knife solo again that’s it for now and then of course we bundle and again with knife solo in its work shelf dot it creates our kitchen with our Brooks file editor Brooks file and now we want since we’re using chef solo we need a special cookbook called chef solo

search normally a recipe is able to call out to a chef server and ask about other nodes in the environment so it can find things to talk to you and configure itself we don’t have that so buddies made a cookbook that sort of makes it work fix it enough to get us by then we’re going to use a rails application cookbook but I made a little fork of it so we’re going to use my fork from github occasion okay of course run Burke’s again copy down those cookbooks let’s have a look at the Brooks file lock just like a gem file lock Brooke shelf keeps blocked versions of all the cookbooks so that we can do this over and over and not have undefined results from from different versions and then I’m going to this time instead of working directly with a node file I’m going to copy in a role that I’ve created I’ll show you what it is but just didn’t want to have to type it all out for you so we now run list we’ve got the chefs will search and we also have our rails application recipes I’m also providing a couple of attributes to the to the rails application cookbook specifying the name of the app and specifying the repository it should pull from pretty basic so quit that and now time to I’ve solo prepare and this time I’m going to give it a run list and I’m going to say that it should use the app role so now its bootstrapping chef which means it’s putting chef solo on that virtual machine and then let’s take a look at what what happened now we have a Rolls file and we have a vagrant Jason file from the preparation if we look at that Jason node file all its all the only thing in its run list is to run that app role and the app role you saw it before but that’s what it’s referring to so last thing we should probably do since it’s all prepared and maybe we’ll want to jump back is take a quick snapshot take prepared so all right and then comes the fun bit where we cook favorite so again copying the cookbooks up running them it’s doing everything from setting up our postgres server deploying you know doing a git clone doing a bundle install its pre compiling our assets for us all those things are part of this big cookbook Yeah right now it’s doing the bundle install takes a little bit but when it’s finished we should have a running app on port 80 81 81 and we do so here’s our sample app something I just grabbed from github it’s running on our VM which is fantastic but obviously we don’t just want things running on our VM so let’s put it on a server out on the real internet somewhere first I need to copy over an API key i’m going to use digital oceans so i’m copying my api key over to a local knife file now i need to add the gem for digital ocean that plugs into knife ocean and let’s see course the next step is to bundle that was quick and then then we have to configure this digital ocean plug-in or we have to find out the parameters we want to pass to it so first of all we’re going to list our region’s they just opened up in New York to which is ID for that’s what I want to use digital ocean sighs list I’m cheap I want to go with the five dollars a month one so the 512 Meg will be fine I d66 H key list here’s an sht I’ve already uploaded so it’s going to put that on the the machine for me which lists global this will list out all the images that digital ocean makes available for droplets I’m going to find the boon to one that I want looks like it’s yep the bottom one there ocean

droplet create now I’m going to actually create the server with digital ocean I have to give it a name sample app to location 64 size is 66 904 and I’m giving it a the solo flag so I’m telling it go ahead and create that droplet on digital ocean and when you’re done run knife solo and then I’m going to give it a run list just like I did before to tell it to run the the Apple so now its spitting up that droplet droplet on digital ocean and it’s going to use knife sola to cook just like we did before on a vm but it’s happening out there on the real internet it’s all hands off it’s wonderful of course it’s not always this easy in real life you know lots of hair-pulling that happens but but the great thing is especially when you’re working with vm you just snap it out back and try again so that’s it should be running now let’s go to it on digital shin hey so it’s a public IP it starts with 192 but but it’s a real digitalocean server and it’s live on the Internet that’s great so that’s the quick run through of how to use knife solo and chef solo that’s certainly not everything there is to know about it let’s see and like I say it’s it’s not that easy all the time but but it’s better than configuring by hand and not knowing where where you’re at a few more resources for learning chef learning about knife learn chef com site maintained by Opscode chef conf 2013 youtube videos are all online on YouTube very good resources my buddy Trotter cash in one of the presenters a lot of great information there there’s also a podcast called the food fight show which is a wonderful resource as well to learn about chef so thanks very much for listening and watching you can here’s my email address a handle on twitter alright thanks very much you