Continuous Delivery of Debian packages @debconf

okay I’d like to welcome here bob was giving his talk about continuous delivery of Debian packages so given a big applause please I’d like to start with clarifying the terminology and every one of you might know continuous integration coming from software development continuous deployment is what we understand with as soon as the QA criteria is fine we ship or deployed we don’t wait for any further feedback continuous delivery this is what we are talking about now is you release whenever you decide it’s useful to release so it’s kind of a business decision does it make sense to release right now because some customers are paying us for this or users are waiting for a new release and it makes sense to release it right now so in terms of we can call the continuous integration and in a further extended way and why are we talking about all of this M it’s like what are the benefits we get from continuous delivery in terms of demon project or in terms of what we are talking about here is the cost of a bugfix are getting bigger and bigger in terms of when we are in the pipeline if we are in the in the beginning where we we have requirements design it’s quite cheap to change things and define things but what should it look like and it’s getting much more expensive later on code development accounting operations once we deploy it all the system all the changes are getting much more expensive so what we would like to have is some kind of independence we don’t want to rely on this specific laptop or on this specific machine like nobody knows how to rebuild it from scratch or how to change anything what if we just take it out of operations will anything break we want to be able to to kind of scale not just in terms of one bill too many bills of course this is nice but we would be we would like to be able to also start small and just grow as needed in terms of people maybe it’s a small team packaging team like you have five people what happens if we grow to like 500 people reproduce able we would like to have a way of making sure the package we built we can also rebuild like two years lay down that even reproducible M project might be well known and what’s also important is that we have it predictable like we really want to have metrics to estimate filled x times to fix so if we change anything in our software and the build time increases it would be nice to identify what was responsible is it of infrastructure change is it a packaging change is it like upstream change do we have any further built dependencies so what we would like to know is assuming we would have to fix this and that package how long would it take us to deliver it to our end user or customer and I’d like to talk about a few problems and we had at a company where I’m working for and there was a kind of mess with with golden images to ship custom software tech to customers it was a long build time as soon as you change something you had to rebuild all the image you had to upload the image to the customer for one single small change we also had like builds non reproducible it was unmanaged infrastructure so nobody in New on which machine was it built developers could even build their own package and uploaded it or maybe even a binary and just include it in the in the image also the release process was holding back the ongoing development as soon as we were heading for a new release the code was frozen in the version control nobody could took any further work in master branches and all the release process was holding back actual ongoing work getting more and more customers means that we even had to build even more images and diverting from each other like customers have specific needs or nothing to share maybe even we tried like demon source package uploads to a custom build servers not not so sophisticated as the Debian one but in times of weed don’t even upload any binaries which has tripled to step the package from scratch but the developers still needed to manually built or release packages themselves this means then needs some tools from debian org ubuntu they might have problems with

actually releasing stuff what should go in the changelog what version number do i have to use so this was kind of a big problem it solved some some smallish problems in the in the first M part but not the overall vision we had so what actually do we want to have if you look at the continuous delivery booklet the famous one this is what continual is a continuous the deployment pipeline looks like we have a delivery team which has version control checks in some stuff we have built and unit tests and if they fail we just go back in the cycling and start again once they pass we can get to the next step which is like automated acceptance tests if they still fail we’ll go back in the queue check-in trigger trigger nowaday we have user acceptance test and if they pass we can get to the release so we we transfer us into and some kind of workflow for us what would we like to have the developer should just have something like git commit and get review just push it to the code review system that’s that it nothing nothing further shouldn’t touched even change log at all shouldn’t care about what release re are we actually facing if we want to just go for ongoing development just push to master Jenkins one continuous integration server then verifies what we actually build it relies on demon builds we we automatically get once you do their kit review we get custom pas which can be used for development and testing like I’d want to test is what I’m doing actually working like the developer might not know what that even package looked like in the entry side so it just pushes it get a demon package and can just install it on the environment if Jenkins says no I’m not good we are just going back in there in the queue and say well just fix it get review again and once you have the plus one like I’m good you can integrate the code reviewers like other people just look at their demon packaging stuff or possibly just their the software itself and say does this make sense can we push this yes no once we’re good we just admitted to it master branch or whatever maintainence branch we will look at that later and we have some kind of different needs the internal tooling like we run our own infrastructure we don’t need a release dashboard we can just push it and apply it on our infrastructure once we’re happy with it but we also have the product cycle which is like a given release and you’d even release and we r e then decided to go for a release dashboard where we have all the projects and just say we want to release a new version I don’t care about what’s in the deep in change lock and we will automate this this handling we create according branches we create the according text we create the according change log entries and apply the final build and then all the release workflow like automated testing acceptance testing the QA team can decide whether we are fit for release or not just continuous and then we should be to customers like deviant packages it’s really just F get update F get up trade that’s it so that’s what we actually want to have now how did we get there we have we decided on some principles we just rely on deviant packages for and even repositories for everything no exceptions whatever reading nothing only what’s on a version control met us and you don’t have any chance to pay us anything it with the final product into the final system without having it under version control otherwise it can’t even end up there and automate all the infrastructure handling like we don’t want to touch any systems manually all the configuration management systems that should take place we using puppet and unstable automation we have automated even change lock handling to simplify them the releasing of new package versions like you don’t want to think about what version number do we actually need we know this is a new built or this is a hotfix or this is a minor change and we can all decide on that and so developers don’t need even you wounded all day of course are encouraged to do so but sometimes you have developers working very specific components of software and they aren’t using even or don’t want to use even bunted for whatever reason and we have automated release branch handling so whenever we have a new release at the according new release branch it gets created the same for a new what fix everything is created automatically so once you want to fix an existing release you know you can go at every single project just go into the according branch and everything is there as released we have VMs for testing and development we are using vagrant so

whenever you have a problem and someone says there’s this pack report the steps to reproduce all you have to do is like vagrant apt take the project the product in the according name and choose the according version and everything else is set up automatically so we have automated box builds at least once per day so that this so-called base boxes for vagrant automatically the effort according releases and important part we have PPAs for development so no version control freezes at all you can always push to master of masters should always be good always good to release but what is actually released is in separate branches so we just have fast forward and approach you always have to rebase there’s no no option for merchant you don’t know what you will get it’s just fast forward and the according release branches and some improvements that we made in the in this process is usage of temper FS and eat metadata for just building faster or the see cash so we try to get our builds as fast as possible once the user or developer just pushes kit review the demon package should come out as soon as possible we use dashboards for extractions so people can focus on the actual task they don’t have to look at Jenkins is this set now blue or yellow or red where’s the error whatever and which build parameters will actually need to use to end up in the according release that the dashboard takes care of this it knows which project should go into a release which branches do we have which text do we have so the release manager as well as as that according developers have according front and for the actual needs and very important is the code review system the code river system improves of course code quality but it’s also nice for sharing knowledge amongst people like and you are not yet working in some project but there needs to be someone reviewing the code so you tend to ask is this useful what we are doing there could you maybe be more robust in the commit message to explain the actual situation you’re trying to fix and it helps of course introducing new people because new new developers can just start hacking and they get good feedback from people used to work with this project so that you can actually see the progress in a in a bug fix see what other people are fixing what’s the workflow there so code review isn’t just about the quality of code is also about the quality of a team and what what’s under our system is the so called Jenkins given you and many of you might notice the nice thing about standards is that there are so many of them to choose from which is kind of like why I asked in the beginning and we do you use kalpita or s built and then we have code ensign people de we have D build we have the package build packet at APEC a package build package we have tons of rappers or existing tools and there are like different flavors and this this is good there and and that’s bad and whatever and so one one of the main issues important for me was I don’t want to build another tool I just want to view existing tools together to just be able to replace one component by the other if for whatever reason I’m happy and unhappy with something and its billing on top of Jenkins Jenkins was called hudson before in 2011 it was renamed to Jenkins they have weekly releases where you can just follow current development they have LTS versions which is recommended if you run it in production or for actual usage it’s MIT licensed and nowadays there are more than 1000 plugins available for good and bad purposes like you need to identify which plugins are actually useful and which ones actually help and provide something for you they’re more than 120,000 registered installations as of July and just as a disclaimer it’s written in Java but it’s absolutely not restricted to to it at all you can run whatever kind of project with it it’s like grown on steroids it’s really just an and way of scheduling chops of managing artifacts in such stuff so wyd Jenkins leave in view it’s we restarted to formalize the existing knowledge we know about even packaging provide a framework we can work with provide a common ground to base further work on if we decide to to integrate new stuff it should be built on top of it we wanted to gather feedback from other users what

you might they need and what definitely useful and already happened is like you get contributions for further improving your internal system so it was also a kind of community building so we can talk to each other about problems to other companies or to other dave and developers have and don’t trade new tools or standards really we are just relying on what’s available in the deviant ecosystem and it should be easy to use or even eat non give young folks like their people developing upstream software they would like to provide event packages as soon as they have some kind of working team and directory it’s quite easy to get according pills with Jenkins Stephen blue so what’s behind Jenkins deviant view it’s open source project I’m or so MIT license started in 2011 we have more than 25 contributors to read so far it’s written mainly in shell easy to adjust and extend and mainly through hooks and according a configuration very vers environment variables it’s just relying on see I wanna see I so ever so technically it would be possible to just switch Jenkins to something else but it’s the easiest option and open source CIC are available I’m it uses call builder p bilder as a built environment has out-of-the-box support for get and subversion and I know that their users of bizarre and McCool and whatever but out of the box we support get and subversion and with ready to go scripts it has repository management included with free pre pro which every one of you should know and and the so-called fright which is a very simple tool but seems to be useful for some specific purposes and with plenty of q8 who is included Orrin comes of support like you part slinky and auto package test pep age / critic check and check purchasing so who is using and changing Steve and view we have in a grimmer project we also host d package file in a drama fest tools also all that the camera projects order the k’eremu packages postgres has a pretty sophisticated and pick set up building for all the supported both crest versions for all the all kinds of distributions LLVM compiler Wikimedia so there are plenty of users and we get quite interesting feedback form in terms of what they actually need if you want to test it after the talk and there’s the manual approach where you can just set up anything manually but there’s a automatic setup where just get some script easy to review its trivial it’s just a puppet resize or module which sets up Jenkins Jenkins diamond view and m3 chops for playing around so you get everything what you need it’s set up in like five minutes on faster machines it just depends on your system it’s set up in in some few minutes and this is what you’ll get you get a chanc in set up with with three Jenkins jobs Jenkins even view source Jenkins deviant view binaries and Jenkins even view pew birds but this is what we will talk about now the so-called whatever project you are working on source is generating the Stephen source package for you it’s relying on the version control system so it’s accept it expects that everything is there in version control everything what’s on a version control medicine and it generates the upstream source the event changes if applies the control file it’s actually executing a so-called script generate get or generate svn sep shot this also automates to change log handling so you don’t have to manually write anything to the change log it’s it’s looking at your history thanks for get dch gido very useful important it needs to be run only once per project except if you’re building for multiple distributions something different but out of the box you just needed to build once as usual for deven packaging in the binaries job then we do the actual DBN binary package field we have a script called built and provide package it automates the people lower cal bilder actually set up so usually if you don’t have any special needs it does everything for you automatically so you don’t even have to set up capital or whatever everything will be set up for you and you build once per architecture or distribution whatever you are targeting except for architecture all packages of course the Bluebirds jobs is useful to get automated install upgrade and removal tests it’s optional you don’t have to use it of

but it’s useful since you might forget about it you know you don’t have the according pubert setup available if you are working on a package Jenkees d vaguely automates this and you don’t have to take care of this manually as well we have the repository handling its automatic clearly handling all the repositories without any manual interaction so you don’t have to call any repro command lines or yourself the configuration set up for you by default it’s included in this so-called binaries Jenkins job just 44 to make it easy and once you scale out or have specific needs you might want to just separate it into a specific Jenkins job we by default assume that it’s the call so called Project repos job and you can’t control it then to just built only in the binaries job and provide only then in there in the repos job so you can be very specific what you need you might even want to use D put or whatever add a tool to just upload to a repository then you can just split off the binaries part in the repository but it’s just for having it configurable as needed then we apply for the QA testing Lincoln is automatically executed in the source and in the binaries job auto package test is also executed automatically so once you have a deep in tests a directory in your in your package it automatically invokes auto package test and it looks at the according code policies or for pearl shell code Titan and all the results are available as tap and j unit testing for Jenkins usage like you see in this line of code if you have a problem jenkins then can just provide the according feedback to a failed appealed or should we continue its fine for me if just shell check is is unhappy so all the results are available as according reports then so an example of a build pipeline we use is like you review a pushy to review and run some unit tests available in your patent project for example then if this succeeds you continue to building the source package you run all the binary builds run the pupils checks make sure that the package itself is fine include according reciting the in the repo and then be able to just app get install the package from their repos you’re interested in now managing mainly Jenkins jobs without driving nuts and once you start for every single project to have like five or maybe even 10 Jenkins jobs and you have a product with 50 to 100 even packages and this is quite difficult to manage manually and you might want to change the behavior of all the binaries jobs or of all the source jobs so what what to do and the the openstack project has a very nice tool called Jenkins job builder and it relies on on llaman fires for configuration you just have plain text files describing how your project should look like and we have on on github from the supplies company there’s a kamailio depth jenkins project which has an example of how this could look like they’re plenty of others available as well and it’s very nice to use that way because you don’t have to click anything in Jenkins web interface at all for handling that the jobs you have the possibility to just use it and a version control so we have your repos with all that your Jenkins configs in there and if you apply a change you just committed with the according message of course you can include it in the code review assistive system again like does this change make actually sense for our infrastructure will the result look good included in testing environments etc now during this process I mean this is like I was talking now for 25 minutes but this took us more than than a few years to actually be there where we are and we had quite some lessons on the way developer needs might be quite different from operational Rasputia needs they might want to have a specific package which isn’t available in the distribution yet or they might need a specific version of a package which isn’t available in the according distribution yet of course you should contribute back to the upstream distribution in terms of here of course Vivian and when we said never there might be packages which aren’t distributable or whatever for for some reason but it makes sense to just just push back upstream as as possible diverse people improve the overall

quality it’s it’s it’s interesting to have some common ground of infrastructure for systems but for for code quality it’s really interesting to have diverse people and this includes different distributions as well like not just to think about what even provides outside as might provide interest input from other distributions code review requires good remote working culture and open source Forks are used to remote working so I’m not actually here to to promote this because every one of you might be used to working in this kind of working style but it’s something not not so much used in in in corporate environments that aren’t written by remote working culture external dependencies like we have failures on github or sipping ease down or unreachable or pie pie pie p ruby games puppet labs percona this is a these are all examples we we hit in production usage so what you definitely want to have is local mirrors of every external dependency you have it’s also good because its speed you get a speed-up of your built environment and you have staging options like you before going and shipping this to the customer you decide can i push an update from this specific module or package and only then i will provided in the production mirror configuration management it’s essential to to just have the infrastructure as code if you want to apply any changes to all your qianqian slaves this should just go out through the configuration management whatever you are using if it’s puppet or enter or chef doesn’t matter it but it’s essential that you have some kind of configuration management to ensure consistency also consistent time zones and make all the systems in the same time zone make sure they use some network time protocol so all the systems are you can compare locks from different systems also the catch-22 situation like we ran into this the build scripts are broken but a built infrastructure itself receives the updates for the build infrastructure so we have some kind of recursion problem how do you fix a problem with the underlying infrastructure actually applying those changes is broken also upgrading from from weezy to jesse was everything working but the deployment of the configuration management depends on unit tests which don’t work on chassis yet so in terms of this recursion problem you should definitely have some test infrastructure for for setup for configuration changes so you don’t break production systems and this doesn’t mean just the production system for the customer it’s also the the production system of your own infrastructure then every build of a system might look different from current they’re currently running one even with config management because just because you install the package now doesn’t mean the same result will be in two years if you rebuild the system from scratch so you should also have some testing for the configuration management in place there are plenty of projects like service pack em spectator test server Test Kitchen whatever you prefer but you should definitely have some tests also for your own infrastructure available some tips regular rebuilds of all packages are very good and important because you apply recent policies and and package build infrastructure changes to the packages once you change the underlying build infrastructure the result might be different from whatever you’re building you might have parallel built for your different packages and this is a change in your infrastructure and you have to rebuild the packages and this means what we are doing is like for every single release we rebuild all the packages we never take the package from the previous release all the packages are built with all this current infrastructure if you have to deal with plenty of repositories no matter if it’s get subversion or whatever then mr or my repos nowadays tool is very useful for dealing with large amounts of repositories that the pearl package group has a very good documentation about this on the event vicky and very important integrate your continuous delivery system in your monitoring infrastructure if there’s something broke it should get the same attention as fixing something in customer production it as soon as you can’t build any more packages all the developers are stuck in in development it’s interesting also to to gather some some metrics independent from whatever you use if you’re pushing data into Jenkins you get like build times and and locks and whatever into Jenkins but once

you delete a chop it’s gone also there might be infrastructure changes or clean up for for the for the Jenkins jobs until then you lose all this kind of data so it’s interesting to just provide the meatrix into some independent project or database or whatever and we are using a garret as a code review system and if you don’t like the web interface many don’t know this yet there’s also from the OpenStack community or project itself the gerke command-line tool which provides a command-line interface to Garrett so you can just use it on the command line it has great support for offline so you can actually go to your airplane heck on it review and once you go back online you can just push all the things you did which is absolutely great for working with it and also good is some kind of Jenkins verified job like you whatever mci or CD server you might use but let’s call it a verified shop 2 inch the system actually is working as needed artists slaves there I need and can I’d rig I built of some specific chop or test shop for EM does authentication work do you have the users look there that do the nodes actually look good because once you get get problems in your infrastructure and restarting the CI server and things get out of control and we identified some anti patterns for for a continuous delivery environment as soon as you start with a manual ssh2 some system you might change the configuration you might change the underlying system install additional packages which change existing behavior and provide according debugging options instead this is like keep the coop Eli environment up and running and provided to us to another system or stuff like that but try to avoid manual SSH as soon as tests go ok not ok ok not ok and you don’t know why people will just stop looking looking at them and take them serious and they won’t have any trust and they won’t care at all so once you integrate a new QA tool make sure that it’s working properly before making it mandatory to accept in a pipeline like if we want to have all the pupils jobs to be ok they should at least be once ok everywhere and i also tried to avoid the pulling or cron jobs like at the specific times this is like the dean store run of vivian like you know i have to wait four hours to get my change in instead try to trigger immediate actions once you push something immediately start with it this is like dink Jenkins jobs don’t pull for them or the polling for their the version control changes instead once you know that something is going up to the kids over to the Garrett or code review system whatever just provide according triggers to trigger the according builds I want immediate sent and effect and the manual setup of machine configs I’m I’m not sure if this is an official term but i just recently read it for from markaba that there’s no flake it all they look alike but they are still different and as soon as you start to run installer manually and there’s one single change in it that the result might be just different so really it should be all about automation and what was kind of a problem for us and we have in jenkins given you several wrappers for that purposes is like you have tools with no standardized output so it makes parsing harem if you’re developing a new tool make sure that you can rely on the ultra potent and parse it appropriately and checklists are at a good way to identify that something is going wrong because you might just miss something from the checklist the checklist might be out of date just use automation instead if you want to check for something user champions job hard coding IP addresses hostnames port numbers whatever instead of configurability is is bad thing and if you build the same thing in the in the continuous delivery pipeline once and once again don’t do this it should be will once and then reused for tests for deploying whatever and if you don’t have according notifications developers start to wait and to pull for something instead of just continuing to work on it we have some unresolved problems

actually dependence manager dependency management is is kind of an unsolved problem like if you build a pen on packaged bar and you need another package built before it’s kind of tricky to get this automated so we are researching on that front we have built depends and depends but we have no test depend so we can’t build packages which just have which any decent is in this package but just for testing not for building and not for shipping for the runtime and once you have high frequency and continuous delivery TV reporter stories cause apt to fail quite often because the the mirror is updated and we have had some mismatches and we are seeing this more and more often and I think we have some kind of ideas how to tackle that but I’d like to talk to the according maintenance before pew parts we had some like successful runs even if the package couldn’t be installed so it is the actual solution for buttes was removed the package and everything is fine the actual package will try to to test so just to give you an idea that these are the projects that might be worth a look even every one of you hopefully Jenkins Jenkins even people might help you a vagrant this youth football for all these automated testing as a developer Garrett and Gertie for code review is really nice Jenkins chocolate definitely worth a look everything on our version control automation is really important use dashboards for for for abstraction to not have users get into the details of Jenkins test test test really and built on established workflows and tools like the build package workflow works very well for us and the only bad thing about this is once you’re used to that working it’s really horrible to move outside of such environments and the one of you who are interested in getting deeper into this we have a Jenkins even gloop off on the 21st in room Helsinki I would invite to show up there I think time is over yeah are there Christians hello so I’m under the piano protect my maintainer yeah and so I’ve been exposed to a lot of sea ice from OpenStack world I’ve been using Jenkins to build packages for like two years without Jenkins debian grew the thing is I’m really happy to see that this kind of usage of a CI is spreading and that you are using it too but what I would like to happen is that we use it widely inside the gun to do that unfortunately we have to package Chang Garrett okay so I started to do that drink that cone 5 package 1 travel every invite everyone to join that effort so I don’t know much about Java before that comfort didn’t even know how to maintain something with maven and so obviously I’d need help and I don’t want to have it done all by myself I simply would won’t have the time to do it alone so first like please join join me in doing that effort I I know everybody hates packaging java java in debian but we have to do it that’s the first thing and then so the final goal would be you know already all that we have des right Saudi gate makes the whole of the Debian archive available for everyone to use using it what what nvision would be using Garrett on top of the gate and then the the people that are in the uploaders field would be set as corey reversing getting Garrett and then so the we would have this kind of see I you were walking about like building the package running pew past engine and whatnot and then once once it’s done then the people’s in the lower fields would be able to say vote + 2 pro so that bitter goal and I think it’s really really important that we do that and that we have it available on Alioth or in another machine so the GSA already talked with them and they refuse to use Gary if it’s not packaged in the Vienna and I think there are right to do

to reply this way so please join me in packaging get it do you know about how many dependencies we have left to go to package in Java of oral work the question was how many packages are left in papa 60 Garrett used once before once upon a time used ends to build now we choose is back so before we packaged carried we need to package burke it alone is a lot of work and then once we have package back like it maybe 30 packages then we have to do it for Garrett which is maybe again 30 to 40 packages in Java I’m not sure so because it you will discover as we try to build it so continues so just a short question what are you using for the dashboards and we write our own ones started in different languages actually and just cutting it but nothing highly sophisticated is really just an abstraction to get end so maybe if i should start nowadays from scratch it’s some mixture of go and django which would be actually preferred a gnostic just a quicker mind about the test dependent Oh click on my comment about the test dependencies so we ran into the same problem that it’s very hard to find out which packages test depend on me and we discussed this with the the package maintained as a while ago and we actually have a plan how to implement it just not done yet but we have their agreement we know it’s going to look like and so it’s that’ll help us all for figuring our reverse dependency testing properly without a nice I would like to chat about this later just a technical question so the packages you deliver into production yeah they are all coming out of the continuous integration environment if so how do i add are they scientist then because do we have trusted keys on the infrastructure we have entrusted keys on infrastructure so designing is essentially anonymously so to speak of it once you have access to the review system and you get approved packages they end up in the cap or might end up in a product if everything goes straight forward in the pipeline but no manual signing I mean you are it’s if you want to tag something for whatever reason for example internal tooling or stuff like that you’re encouraged to get tagged with your sign but no manual signing is needed to to bypass the pier so the personal signing is in the versioning and yes the wedding packages you don’t rely on on that end okay thank you any further questions ox ox Linda okay um how much effort do you think would take to write of wanna build front one built like front end for it like say I’m adding a new distribution stretch comes out and I want to rebuild everything for scratch and the system needs to figure out which package is still missing say I don’t want to recompile everything but just admit at the missing bits no idea but they might be worse attract we might join on there yeah sorry for the question but I are still of the opinion that it’s not the white idea to package Jenkins debian glue and debian and it’s on my to-do list for this week actually am yay yeah at that depth that the main concerns I have is once it’s in that even pipeline it should just the diviner back attitude should really just fit and so far i basically had the company vision or from different companies what what’s working for them so i’m just looking at them but it’s definitely on the to-do list i have a pack opening on github issues so there’s time for one more question questions so thank you very much to the crowd and you