I remember not so long back (about 8 months), when we use to have project leads looking after all the git repos touched for a given project. They were the only ones who could push to the project repo (bottleneck) and they had to keep the project branches in sync with the master/trunk/head (referred to as master from now on) branches (massive overhead). I’m not really sure why people didn’t deal with this sooner, but that’s history I guess. Ask yourself what happened when that project lead went on holiday (ouch)?
So, 8 months ago I was tasked with being project lead to create a new UK based ISP, and the deadlines were tight and the resource was scarce. I needed to be a full time developer on the project, and therefore didn’t have time for all the processes mentioned above.
Solution? Jenkins!
Why have an expensive developer sat there running mundane tasks, over and over again. Why not trust everyone on the project to push to the project branch? If they break the project branch, then they fix it! If they turn a build red, they are the developer who has to turn it green again. It’s a rather simple way of looking at it, and it isn’t always that easy, but it’s certainly the ideal.
So, what did we need to automate?
Syncing Project Branch with the Master branch
This was the easiest one of them all. So, we kept a bash properties file (we should have used a different format if I am honest), to keep track of all git repos touched (There was about 50 repos in total for my project). We then had Jenkins checkout each repo, check for differences between project and master and then pull the master branch into the project branch and then push back to the git server. This was automated to run every 2 hours. If there was a conflict, Jenkins turned red, emailed the team, jabbered the team and posted in IRC. It was the teams responsibility to fix the conflict. To be honest it actually still falls to the project lead, after all they are the lead, but we shared the burden of fixing conflicts, which freed up my time. We also got very early indicators if we had conflicts we needed to resolve.
Two versions of this script exist for different reasons [1] and [2]. Yes the git server was hard coded (!), so I’ve removed that form version 2.
OK, so we now have the project branch getting updated with master every 2 hours, fab, but now we need to branch the code to perform a release!!
Branching the entire project line and create a new release branch
So, we have a weekly release cycle, and you have to have some paperwork finished by noon on a Tuesday in order to enter into the release cycle. So, every Friday (or a Friday when you would like to release something), you need to branch your project repos, in my case it was aboute 50 repos? Wow, boring! Jenkins loves that kind of stuff, so we had another bash script that took the repos.properties file I mentioned before and take the project branch and create a new release branch. Once this was finished it kicked off…
What’s changed in my project since last release
As mentioned above, we have paperwork to fill in before we can enter into a release cycle, sometimes we would also be the release team (not always). This paperwork was the fluffy questions of why are you making the change, what happens if it fails? How are you rolling it out? How would you roll it back? And then, repo list, file lists and a summary of what’s changed etc and so forth. The problem with a summary is, it’s just that, a summary.. What I summarise is going to be different to what you summarise!! Ideally it is black and white, not subjective to whatever someone feels is pertinent information on a given day.
So I created another script to answer some of those questions. This script takes the repos.properties file (which we all love by now), and performs some git diffing love to produce a file list, repo list and a full git log of what has changed in my project since the last release.
What I love about this script is the sheer volume of information it can produce.. You want to know what has changed? Well go look at the git log and I don’t need to summarise it. It’s all in there as long as people are filling in the git commit messages correctly. Which repos have I touched since last time, boom – it’s all in there.
I love this script, and it took me about 30 minutes to knock up one afternoon [3].
So, now we have a release branch that goes into some release testing. But if we have to bug fix that branch (I know, I know!), I want the same bug fix in my project branch as well. I don’t want to have to wait a week or two before it makes it into product before I can use the “Pull live master script mentioned above”. Also, I don’t want to have to remember to push to the release branch and project branch on release bugs. And if I don’t want to do it, I certainly don’t want to make my team do it. So..
Merge the release branch back into project branch
Does exactly what it says on the tin [4], using the repos.properties file. So we now have the master branch and release branch all trickling back into my project branch, and it’s all done for me by Jenkins. Unless something goes wrong in which case the build fails and we go fix it.
So, let’s assume everything went well and we released our code into the production environment, now we have what is essentially a dead release branch, as it’s now merged into the master branch. So..
Delete dead branches
This script was created by Pete Smith and it essentially takes a branch and iterates over the repos.properties list and removes the branch (if it exists) from the remote repo, so it isn’t cluttering up the objects in the repository. Excellent, we even tidy up after ourselves.
So, there we have it, Jenkins as Project Maintainer, now all we have to do is automate line management and we are set!!!
Without the above scripts, we wouldn’t have been able to hit our deadlines on this project, we would have had huge bottlenecks with me having to merge all the code and keep everything in sync. Another bonus was it enabled us to release every month for 8 months, something which we had not been able to do in the 4 years I’ve worked here.