Software, Continuous Delivery

Sometimes when I meet with technology teams for the first time to discuss their software delivery process, the word ‘release' comes up an awful lot: "We always have production problems after releases" "Problems arise right before we release code, leading to a scramble" "Our releases have too much overhead in terms of testing and operational support" "We don’t have enough release slots so our releases are too big" "We can’t roll back our releases so they are risky" "Release testing is sometimes squeezed at the end of the process" "Releasing change is too beauracratic due to change management" The typical enterprise organisation might be releasing code on a 6 weekly cycle, so it’s quite surprising that their challenges and pain aren’t around the 30+ working days of the cycle where they are working on value producing activity, but instead are mainly derived from the 1 working day per month when they are pushing code out of the door on the final leg of it’s journey. For these teams, the software release seems to become the centrepiece of the IT functions activity:

  • Releases are big events marked months ahead on the calendar;
  • A lot of planning goes into each one, with negotiation and sometimes ‘pulling rank’ in order to decide what features go in;
  • The code is in a broken state through the cycle and not approaching release-ready until a few weeks before the date of reckoning;
  • Huge amounts of release testing occur prior to a release, testing both the new code and lots of old code that hasn’t changed in years;
  • A big team is brought together out of hours to deploy and test the new system;
  • The system is bought down for a long maintenance window;
  • The deployment is carried out by following a long manual run-book;
  • Everyone is nervous and managers are on call at the end of the phone in case anyone goes wrong;
  • As the release takes place, people start getting nervous about progress as there is always half an eye on needing to roll-back the release;
  • The system is opened up and everyone holds their breath as the first users log back in;
  • There are 2 weeks after the releases where we have a collective sigh of relief and nobody does any real work whilst we try to forget the whole thing.
Customers don’t care about software releases, they care about valuable features in their hands."

I view this release centricity as an anti-pattern. Customers don’t care about software releases, they care about valuable features in their hands. Every time someone in the business thinks about or works on a software release they are working on non-value producing stuff. This is a massive source of waste. If we can shift the organisations time and mental energy away from this overhead and back to value producing activity then the effect can be huge.


Breaking The Release Anti-Pattern With Continuous Delivery

The wisdom for getting away from this situation lies in Continuous Delivery. Continuous Delivery is about making software releases boring. It’s about moving from big, risky, heavyweight releases to a world where smaller batches of change leave the building much more frequently than before. It is about reducing the cost of change so that small batches again become economic and the kind of overhead above is stripped away. Moving to Continuous Delivery requires investment. It needs high levels of test and release automation, it has architectural implications, and also needs significant changes to ways of working. The big win however is that we are not wasting significant amounts of our time on the kind of stuff shown in the list above, instead getting people concentrating on truly value producing activities.

The Release Jar

Sometimes people put a 'swear jar' in their place of work. Every time someone says a naughty word, they have to pay a small fee into the jar as a light hearted way of discouraging people from swearing. To support Continuous Delivery, I think it would be interesting exercise to do this so that every time someone mentions or even thinks about the word release, they also have to pay a small fee into the jar. If we did this, we would strive to make activities associated with a release become business as usual, or make much more lightweight so we could avoid the fee.

  • We would automate releases and deployments so they have less mental overhead and can be ran in one fast, consistent step - instead of 127 manual steps taking all weekend
  • We would make our releases more reliable and hardened so they create less risk and noise
  • We would release small batches frequently, so we don’t have all the debate about what is in and what is out of the next release - people can just wait for the next slot
  • Code integration and testing would be automated and made to be continual business as usual activities, not events which happen rarely as we move towards a release date
  • We would simplify branching, and not having these complex merging models which people get themselve into
  • We would be pragmatic about reducing the process overhead associated with a release
  • We would try to avoid all of the effort, disruption and chaos associated with the period leading up to and post releaseThe Release Jar is a light hearted idea, but a mindset where people recognise that release noise and overhead is wasteful, actively try to avoid it, and shift the focus back onto optimising the value creating activities would be a really positive thing for many organisations in my experience
  • Benjamin Wootton

    Co-Founder and CTO

    Benjamin Wootton is the Co-Founder and CTO, EMEA of Contino. He has worked with tens of enterprise organisations on DevOps transformation and is a hands-on DevOps engineer with expertise in cloud and containers.

    More Articles by Benjamin