DevOps Requires Metrics

Metrics have always been a central theme of DevOps.  By measuring how a team is working, you can track how they are improving, whilst breaking down the walls between different groups within the organisation through transparency.

Example metrics we might want to surface and track as part of a DevOps transformation include:

  • Cycle time for an idea to go from concept to production implementation
  • Number of test failures in the test environments
  • Number of failed deployments
  • Number of operational incidents
  • Application performance 
  • Revenue realised through the application

Metrics such as this together paint a picture of how effectively a team are delivering software, and delivering value to their business.  

From Metrics to ROI

Companies don’t just invest in DevOps for the fun of it - they want to know how these changes map back to their revenue & costs.  Without this view, its’s very unlikely you would ever get a DevOps transformation off the ground.  

In our work supporting Enterprise DevOps transformations at Contino, we therefore had to get very good at measuring and tracking metrics defined above, and then mapping them back to business value.  

Metrics Add Science to DevOps

Over time, we have ended up tracking an increasing number of metrics.  

Our first DevOps maturity assessment developed in 2015 consisted of around 30 maturity points that we assessed and measured.  Over time as we got deeper into DevOps transformation, we gradually built this into a database of over 200 finely grained maturity indicators.  

As the dataset grew, we started using tools such as Survey Monkey, Google Docs and Excel spreadsheets to request information and manage the results, but this was starting to become unwieldy and repetitive.  So we decided to build our own lightweight tool, and open it up to the community with a free version at DevOpsMaturity.com.

Introducing DevOpsMaturity.com

DevOpsMaturity.com is aimed at organisations who would like to assess the DevOps practices in either one application, or a portfolio of applications across their business.

For each application that you survey, a nominated user or the group get together to assess the practices in use in that area. The respondents complete a number of different online assessments and rate themselves on a 1-10 scale in lots of different areas.  The system is designed to be lightweight, collaborative and engaging rather than a burden.

The data is then aggregated to be displayed in a way which shows particular areas of strengths or weaknesses, inspiring the team where to improve in order to raise their score.

If you are looking at a portfolio view, you can surface data which shows the highest and lowest scores in your application portfolio, and the specific areas which you may benefit by focussing on - e.g. test automation or deployment automation. 

Why Do This?

There are significant benefits to moving to a self assessment and data driven approach to IT transformation:

You can't improve what you can't measure!  As part of their digital transformation, many companies are looking to improve how they deliver software.  To improve, we first need to understand where we are, and what we need to do to 'level up'.  These kind of maturity assessments are a great way to kick that off.  

Gamify performance improvements!  By adopting a tool like this, your teams will be able to bring continuous improvement into their day to day workflows.  They can assess during retrospectives, and then bite off one or two capability improvements on each sprint.  

Focus your investment!  By surfacing this data across a portfolio of applications, you will know where to invest to help the widest number of people or teams,.  For instance, our early beta users have flagged areas such as performance testing as their top challenge within the software development lifecycle.

Try It Now

DevOpsMaturity.com is free to try and available now.  We are currently in beta, so please give it a try and let us know your feedback. 

Private Version

Finally, we are also able to create instances of DevOpsMaturity.com behind the firewall or in a private cloud hosted instance.  We can also customise the question sets for your particular priorities.  Please reach out if this is something you would potentially like to execute.  

  • Benjamin Wootton

    Co-Founder and CTO

    Benjamin Wootton is the Co-Founder and CTO, EMEA of Contino. He has worked with tens of enterprise organisations on DevOps transformation and is a hands-on DevOps engineer with expertise in cloud and containers.

    More Articles by Benjamin