AWS, DevOps

Spend enough time helping enterprises to improve their software delivery engines, and you find yourself starting to come across the same problems over and over again.  Every client’s situation is unique, but many individual issues overlap. Over time, you begin to identify and understand the patterns and commonalities. It struck us that you should find a way of systematising the common solutions to these common problems, so that the overlapping elements can be delivered in a more efficient and repeatable manner. With the time saved you can then move up the stack and concentrate on issues that provide more value for your clients and truly differentiate them from their competition.  Two problems fit into this mould. These are problems that we’ve solved for your clients many, many times over:

  • Clients want to leverage Amazon Web Services using a DevOps model (automation, infrastructure as code, federated access, etc.).
  • Clients want a Continuous Delivery pipeline to automate deployments and testing, which will then scale to microservices and container-based deployments.  

Digital or DevOps transformation is much more complicated than this, but the above are the two most common building blocks for companies who are looking to accelerate their software delivery time-to-market. It’s almost table stakes to move to CI/CD in the cloud for companies on this journey.  To minimise the time we spend repeating these core activities - and maximise the time we spend on our client’s more unique issues - we developed a solution to put these problems to bed once-and-for-all and in a highly flexible manner.

Introducing Convoy 

Convoy (previously known as 'Continuum') is our framework for accelerating DevOps and Continuous Delivery on Amazon Web Services. It is a lightweight solution for rapidly and repeatedly building cloud-based environments and CI/CD pipelines in a way that incorporates all best practices for agility, operability, and security.

We’ve come across similar frameworks before, open source, proprietary or by other technical consultancies. However, we always saw lots of room for improvement. They are often limiting and inflexible. They get in the way. Maybe they are monolithic and clunky and in no way 'cloud native'. Developers dislike using them.

We developed Convoy to overcome these issues, applying best-in-class open source tools such as Docker and HashiCorp stacks, together with technical approaches and best practices, in a much more modern, open, flexible and lightweight way than we’ve seen anywhere else.

We believe that Convoy can hugely accelerate any DevOps transformation on the AWS platform.

How Convoy works

Within ten minutes, with minimal configuration and a single command, Convoy will give organisations two primary capabilities:

  • Build segregated environments on demand in AWS, configured for all best practices for agility, operability, and security.
  • Build a CI/CD pipeline of integrated open source tools that enables continuous delivery of applications into these environments.

Screen Shot 2017-01-12 at 13.10.24.png

Let’s go into more detail on both of these. 

1. Environments On Demand In AWS

To provide environments on demand in AWS we use three key automation tools.

HashiCorp Packer: we use Packer to build AMI images for all of the major components in our system using Ansible scripts.

HashiCorp Terraform: we use Terraform as an orchestration tool to define and provision our servers and environments.

Docker: we heavily leverage Docker containers both internally and as a means of deploying applications on top of Convoy.

We combine these tools into a simple pipeline where we can provision infrastructure and environments in a very simple and repeatable way.  

Taken together, this pipeline provides an 'environment on demand capability'.  If teams need new environments to support development or testing, they change a simple configuration file and the environment is created in minutes from source controlled code.  All environments are consistent from development, to test, to production.  When work is finished, the environment can be scaled down or turned off to save money.  

2. Accelerating Continuous Delivery 

Convoy also stands up a CI/CD pipeline, incorporating tools such as Jenkins, Nexus, SonarQube and Cucumber Test Runners. It also provides operational tooling such as ELK, Prometheus, Consul and Vault.  

These tools are very widely deployed and used in industry by DevOps mature teams, so we make it quick to stand up baseline versions again using infrastructure as code and immutable containers. 

Reference Cases For Continuous Delivery Of Microservices

Once Convoy is up and running and self hosted, we then provide a number of sample applications and pipelines which teams can use to learn from, or bootstrap their own projects from.

Our two example applications right now are: 

  • The Tribeca trading application: this is a distributed application based on Node.js that we use to show how financial trading systems can be used for highly reliable CI/CD of a microservice-based application.  
  • An InSpec pipeline to show how governance, controls and security can be built into a CI/CD pipeline to achieve DevSecOps.  

Both applications are deployed using containers on a Docker Swarm cluster that provides scalability, resilience, high-quality deployments and additional security features.

We are currently working on adding more microservice-based applications to demonstrate how teams can use this architectural style.

Design Principles Behind Convoy

The key principles during the development of Convoy were:

Avoid lock in: there should be no lock in to the framework – It is a thin wrapper around best-of-breed open source tools such as Packer, Terraform and Docker.  It is not limiting in any way. We can drop out of it at any time into the underlying tools.

Best-practice but flexible: the framework proposes best-of-breed open source tools, but anything is pluggable and swappable into the framework if you prefer to use something else.

Fully-automated Infrastructure as Code: the entire stack is automated using infrastructure as code, including compute, network, storage, applications and development tooling. This gives us the ability to stand up environments and tear them down on demand.

Immutable infrastructure and containers: we make use of immutable server images and application containers to support consistent environments and application deployments.  

Secure by default: Convoy is highly secure, making use of all platform features. It also incorporates a demo of using infrastructure as code to achieve compliance and governance.

Why use Convoy to accelerate your DevOps journey?

Ultimately, Convoy supports rapid iteration of your full infrastructure and software stack so that your teams can focus on creating software that differentiates your business, rather than configuring infrastructure.

At the same time, the automated, repeatable nature of the solution means that infrastructure and engineering costs are minimised, quality is improved through the consistency of the on-demand environments, and deployment risk is decreased.

Lastly, your exact enterprise security and governance requirements can be baked into the framework - guaranteeing compliance, even at speed.

The Future

We’re very excited by the potential of Convoy.

It will accelerate our clients projects and get them to a highly optimised model for DevOps on Amazon Web Services within days, not months.

It incorporates what we have learnt from tens of thousands of man days in the enterprise, and distills them into an elegant and flexible framework that companies can use to rapidly accelerate their journey to software success.

If you would like to learn more, please get in touch via hello@contino.io.

  • Benjamin Wootton

    Co-Founder and CTO

    Benjamin Wootton is the Co-Founder and CTO, EMEA of Contino. He has worked with tens of enterprise organisations on DevOps transformation and is a hands-on DevOps engineer with expertise in cloud and containers.

    More Articles by Benjamin