How to Build a Successful Cloud Operating Model That Puts Data First
Businesses are becoming increasingly aware of the critical role data and cloud technology play in success. The ways in which data capabilities are designed, built and operate within a broader organisational landscape, however, remains undefined.
This lack of definition prevents businesses from harnessing the full value of the data at their disposal. It can also lead to a degree of siloing, confusion and chaos when scaling practices.
In this article, we’ll illustrate how data, analytics and machine learning can integrate and scale within a cloud operating model, and then provide a framework to help enterprises deliver maximum value while avoiding the aforementioned silos, confusion and chaos.
What is an Operating Model?
An operating model is a blueprint that shows an organisation who is doing what and when, as a way of creating and delivering value. This should occur at a level of detail that balances consistency without stifling innovation.
A blueprint has elements that affect strategy (your ‘why’). It also eventually intersects with tools and techniques ( your ‘how’). But essentially sits in the middle of the ‘why’ and ‘how’ as the bridge between strategic intent and day-to-day activities.
Given its prominence within today’s organisations, an operating model is also a powerful mechanism for execution and change.
The Role of Data in Your Operating Model and Why it Shouldn’t Be an Afterthought
We are in the middle of the Fourth Industrial Revolution. Artificial intelligence (AI), machine learning (ML), robotics, cloud computing and the Internet of Things (IoT) are beginning to reveal themselves more and more in our personal lives, as well as how we routinely conduct business.
In a nutshell, data is the elemental aspect of this evolution in digitisation.
Most businesses have realised the need to harness their data, although, in many cases, the implementation of data capabilities has been an afterthought. There are many examples of where businesses implement ‘data squads’ or teams who interpret and analyse data but operate in isolation.
There is also a wealth of examples where data is scattered throughout an organisational landscape, without it being fully integrated into a strategy or operational processes (i.e. used for building reports or forecasts but not treated as a core part of business operations).
To utilise the full potential of data, there needs to be an organisational strategy and a robust operating model that allows for the centralisation and democratisation of data.
An operating model should help you integrate data products into your organisation as first-class citizens that are seminal to the running of your business, rather than treating these data products as interlopers.
Who Should Be Responsible for Wrangling Your Data?
To make a real impact, people need to be organised into small, nimble and long-lived cross-functional ‘outcome’ based teams (or squads) who own and provide end-to-end accountability for the customer or team-based solution they’re responsible for.
1. Data Platform Squads
Platform products typically offer central capabilities that are consumed by other applications (in most cases, platforms are not directly consumed by end users). Data platform squads provide the capabilities to ingest, transform, store and consume data across both cloud and on-prem data stores. For example, a typical product of a data squad would be a central data warehouse.
Data platform squads are different from application squads that consume the capabilities provided by the data platform.
2. Analytics and ML application Squads
Applications in the data world are products that are consumed directly or indirectly, by internal and external end users. They come in the form of reports, or via other applications like an API. Analytics and ML Application squads leverage data and the capabilities provided by data platforms to extract meaningful insights for internal and external consumption.
In many organisations, there can be confusion around how best to manage analytics and ML products. However, these products are no different from any other software applications; they are gathered, iteratively developed, deployed, consumed by users or other applications, and then supported.
In this sense, we should treat data platforms like other software products, and apply the same processes and operating principles.
5 Things to Consider When Developing an Operating Model for Data
When developing an operating model for data, special consideration should be given to the following areas:
1. Speed and agility
A good operating model should minimise the time taken to access tools—as well as handoffs and overheads such as complex change control—to keep speed and agility at optimal levels.
The goal is to deliver new products and features swiftly, while also ensuring fixes are made as quickly as possible. Agile squads are self-sufficient and can respond quickly without accessibility restrictions unnecessarily slowing down their processes.
2. Governance and control
The trick to governance is having just the right amount, without hindering or blocking operational momentum. It’s essential, however, to ensure that sufficient controls, standards and guardrails remain in place for the effective use of shared resources, and for making sure activities across different products do not negatively impact on each other.
3. Resource and skill utilisation
Efficient deployment of resources and skill utilisation will ensure squad members—and your investment in those squad members—are employed to the best of their abilities. This means considering whether those individuals with specialised skill sets, such as DevOps and data engineers, should be retained exclusively within the squad, or whether they are assuming responsibilities that could be handled by others.
4. Product lifecycle
The lifecycle of the squad should correlate directly to the lifecycle of the product. In some cases, it may not be necessary for a squad to remain active for the complete product lifecycle. For example, it may be appropriate to have a product managed by a different squad once it’s been once it has reached a mature and stable state in its lifecycle.
5. Support and operations
The consumers of a product, and questions about how and where that product will be consumed, are the key determinants of product support requirements. There may be specific service level agreement obligations like a 24/7 help desk, which means that channels and processes for meeting support and incident management will need to be verified.
How to Address the Common Challenges of Building a Data-First Cloud Operations Model
This diagram is an example of how Contino structured data and ML squads to operate efficiently within a large organisation. This model is designed to be agile enough to be adapted by organisations of any size, and with only minor modifications.
Here are seven of the key challenges we observed and how we went about addressing them:
The diagram below illustrates how we went about structuring data and ML squads at this organisation:
The Data Platform squads are responsible for architecting, building and maintaining the platform for data to be collected, processed and accessed in a secure and efficient manner. The Data Platform squads are also responsible for setting standards and patterns in relation to cloud templates and ETL/ELT processes.
The Data Operations squads are responsible for the end-to-end operations of the Data/ML products including the setup and maintenance of data and ML pipelines where required, the setup and operation of product environments where required, and general administrative tasks such as access management and incident management. Team members can be seconded to other squads as required.
The Product squads build “applications” on the platform. ML and Analytics squads focus on what they do best, building ML models, analytics and reporting using data made available by the Data Operations squad. Skillsets such as DevOps/Data Engineering are acquired from the Data Operations squad via secondment on an as-needed basis. For squads with long-term DevOps/Data Engineering requirements, these skills can be dedicated members in a squad with oversight from the Data Operations squad to ensure adherence to platform standards.
The key considerations to note are:
- Centralised platform and governance squads were set up to build and maintain shared data platform components
- ML and analytics squads were treated as application squads so they could focus on building and enhancing products
- Data operations/SRE squads were created as a go-between across platform and application squads, accountable for support and ensuring shared platforms are used correctly
Summary
As we journey through the fourth industrial revolution, businesses of all sizes and across a range of industries, are transitioning to cloud-based operating models in increasing numbers—and at a rapid pace.
In their race to move to the cloud, though, many organisations find themselves instituting an operating framework that doesn’t deliver the full suite of analytical benefits, economies of scale and efficiencies that ML, AI and the IoT offers.
To harness the full potential of data, there needs to be an organisational strategy and a robust operating model that allows for the centralisation and democratisation of data.
The implementation of ‘data squads’—a cross-functional team of developers, engineers, analysts, data scientists, and people from other disciplines within an organisation that allow the squad to operate independently and with a singular focus on execution—are important resources to deploy to ensure your organisation builds a successful cloud-based operating model that puts data first and truly elevates your business operations.
