Why DevSecOps Is an Auditor’s Best Friend

Auditors: harbingers of regulatory discipline!

When it comes to auditing, heavily-regulated enterprise organizations are stuck between a rock and a hard place.

On the one hand, complying with regulation and compliance can cost a huge amount of time and effort. In the investment banking sector, for example, the cost of complying with regulatory demands means that 72% of profits are put back into compliance. As a result, the average Return on Equity across the industry has fallen from over 20% to over 7%, barely enough to cover the cost of capital [1]. As JPMorgan state: “[the] real issue ... is the dramatic increase in infrastructure costs, driven by regulatory and IT spend. Unless banks can get this under control, they will never achieve the sort of return on equity investors are looking for”.

On the other, not having the proper controls and governance in place can have disastrous business consequences: fines, reputational damage, crippled time-to-market, low-quality products and, ultimately, lower revenues.

Enterprises must deal with the issue, one way or the other. Yet many struggle to properly integrate regulations and governance controls into their operations without incurring huge costs or investing far too much time and limiting the resources available to innovate elsewhere.

Why is this? Well, let me regale you with a tale of fear, love and auditing!

A Tale of Auditing and Flawed Models

In a previous role, I was involved in auditing both internally and externally as a point of contact for Sarbanes-Oxley (SOX) and Payment Card Industry (PCI) audits for a Fortune 1000 in the US.

It was not one of the more glamorous aspects of my job, but at the age of 25 back in 2005, it was a task I took seriously. My job was to regularly go through the typical audit drills of reviewing key control documents, interviewing stakeholders, ensuring proper software policies were in place and regularly monitored the shepherding of code from development through production. At the time, the core focus was on an internally-managed, Oracle-based ERP stored in physical data and network centers.

Most of the key controls and policy documents were stored in either MS Word or Excel documents that were checked in and out of a MS Visual Source Safe versioning repository. Each of these documents were meant to model how the value stream flowed in a controlled fashion. Ideas or changes emerged, they were entered into a system of record, elaborated within that system, designed, developed, tested locally, integrated and then certified. These changes were then finally released to production with a high degree of manual code promotion and with various degrees of test and production data. (At the time, there was a heavy usage of the AIM [Application Implementation Methodology] artifacts).

The purpose of the internal and external audit was to ensure the integrity of customer data, stakeholder confidence and security best practices were being followed. This was the best process we knew of at the time, but it ignored one simple fact: models are flawed by design.

At the time of an audit, we would be given a sample set of changes for which to prove the delivery provenance. I would scramble off to hope and pray these samples went through the ideal flow. I would run to the system of record to see all the proper boxes were ticked, the proper approvals and time stamps were recorded, and that there was a paper trail in another tracking system to prove to the auditors we followed the modeled process. Then, I would work with various DBAs to prove data was being cleansed appropriately of PII and other sensitive records governed by law and/or policy.

This was my process 12 years ago, and I am still see companies following relatively similar processes today. It is a practice that is based on escapable standards and that involves heavy manual research to validate/invalidate outcomes - all of which is extremely wasteful and has a very high opportunity cost.

You deserve better!

What is Auditing?

At its core the audit process is about ensuring security and accountability for those that handle sensitive systems. At a broad view, when we discuss Compliance and Governance, we are trying to gain insights into whether one or more of the following are in line with our given regulatory or governance standards:

1. System Functionality

2. How the Product is Built

3. How the Data is Managed

4. Financial Spend

We have a responsibility to our stakeholders, investors, clients and to the craft of software development to ensure rigor and focus on best practices and quality. We are not, however, bound to follow archaic reporting and auditing practices similar to my 2005 practice mentioned above.

Ever since the authors of the Agile Manifesto codified the principle “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”, we have been continuously iterating and improving our practices. With the emergence of continuous integration, continuous delivery and the growing adoption of a DevOps mindset, we are delivering software at unheard-of cadence. As we deliver this software faster, we need to engage the audit arms into the delivery process and set agreements as to when controls need to be made available. Often, teams optimize the pace locally without accounting for the whole system view that is taught in lean and TPS. Extending the value stream to include regulatory and governance stakeholders is a critical step to ensure the high-delivery pace is not negatively impacted by avoidable waste.

To build this bridge we need to better communicate and educate our auditing partners on what this journey is, and why it is beneficial to them. This is where DevSecOps, a security-focused software delivery methodology is crucial.

What is DevSecOps?

DevSecOps is the aggregation of people, process and technology aligned with vision, leadership and strategy with an overarching focus on security. We have an expressed interest in automation and integration to ensure compliance and ease of flow through our value stream, and to responsibly release software to end users.

Central to DevSecOps is automation and the development of clear processes. Through these, we make many things transparent and immutable. Areas such as logging and monitoring become part of the operating model of value delivery and express when check-ins occur, when builds are compiled, instances are provisioned, tests are run, etc. No longer is there the dependency on manual sifting and pecking. Nor is there a need for the complex chronological sorting of events. All of this is declared in our logs and easily queried and reportable in basic systems.

Given the high degree of insight these logs provide, we are creating windows into the audit train that bring delight to the modern auditor.

People

When we approach organizations that are looking to adopt the DevSecOps mindset, we need to take a specific look at the existing Organizational Structures. As stated in Conway’s Law, systems tend to mimic the structure of the organizations that created them.

"Organizations which design systems are constrained to produce systems which are copies of the communication structures of these organizations." M.E. Conway.

So, the logical source of some architectural, compliance or governance deficiency may realistically lie in the bureaucratic alignment of the organization itself. In contrast, organizations with a DevSecOps mindset favor limited hierarchy and team binding, and embrace safety, trust and empowered teams to deliver value within the system.

To ensure the value is delivered, there needs to be a focus on the skills, recruitment and retention of team members that have DevSecOps mindset and interest. Organizations that embrace this culture, such as Allianz, Amazon, Etsy and Rightmove see team member values and interest as more conducive to competitive advantage than technology. To ensure that the delivery framework and auditing controls are curated over time and technical evolution requires a certain type of team member, and ensuring they are in an environment that promotes purpose, autonomy and mastery (hat tip to Daniel Pink’s Drive) is critical to sustaining the DevSecOps culture for the long term.

The term culture is used frequently today, because often the accepted culture requires tuning. I recently did a piece on my personal blog about culture and organizational change, at its core I focus on the learned behavior of organizations. Transformations require an awareness to the existing cultural inertia within the organization.

It is not by accident I mention people first, the hardest part of a meaningful and sustainable shift to DevSecOps, and delighting auditors, is a base structure where people buy-in to the change and are a major part of the transformation.

Process

With an empowered and embraced workforce, a solid DevSecOps process emerges and continuously improves. Teams establish standards and frameworks to deliver value quickly while auditors are delighted with delivery insights that simplify the auditing process. While the process is emergent, there are some basics seen in nearly all DevSecOps organizations.

One core area is in artifacts and traceability management. Nearly all DevSecOps shops strictly limit root access to adhere to the Principle of Least Privilege (POLP), and to make explicit the traceability of access and change to non-local environments. From an auditing perspective, non-shared credentials is a major benefit. Understanding who (or which person invoked a process) made a change and what associated processes associated a change significantly improves auditing transparency. Having system logging of who changed what and where is the Rosetta stone of change monitoring and review.

Compliance is another area of core process attention for most organizations. In a recent Contino project, we partnered with a global organization in a highly regulated industry to ensure 100% compliance with the CIS benchmark level 1 for a cloud-native AWS application. The application was built with thousands of lines of code and built with a high degree of attention to security standards and encryption. Having compliance standards expressed and shared as a core value of the build team members is a critical part of the DevSecOps team process.

Another core tenet of the modern DevSecOps process is a staunch focus on security incident management. While technology and malfeasance grow on linear paths, it is reasonable to assume there will be security incidents that arise over time. As such, a management plan that allows for active monitoring and alerting with robust logging and review is critical. Most auditors are aware that even the best software teams may fall victim to attempted security hacks, but they are also keen to see how the teams have defensively planned for these attempts with sound management, monitoring and response planning. Having DevOps practices in place, means that security patches that used to take weeks or months to migrate through pre-production environments to production can now be managed in under an hour. With recent breaches like WannaCry, executives, customers and auditors are putting an even stricter eye on system patching and preventative security measure and delivery pipelines. If you have a stake in the security integrity of your system, ask yourself ‘How long would it take us to patch production, with confidence?’. If you have that little knot in your stomach right now, maybe it’s a sign that something needs to change!

Technology

With sound people and process, there is now a place for tools to come in and compliment our DevSecOps stance. Often mistaken for ‘DevOps in a box’, configuration management tends to be at the hub of a lot of teams’ technology stack. While tooling is critical to enable the DevSecOps mindset, it is only effective when built with sound people and process.

A fool with a tool is still a fool

Grady Booch

Modern configuration management/containerization/orchestration tools like Puppet, Ansible, CloudFormation, ElasticBeanstalk, Docker and Kubernetes all are examples of highly configurable tools that can take your process and make it work for the teams building software. They also enable seamless and painless tasks for the auditors due to the workflow and scripting, followed by robust system and release logging. As we partner with our auditing partners, educating them on what these tools are, how they work for you and what they tell us is a critical investment of time that makes future activities much more of a pull-process than the traditional push-process of the past.

By incorporating configuration management and build hooks into quality testing frameworks (and SOLID development practices), organizations can put themselves in a very strong position to enjoy the benefits of Continuous Integration/Continuous Deployment. Commonly, Jenkins is the hub of CI/CD process due to its Open Source pedigree and highly extensible platform. Teams and auditors alike learn to love Jenkins as if it were a member of the family team. Pulling code from the SCM, Jenkins then runs builds, executes unit tests, orchestrates deployments and launches automated regression and integration tests amongst other things. From the perspective of the auditor, seeing this all scripted with explicit outputs and logging means simple review and even their own extendable alerting/monitoring for abnormalities.

Thankfully, the idea of building quality into software is becoming more commonplace. On its heels emerges the idea of building security into your architecture and security as code. Often, security is thought of in the inverted view. You are aware of security when an incident happens. From an investor and customer standpoint, the only expected security lapse is the one that never occurs. There is a fantastic whitepaper by AWS from 2015 that discusses Security-by-Design and the four-phased approach to it in AWS that is worth a read. There are some basic standards that are followed when discussing security as code and when selecting tooling to align with your standards:

· Codify your security standards and make highly available

· Make code analysis part of your build process

· Regular penetration test of your application/infrastructure

· Limit access management policies that respect the Principle of Least Privilege

Thankfully, providers like AWS make a vast array of services and tools available to customers that allow administrators to factor for security as code. The DevSecOps team auditor should learn the interplay of the security tooling, since any negative audit findings in this arena are typically easily managed with an update to the existing security policies.

I was fortunate enough to interview Martin Fowler and Rebecca Wirfs-Brock earlier this year about the impact of agile architecture. In this interview, they help to paint a picture of how modern teams express the intent of code, environments, configuration and data in code using modern practices. When we look at the tooling of the contemporary team, they are built with DevSecOps in mind. Red/Green refactoring indicators, Open Source systems, cloud-native infrastructures and simple integration with containerization and orchestration tools are expected features. As DevSecOps becomes a standard for disruptive and high performing teams, these tools – and the awareness of those that audit them – will continue to grow and foster more rapid production releases with high confidence.

Risk Management

Going back to first principles, having a team of DevSecOps minded people is imperative to building out this mindset-shift. To shepherd this degree of change, leaders need to partner with other leaders who live by a dual delivery and upskilling mandate (delivering a transformation project whilst empowering client teams to own their own transformation going forward) and prove success before promising to scale. It is a risk management position that separates Contino from other consultancies and which embodies the experiment-driven practice that disruptive organizations enjoy today.

As teams see the growth of DevSecOps and the efficiencies in value delivery, there will be a natural shift to many, smaller, releases to production over time. It is when this occurs that leaders and auditors see the single greatest benefit of the new mindset – true system risk management. The smaller, faster and contained a change to production is, the easier and less fear-driven the thought of rollback or wider release becomes. Teams will enjoy unprecedented system confidence, ability to experiment, business teams will be empowered with better data and leaders will be the envy of their competition. All while our auditing friends are embedded in the technology shift with us.

Conclusion

Thinking back to my time as an audit support professional, I see clearly how many organizations could find rapid optimization and efficiencies with the even most basic adoption of DevSecOps. I look at teams that struggle with lengthy deployment and release management overheads with the same desire to help with foundational DevOps practices.

If you have a story to share about how DevSecOps has improved the software audits in your organization, or you yourself are an auditor with whom this resonates, please leave a comment or send a tweet to me at @AgilePHL.

References

[1] Accenture. From Impact to Implementation: Addressing the Key Technology Impacts from Markets in Financial Instruments Directive II