DevOps Fragility, Antipatterns, and Consequences
Mark Eisenberg and Keith Anderson spotlight app architecture, organizational structure, and testing responsibilities and the importance of how these areas can improve or hinder the effectiveness of DevOps practices.
Overview
DevOps was born from the idea of applying development practices and tooling to the creation and management of IT infrastructure. The specific practice that caught the attention of these IT experts was Agile. They saw the promise of applying iterative development to infrastructure automation in a practice that would come to be known as infrastructure as code.
From that simple idea, DevOps has expanded to a set of practices that are intended to improve the speed of the delivery of value while improving the quality of the software deliverables. DevOps requires a full analysis of the value chain connecting the needs of the business stakeholders to the needs of the operations stakeholders. That analysis yields many blockers to delivering code quickly that does what it is supposed to do. The three we will be discussing here are application architecture, organizational structure, and testing.
Antipatterns
Team Composition (creating a ‘Dev Ops’ Team)
One tenet of DevOps philosophy is to give the entire developer team the shared responsibility for the application through all phases of the application lifecycle, including deployment and operations. This shared responsibility, sometimes called shared pain, is designed to mitigate the issue of a team taking care of one phase, then washing their hands of it, throwing the next task over the wall to the next team. All too often this resulted in confusion, mismatched expectations, build failures or deployment failures that broke the application.
However, repeatedly, enterprises create a ‘dev ops’ team outside of the team of developers. This ‘dev ops’ team is usually responsible for creating pipelines, advising on source code branching strategies, infrastructure, security, and any number of other concerns that used to be under the umbrella of ‘operations’. This completely misses the purpose of Dev Ops philosophy and does nothing to address the original problem. Proper team composition would see team members who were good at deployment pipelines and operations sit side by side with team members who were good at test scripting and writing code. They would all be on the hook for a proper CI/CD (Continuous Integration and Continuous Delivery) pipeline, properly exercised tests, and deploying code that worked as intended.
The same anti-pattern is also common with security and quality assurance. Many enterprises understand that by keeping teams dedicated to these aspects of the development lifecycle outside of the development team, they are breaking the Dev Ops methodology, but they seem uncertain how to fix it.
Teams of software engineers can happily plan sprints, hold daily scrums, estimate, own and deliver work items from a back log, but without proper team composition, there will always be friction that results in anger, confusion, and dysfunction.
Lack of shift-left testing
Prior to DevOps and Agile, the burden of testing an application usually fell to a team specially focused on exercising test cases during a specific phase in the development lifecycle. One tenet of DevOps is about making all the individual loops within the dev lifecycle continuous, in that they are happening constantly during the cycle. Testing is one of the more difficult things to reorganize into a DevOps continuous pattern.
The solution is simple enough to understand but challenging for many teams to implement correctly. Unit tests allow testing to ‘shift-left’ and introduce quality control as soon as the first line of code for a new feature is written. In practice, many organizations do not mandate unit tests or with enough coverage to be effective.
By not implementing unit tests correctly or completely, future changes will eventually introduce regression bugs that never get uncovered prior to a release and can be insidious to find.
There is also a team composition aspect to this. Many organizations maintain separate test teams rather than bring test engineers onto the dev team, making the same mistake as forming a ‘DevOps’ team discussed above, resulting in the same friction and lack of accountability.
Monolithic Architectures
One of the tenets of DevOps is the need to be able to deploy new functionality to production without breaking existing functionality. The best way to do this is for the application architecture to be granular enough to permit portions of the code to be deployed without impacting all the code. To be truly independently deployable, the software components must also be loosely coupled to each other. The interfaces of the components are opaque. The internal state and function of a component must not be visible to any other component. Only the interface is visible. In this way each component can reach production while not impacting any existing component. The importance of this architectural approach cannot be understated when seeking to achieve the full potential of DevOps practices. It is possible to reach the quality goals of DevOps without this approach, but the velocity component will be severely constrained without it.
Conclusion
We have chosen to highlight these three areas of DevOps implementation because they are critical success factors and without their inclusion in enterprise DevOps implementations there will be a significant gap between the goals and the results of the effort. Many great promises are made about what DevOps can deliver to an enterprise IT organization, but without a commitment to these key concepts, expectations should be moderated for the operational benefits that will be achieved. A careful reading of enterprise DevOps case studies will reveal not only the magnitude of the outcomes, but also the magnitude of the effort.
References/Additional Reading
The DevOps Handbook, Second Edition (itrevolution.com)
Light
Dark
3 comments
Why do we have an “DevOps team”?
For the same reason I don’t ask my electrician to do my dental work. Just because they both know how to use a drill doesn’t make them interchangeable.
Writing good software is hard enough already. Asking my developers to also spend an additional 40 hours a week learning the latest fad in deployment scripts is unreasonable.
You’re living in a fairy tale world where one person is expected to do everything.
You then go onto to say that maybe we just have DevOps people mixed in the development teams.
How big are your teams? My typical dev team is only 4 to 6 people. I can’t afford to have half of them doing network admin work. And if I only have 1 or 2 “DevOps” members, who are they going to turn to when they get stuck on a problem?
Then there are task cycle considerations. The flow of work for development doesn’t match that of operations. Not are the types of tasks. So internally these hybrid teams get chopped up anyways.
We put together teams of specialists for a reason. Work and information can flow from one member to another easily. People can support and train each other because their skills overlap.
Your claims about unit testing bring able to solve every problem display a level of magical thinking.
Yes, unit tests certainly help. But that doesn’t eliminate the need for a team that’s been trained specifically for QA. They are going to check for things the developers never even thought of. And they will do the expensive, but oh so vital, manual testing that developers often ignore.
I agree with not having a “DevOps” team. DevOps is “development operations” hence development is involved. It is quite easy (and common) for developers to build solutions that may work fine locally but be very difficult to properly deploy/scale/etc. It is important that all members of the team share the responsibility. It is really no different than assuming that all devs have a basic level of understanding of databases and how to build code to work with them.
At the same time I don’t agree that you need a dedicated person as this can be pretty boring and limiting. Like so many other areas you will likely have a couple of gurus who know the details enough to help out when tasks get complicated but a solid process, well documented processes for how to do standardized stuff and consistency will give everyone on the team a chance to dig in and become familiar enough. We’ve been doing this for about 5 years. I’m still the guru but on the simple stuff of bringing up new apps onto servers with all the infrastructure involved then the dev team does it themselves following the existing guidance. If they get stuck they know who to go to. Isolated DevOps, in my experience, is just as bad as having a dedicated database team. There is a disconnect that no amount of “sharing” will resolve.
I also agree with unit testing but I have found they are just as fragile as your code and way more time consuming. So if you can invest in that a lot you might detect the “simpler” bugs but integration testing is really where the problems show up. The earlier and more automated this is the better. Interfaces, in general, are actually a bad measuring stick for quality code. It is just as bad as using an in-memory database to test your database queries. The reality is that no amount of mock database/interface testing is going to actually verify your code works with the actual implementation until you get to integration testing. EF is notorious about this issue as an example. I cannot count the times somebody used `LastOrDefault` in code and it passed the “memory” testing facade but failed when EF was actually used in integration.