This year's report from DevOps Research and Assessment (DORA) starts with The 2019 Accelerate State of DevOps: Elite performance, productivity, and scaling, which links to more detailed information.
DORA provides, in my opinion, the best researched information on DevOps available. Making implementation decisions based on their insights is a safe bet.
The report provides a comprehensive view of the DevOps industry.... With six years of research and data from more than 31,000 professionals worldwide, the 2019 Accelerate State of DevOps Report is the largest and longest-running research of its kind.
My notes, below, are primarily quotes from the reports, even when I haven't explicitly quoted the text. Spelling/grammar mistakes are mine, and I'm sure I'll hear about them from my wife.
You really should download and read the report. I haven't captured everything here, just what caught my attention.
- Highlights from the summary
- The Quick Assessment
- The Full Report
- More Info
Highlights from the summary
From 2018 to 2019, "elite" performers rose from 7% to 20% of all organizations.
I've called out the above because it dramatically "shows that excellence is possible—those that execute on key capabilities see the benefits."
- "For the first time, industry matters" Retail performed better in speed and stability. But that doesn't impact the evidence that all organizations can benefit from DevOps.
- "[T]he highest DevOps performers (that is, the high and elite performers), focus on structural solutions that build community"
- Low performers use more proprietary software, high and elite performers use more open source software. Reason? cost.
- The detailed report has two research models: performance and productivity.
This year’s report revalidates important findings for the sixth year in a row: First, that it’s possible to optimize for stability without sacrificing speed. Second, DevOps delivers value to customers and end users by impacting both commercial and non-commercial goals.
The Quick Assessment
DORA provides on line the same assessment they ask of organizations. It'll give you a quick understanding of where your business stands compared to others in the same industry. I've listed the metrics in the Executive Summary section, below.
The Full Report
- "Our research continues to show that the industry-standard Four Key Metrics of software development and delivery drive organizational performance in technology transformations."
- Lead time (time from code committed to code released to production)
- Deployment frequency
- Mean time to restore (when there's an outage or bug that affects usage)
- Chail fail percentage (of released changes, how many fail and affect usage)
- "For organizations seeking guidance on how to improve, we point to the only real path forward: Start with foundations, and then adopt a continuous improvement mindset by identifying your unique constraint (or set of constraints). Once those constraints no longer hold you back, repeat the process."
- "The use of cloud—as defined by NIST Special Publication 800-145— is predictive of software delivery performance and availability. The highest performing teams were 24 times more likely than low performers to execute on all five capabilities of cloud computing."
Who Took the Survey?
- "Overall, we see similar representation across key demographic and firmographic measures when compared to last year, other than a noticeable drop in the reported percentage of women on teams." (emphasis mine)
Demographics and Firmographics
- Gender: Male 83%, Female 10%, Other 7%
- Disability: 6%
- Underrepresented groups: 14%
- More than 16 years exeperience: 48%
- Departments: Dev/Eng 30%, DevOps/SRE 26%, Manager 16%, IT Ops/Infrastructure 5%
- Industry: Tech 38%, Finance 12%, Retail/Consumer/e-Commerce 9%
- Region: North America 50%, EU/UK 29%. Respondents from Asia fell from 18% to 9%.
- Employees: 10K+ 26%, 100-499 21%, 500-1,999 15%, 20-99 13%
- Operating Systems: Looks like an even split between Windows and Linux.
How do Elite Performers...Perform?
- Deployment Frequency: On Demand (multiple deploys per day)
- Lead time for changes: Less than one day
- Time to restore service: Less than on hour
- Change failure rate: 0-15%
The Research Models
|Culture of Psychological Safety||Culture of Psychological Safety|
|Change Management||Useful, Easy-To-Use Tools|
|Technical Practices||External Search|
|Disaster Recovery Testing||Technical Debt|
Software Delivery & Operational Performance
- Learn from the retail example, who...
- Embraced A/B testing to understand customers' buying habits
- Require efficiency and automation
- Leveraged the cloud to enable handling swings in demand
- Learned how to operate in regulated environments quickly and securely.
- More and more organizations are choosing multi-cloud and hybrid cloud solutions.
- What matters is how teams implement their cloud services. The five essential characteristics of cloud computing are:
- On-demand self-service
- Broad network access (from desktop, phone, etc)
- Resource pooling (multi-tenant models)
- Rapid elasticity (immediate capability provionsing)
- Measured service (automatic resource usage reporting)
- Technical Practices: Execute for maximum effect. It's useful to include these complete statements:
Many organizations wanting to adopt DevOps look for a set of prescriptive steps or best practices to guide their journey. However, every organization is different and which practices to adopt depends on the current state of the organization—including the state of its technology, culture, and processes—and its short- and long-term goals.
The solution is to take a holistic approach, where you first work to understand the constraints in your current software delivery process with an eye to your short- and long-term outcomes in measurable terms. Then empower teams to decide how best to accomplish those outcomes— after all, they are the experts in their work and context.14 Those who adopt this approach see more scalable and flexible solutions, and by not having to micromanage detailed execution plans, management can focus on high-level outcomes, allowing their organizations to grow.
- This year, the research looked at the relationship between continuous integration and automated testing and found that automated testing positively impacts CI.
- Loosely coupled architecture is still important
- Code maintainability positively contiributes to successful continous delivery (CD)
- Disaster recovery plans without testing are useless.
- Only 40% of respondents perform disaster recovery testing at least annually using one or more of the methods listed. Organizations that conduct disaster recovery tests are more likely to have higher levels of service availability—that is, the ability for technology teams and organizations to make and keep promises and assertions about the software product or service they are operating.
- Heavyweight change processes, such as change advisory boards, negatively impact software delivery performance.
- A culture of psychological safety is predictive of software delivery performance.
This is the first year DORA has investigated productivity.
But what is productivity, and how should we measure it? Productivity cannot be captured with a simple metric such as lines of code, story points, or bugs closed; doing so results in unintended consequences that sacrifice the overall goals of the team. For example, teams may refuse to help others because it would negatively impact their velocity, even if their help is important to achieve organizational goals.
Productivity is the ability to get complex, time-consuming tasks completed with minimal distractions and interruptions.
- Developers need useful and easy-to-use tools. Purchasing based on lower cost is the wrong approach, because harder-to-use tools cost developers in productivity.
- "Automation is truly a sound investment."
- Effective ability to search and find internal resources increases productivity.
- Likewise (and slightly lower) productivity is seen when searching external sources such Stack Overflow.
- "Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a standstill under the debt load of an unconsolidated implementation." --Ward Cunningham, 1992
- Additional Benefits of Improved Productivity
- Better work recovery (leaving work at work)
- Reduced burnout
How Do We Transform: What Really Works
Mashups are commonly reported in this sample at 40%, but they lack sufficient funding and resources in any particular investment. We caution that without a strategy to guide a technology transformation, organizations will often make the mistake of hedging their bets and suffer from “death by initiative”: identifying initiatives in too many areas, which ultimately leads to underresourcing important work and dooming them all to failure.
- The top two strategies employed are Communities of Practice and Grassroots, followed by Proof of Concept (PoC) as a Template (a pattern where the PoC copies) and PoC as a Seed.
- The four strategies that work (used by high/elite performers). See the report's Appendix B for details.
- Community Builders, seen 46% of the time
- University, 9%
- Emergent, 23%
- Experimenters, 22%
Every decade has its own trendy software methodology. While they all seem to feel better, history proves them to be ineffective. However, we see continued evidence that DevOps delivers value, and for six consecutive years, we have statistically verified key capabilities and practices that help organizations improve their software development and delivery using DevOps methods.
DevOps is not a trend, and will eventually be the standard way of software development and operations, offering everyone a better quality of life.