Work-in-progress! New chapter notes and quotes will be added as I complete reading them!
- 2018-10-14 Chapters 1-5 released
- 2018-11-05 Chapters 6-7 added
- 2018-11-06 Chapter 8 added
This is a chapter-by-chapter collection of notes and quotes for the book Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations, Nicole Forsgren PhD, Jez Humble, Gene Kim.
Any quotes from the book in this article are copyrighted. My intent is to provide an overview of what struck me in the material, and to encourage any visitor to purchase and read this important work.
Quick Reference: Capabilities to Drive Improvement
Our research has uncovered 24 key capabilities that drive improvements in software delivery performance. This reference will point you to them in the book.
The capabilities are classified into five categories:
- Continuous delivery
- Product and process
- Lean management and monitoring
CONTINUOUS DELIVERY CAPABILITIES
- Version control: Chapter 4
- Deployment automation: Chapter 4
- Continuous integration: Chapter 4
- Trunk-based development: Chapter 4
- Test automation: Chapter 4
- Test data management: Chapter 4
- Shift left on security: Chapter 6
- Continuous delivery (CD): Chapter 4
9. Loosely coupled architecture: Chapter 5
10. Empowered teams: Chapter 5
PRODUCT AND PROCESS CAPABILITIES
11. Customer feedback: Chapter 8
12. Value stream: Chapter 8
13. Working in small batches: Chapter 8
14. Team experimentation: Chapter 8
LEAN MANAGEMENT AND MONITORING CAPABILITIES
15. Change approval processes: Chapter 7
16. Monitoring: Chapter 7
17. Proactive notification: Chapter 13
18. WIP limits: Chapter 7
19. Visualizing work: Chapter 7
20. Westrum organizational culture: Chapter 3
21. Supporting learning: Chapter 10
22. Collaboration among teams: Chapters 3 and 5
23. Job satisfaction: Chapter 10
24. Transformational leadership: Chapter 11
Beginning in late 2013, we embarked on a four-year research journey to investigate what capabilities and practices are important to accelerate the development and delivery of software and, in turn, value to companies.
Chapter 1 - Accelerate
Companies, even big ones, are moving away from big projects, instead using small teams using small development cycles.
Software is at the heart of (most of) these transformations. [I don't personally believe that software is the end-all of business improvement, but it's true that many businesses don't realize how much software drives them.]
Maturity models don't work. Capability models do.
None of these factors predict performance:
- age and technology used for the application (for example, mainframe “systems of record” vs. greenfield “systems of engagement”)
- whether operations teams or development teams performed deployments
- whether a change approval board (CAB) is implemented
DevOps emerged from a small number of organizations facing a wicked problem: how to build secure, resilient, rapidly evolving distributed systems at scale.
A recent Forrester (Stroud et al. 2017) report found that 31% of the industry is not using practices and principles that are widely considered to be necessary for accelerating technology transformations, such as continuous integration and continuous delivery, Lean practices, and a collaborative culture (i.e., capabilities championed by the DevOps movement).
Another Forrester report states that DevOps is accelerating technology, but that organizations often overestimate their progress (Klavens et al. 2017). Furthermore, the report points out that executives are especially prone to overestimating their progress when compared to those who are actually doing the work.
To summarize, in 2017 we found that, when compared to low performers, the high performers have:
- 46 times more frequent code deployments
- 440 times faster lead time from commit to deploy
- 170 times faster mean time to recover from downtime
- 5 times lower change failure rate (1/5 as likely for a change to fail)
Chapter 2 - Measuring Performance
The academic rigor of this book is exceptionally high.
Three ways to masure performance that don't work: lines of code, velocity, and utilization.
Two characteristics of successful measures: focus on global outcomes, and focus on outcomes not output.
Lead Time is "the time it takes to go from a customer making a request to the request being satisfied." The authors focused on the delivery part of lead time--"the time it takes for work to be implemented, tested, and delivered."
The stats on "the impact of delivery perofrmance on organization performance" shows that software development and IT are not cost centers; they can provide a competitive advantage.
It's important to distinguish between strategic and non-strategic software. Strategic software should be kept in-house. See Simon Wardley and the Wardley mapping method.
There are many frameworks and methodologies that aim to improve the way we build software products and services. We wanted to discover what works and what doesn’t in a scientific way,...
Most of these measurements focus on productivity. In general, they suffer from two drawbacks. First, they focus on outputs rather than outcomes. Second, they focus on individual or local measures rather than team or global ones.
In our search for measures of delivery performance that meet these criteria, we settled on four: delivery lead time, deployment frequency, time to restore service, and change fail rate.
Astonishingly, these results demonstrate that there is no tradeoff between improving performance and achieving higher levels of stability and quality. Rather, high performers do better at all of these measures. This is precisely what the Agile and Lean movements predict,...
It’s worth noting that the ability to take an experimental approach to product development is highly correlated with the technical practices that contribute to continuous delivery.
The fact that software delivery performance matters provides a strong argument against outsourcing the development of software that is strategic to your business, and instead bringing this capability into the core of your organization.
The measurement tools can be used by any organization,...
However, it is essential to use these tools carefully. In organizations with a learning culture, they are incredibly powerful. But “in pathological and bureaucratic organizational cultures, measurement is used as a form of control, and people hide information that challenges existing rules, strategies, and power structures. As Deming said, 'whenever there is fear, you get the wrong numbers'”
Chapter 3 - Measuring and Changing Culture
Not only is culture measurable for the purposes of the book, but the authors learned that DevOps can "influence and improve culture."
Westrum's three characteristics of good information:
- Provides the needed answers
- Presented so can be used effectively
A good culture:
- Requires trust and cooperation
- Has higher quality decisionpmaking
- Teams do better job with their people
John Shook, in a video on the Lean Transformation Model, makes it clear that the approach is value-driven approach, the "true north" of any situation. John Shook Explains the Lean Transformation Model - YouTube
Ron Westrum's three kinds of organizational culture
Pathological (power-oriented) organizations are characterized by large amounts of fear and threat. People often hoard information or withhold it for political reasons, or distort it to make themselves look better.
Bureaucratic (rule-oriented) organizations protect departments. Those in the department want to maintain their “turf,” insist on their own rules, and generally do things by the book — their book.
Generative (performance-oriented) organizations focus on the mission. How do we accomplish our goal? Everything is subordinated to good performance, to doing what we are supposed to do.
Table 3.1 Westrum's Typology of Organizational Culture.
||Risks are shared
|Failure leads to scapegoating
||Failure leads to justice
||Failure leads to inquiry
||Novelty leads to problems
Using the Likert-scale questionnaire
To calculate the “score” for each survey response, take the numerical value (1-7) corresponding to the answer to each question and calculate the mean across all questions. Then you can perform statistical analysis on the responses as a whole.
Westrum’s theory posits that organizations with better information flow function more effectively.
Thus, accident investigations that stop at “human error” are not just bad but dangerous. Human error should, instead, be the start of the investigation.
“ what my... experience taught me that was so powerful was that the way to change culture is not to first change how people think, but instead to start by changing how people behave — what they do” --John Shook, leader in lean manufacturing
Our research shows that Lean management, along with a set of other technical practices known collectively as continuous delivery (Humble and Farley 2010), do in fact impact culture,...
Chapter 4 - Technical Practices
CD practices will help improve culture. However, implementing the practices "often requires rethinking everything".
Keeping system and application configuration in version control is more important to delivery performance than keeping code in version control. (But both are important).
Continuous delivery is a set of capabilities that enable us to get changes of all kinds — features, configuration changes, bug fixes, experiments — into production or into the hands of users safely, quickly, and sustainably. There are five key principles at the heart of continuous delivery:
- Build Quality In
Invest in building a culture supported by tools and people where we can detect any issues quickly, so that they can be fixed straight away when they are cheap to detect and resolve.
- Work in Small Batches
By splitting work up into much smaller chunks that deliver measurable business outcomes quickly for a small part of our target market, we get essential feedback on the work we are doing so that we can course correct.
- Computers perform repetitive tasks; people solve problems
One important strategy to reduce the cost of pushing out changes is to take repetitive work that takes a long time, such as regression testing and software deployments, and invest in simplifying and automating this work.
- Relentlessly pursue continuous improvement
The most important characteristic of high-performing teams is that they are never satisfied: they always strive to get better.
- Everyone is responsible
in bureaucratic organizations teams tend to focus on departmental goals rather than organizational goals. Thus, development focuses on throughput, testing on quality, and operations on stability. However, in reality these are all system-level outcomes, and they can only be achieved by close collaboration between everyone involved in the software delivery process. A key objective for management is making the state of these system-level outcomes transparent, working with the rest of the organization to set measurable, achievable, time-bound goals for these outcomes, and then helping their teams work toward them.
A key goal of continuous delivery is changing the economics of the software delivery process so the cost of pushing out individual changes is very low.
In order to implement continuous delivery, we must create the following foundations:
- Comprehensive configuration management
It should be possible to provision our environments and build, test, and deploy our software in a fully automated fashion purely from information stored in version control.
- Continuous integration
High-performing teams keep branches short-lived (less than one day’s work) and integrate them into trunk/master frequently. Each change triggers a build process that includes running unit tests. If any part of this process fails, developers fix it immediately.
- Continuous testing
Automated unit and acceptance tests should be run against every commit to version control to give developers fast feedback on their changes. Developers should be able to run all automated tests on their workstations in order to triage and fix defects.
Implementing continuous delivery means creating multiple feedback loops to ensure that high-quality software gets delivered to users more frequently and more reliably.
Chapter 5 - Architecture
See the quote below, it's important that teams be loosely coupled, not just the architecture. But how does this apply to a small software/IT shop of, say a half dozen employees?
I see autonomy of teams coming up quite a bit, not having to ask for permission to change from someone outside the team, and also that changes don't strongly affect other teams. This is a result of loose-coupling.
So, it's interesting that good information flow is important, but that teams shouldn't need to overly communicate in order to do their work.
The authors make a great point about service-oriented architecture and microservices, that these both can enable the desired outcome of testability, but, they don't guarantee it. The team has to be vigilant about ensuring the ability to independently test services.
A good, loosely-coupled architecture allows not just the software but the teams to scale. Contrary to assumptions, adding employees when there's a proper software/team architecture leads to increasing deployment frequency.
I've been in the camp that wants to reduce or at least make tooling consistent. I think there's still some value in that from a maintainability perspective. However, using the better tool for the job apparently has more value. I don't think the authors are advocating a gung-ho or cavalier approach, though.
We found that high performance is possible with all kinds of systems, provided that systems—and the teams that build and maintain them—are loosely coupled.
We discovered that low performers were more likely to say that the software they were building—or the set of services they had to interact with—was custom software developed by another company (e.g., an outsourcing partner).... In the rest of the cases, there was no significant correlation between system type and delivery performance. We found this surprising: we had expected teams working on packaged software, systems of record, or embedded systems to perform worse, and teams working on systems of engagement and greenfield systems to perform better. The data shows that this is not the case.
This reinforces the importance of focusing on the architectural characteristics, discussed below, rather than the implementation details of your architecture.
Those who agreed with the following statements were more likely to be in the high-performing group:
- We can do most of our testing without requiring an integrated environment.
- We can and do deploy or release our application independently of other applications / services it depends on.
The goal is for your architecture to support the ability of teams to get their work done — from design through to deployment — without requiring high-bandwidth communication between teams.
When teams can decide which tools they use, it contributes to software delivery performance and, in turn, to organizational performance.
That said, there is a place for standardization, particularly around the architecture and configuration of infrastructure.
Another finding in our research is that teams that build security into their work also do better at continuous delivery. A key element of this is ensuring that information security teams make preapproved, easy-to-consume libraries, packages, toolchains, and processes available for developers and IT operations to use in their work.
Architects should focus on engineers and outcomes, not tools or technologies.... What tools or technologies you use is irrelevant if the people who must use them hate using them, or if they don’t achieve the outcomes and enable the behaviors we care about. What is important is enabling teams to make changes to their products or services without depending on other teams or systems.
Chapter 6 - Integrating InfoSec Into the Delivery Lifecyle
InfoSec is everyone's responsiblity.
Don't leave security reviews until the end of development. Security is equal to Development and Operations.
many developers are ignorant of common security risks, such as the OWASP Top 10
We found that when teams “shift left” on information security — that is, when they build it into the software delivery process instead of making it a separate phase that happens downstream of the development process — this positively impacts their ability to practice continuous delivery.
- First, security reviews are conducted for all major features, and this review process is performed in such a way that it doesn’t slow down the development process.
- the second aspect of this capability: information security should be integrated into the entire software delivery lifecycle from development through operations.
- Finally, we want to make it easy for developers to do the right thing when it comes to infosec.
We found that high performers were spending 50% less time remediating security issues than low performers.
Rugged DevOps is the combination of DevOps with the Rugged Manifesto.
Chapter 7 - Management Practices for Software
Lean management, as applied to software, currently yields better results.
Four components of Lean management applied to software:
- Limit Work in Progress (WIP)
- Visual Management
- Feedback from Production
- Lightweight Change Approvals
- Limiting work in progress (WIP), and using these limits to drive process improvement and increase throughput
- Creating and maintaining visual displays showing key quality and productivity metrics and the current status of work (including defects), making these visual displays available to both engineers and leaders, and aligning these metrics with operational goals
- Using data from application performance and infrastructure monitoring tools to make business decisions on a daily basis
WIP limits on their own do not strongly predict delivery performance. It’s only when they’re combined with the use of visual displays and have a feedback loop from production monitoring tools back to delivery teams or the business that we see a strong effect.
We found that approval only for high-risk changes was not correlated with software delivery performance. Teams that reported no approval process or used peer review achieved higher software delivery performance. Finally, teams that required approval by an external body achieved lower performance.
We found that external approvals were negatively correlated with lead time, deployment frequency, and restore time, and had no correlation with change fail rate. In short, approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability
Chapter 8 - Product Development
Lean Product Development
- Work in Small Batches
- Make Flow of Work Visible
- Gather and Implement Customer Feedback
- Team Experimentation
We wanted to test whether these [Lean] practices have a direct impact on organizational performance, measured in terms of productivity, market share, and profitability.
- The extent to which teams slice up products and features into small batches that can be completed in less than a week and released frequently, including the use of MVPs (minimum viable products).
- Whether teams have a good understanding of the flow of work from the business all the way through to customers, and whether they have visibility into this flow, including the status of products and features.
- Whether organizations actively and regularly seek customer feedback and incorporate this feedback into the design of their products.
- Whether development teams have the authority to create and change specifications as part of the development process without requiring approval.
It’s worth noting that an experimental approach to product development is highly correlated with the technical practices that contribute to continuous delivery.