Shift Leftfrom Detection to Prevention

Ilya Sakharov, Director of Quality Assurance, HelloFresh

Ilya Sakharov, Director of Quality Assurance, HelloFresh

Quality mindset and understanding of how we deliver high-quality software in a fast-paced environment is undergoing a significant change. As the demand for new functionality and speed to market have increased, either by customer demands or competition, we have embraced more lean, agile methodologies of delivery. DevOps culture and CI/CD are in-terms of present day is in most of the delivery and quality related conversation and are put to use in many companies.

This change demands a thorough review of how we introduce a different quality process and avoid making it a bottleneck and an inhibitor of fast delivery. Although any quality check or process is inherently slowing down the process, it is clear that not having one eventually costs much more. Having said that, its crucial to find the proper balance between quality ensuring mechanisms and business impact caused by a potential slowdown in process, and constantly refine the approach for efficiency.

Prevention-Detection-Analysis is a simplified, yet effective approach to describe a quality delivery system, where the system is a set of actions and processes bound together to ensure the highest possible quality at the egress point of the system. Simply put, “Prevention” is a name of a group of mechanisms and processes that are put in place in an offer to prevent the issue to be introduced into the product code, while “Detection” is a set of those meant to detect the issues that have been introduced. The relation between those two can resemble the Agile vs. Waterfall approaches. While the classic waterfall is all about detection, agile, iterative approach is not possible without focusing on the preventive phase. Following the Jones/Capers bug introduction and cost chart below, it’s also clear that the detection phase is expensive to rely on solely.

Companies still often rely heavily on UI based end to end tests that represent a user journey through the product

Many organisations have clear ownership demarcation between the phases. Prevention is stereo-typically followed by development departments and includes processes of peer review, pair programming, static code analysis, unit testing, and more, while detection can mostly be found in QA domain where the functional, E2E, regression and other types of testing are performed. With the exception maybe for monitoring which is the DevOps/SRE domain and gets a lot of focus lately, per growing understanding that early detection and rapid recovery is in many cases what a product being measured with by the users, on top of Mean Time Between Failures. To ensure high MTBF without compromising (too much) on the delivery speed, a preventive focused quality process must be set in place.

Some of the quality related activities will reside in the detection phase for the foreseeable future as they demand a product being compiled, deployed, and properly configured. A security penetration test, system end-to-end, load tests, are some of the examples for such While other quality related activities should be constantly challenged whether they should remain in this, late, phase of testing and whether they can be replaced by more preventive methods while still providing the same level of confidence.

Companies still often rely heavily on UI based end to end tests that represent a user journey through the product. Those tests are using tools that emulate user actions to perform a business flow and confirm that the system behaves as expected and returns an expected result. While those tests are legitimate, they are fragile, slow, heavy on resources and most important - demand constant maintenance if a product has active development cycles. Following the Test Pyramid principle, first defined by Mike Cohn, it would be much cost effective to push those tests to the Service/API/Integration level where automation is much faster, stable and hence, cheaper. By doing that the tests will no longer require the whole product to be created or deployed, which will effectively move this activity from detection to prevention. The problem of API/Integration tests being more isolated and missing the big, user-centric, the picture is, in fact, diminishing as micro service product architecture concepts are being embraced. While not all UI based end-to-end tests should be converted to sets of integrations tests, an organisation can significantly reduce costs and increase speed by converting most of them and leaving minimal amount to ensure user experience of a product.

Ideally, UI based end-to-end tests will completely disappear when the front end holds no business logic whatsoever, or every operation done through UI would have a corresponding API/hook in place for testing purposes. In this case, a visual, maybe even manual inspection, once in a while would be sufficient to ensure UX is intact, the rest will be covered by frequently running, fast and reliable integration tests.

The third and the closing link of a circle is the analysis part. In iterative, fail-fast environments, it is important to constantly reevaluate and redistribute the activities and mechanisms in detection and prevention phases to ensure the most efficient resource allocation and highest quality output.

It is often that prevention and detection terms are used to describe parts of an existing quality process statically. While thinking of those groups as dynamic, moving and complementary parts of a single process turns it into a tool for improvement and practical application of “shift left” doctrine.