The Need For an Agile Test Strategy

Derk-Jan de Grood, Senior Test Manager and Agile Transition Coach

Derk-Jan de Grood, Senior Test Manager and Agile Transition Coach

“You have to realize,” the project manager explains, “that this project is very important for the organization. The planned adjustments cut across all systems, but they are necessary to be distinctive in the future. I want to be confident that the customer journeys still work after every release, but how do I know if this is the case? Most of our teams are agile. When I start talking about a test strategy, they say that this is old school and outdated.” She looks at me questioningly. I have been invited to advise her on the test approach and now she expects an answer.

With the arrival of Agile, a lot has changed within software development. The approach brings new challenges for the testers. Perhaps the biggest change is that the agile teams themselves are responsible for delivering quality. Michael Sowers puts it very well in one of his blogs:“In Agile, the accountability for the right level of quality delivered at the right time belongs to the collective team. The team should embrace the best skill sets of each contributor and plan for quality and testing at every step of the release and within each sprint.”

Consequently, the responsibility for testing shifts from the testers to the entire team. We prefer to include test in the sprint as much as possible. The need to organize them as separate sub-projects is no longer necessary and test managers are therefore no longer needed.

Coloring the map green

An obvious response to the project manager's question would therefore be to explain that we do not need a comprehensive strategy at all and that the teams themselves need to think about how they approach their tests. Not having a strategy has the advantage that we reduce the overhead big time and limit the number of artifacts. We are relieved of the big design up front and can define the tests just in time based on the knowledge that we have at that moment. Despite these advantages, I do not believe so. I have spoken to a many development teams that really take quality seriously. But when I asked them at the start of a sprint how they were going to tackle testing, the reactions were often in the style of ‘we know our application’ and ‘trust us, we know what we are doing’. I rarely received a clear answer.

Testing is more than finding errors. Tests must also show what works. They not only provide insight into product risks, but also create trust. In addition, testing provides insight into the progress of the realization. This enables the organization to make adjustments, take mitigating actions or to prioritize work items.

The business wants to accept the requested solution - a release, a minimal viable product or a project result - with confidence. To this end, we will need agreement on the tests that need to be executed. I like to visualize this as a map containing plots. The plots describe high over what tests need to be executed, but they do not describe how the tests need to be done; that's up to the agile teams.

During the realization we color in the map plot by plot. Green to indicate that the associated tests are done. We do this based on the test results that we collect from the teams. As the map gradually turns green, trust develops in the solution.

The teams are in the lead, but a test strategy can help them to coordinate the activities, to be able address responsibilities properly and not to let the team-transcending tests fall between two stools

Teams that test in their silo and use “we always work this way” to restrain themselves from providing insight undermine this process. They must learn that just doing the right tests is not enough; they must also be transparent about progress and dependencies. They will have to explain that they have done the right tests. Tell the test story, provide the proof.

Embedding in the organization

Test challenges become more technical. Automation plays an increasingly important role. Many organizations strive for continuous integration and delivery (CI / CD) to accelerate their release. Testing must take pace in this acceleration and therefore needs to be automated. Stubbing and service virtualization are ways to test the integration of a component in conjunction with the other systems without being dependent on availability. Not every team has the knowledge to realize this on its own.

At the same time, I see the test profession widening. It is becoming increasingly difficult to ignore the non-functional tests for security, performance and UX. Testing IoT and mobile applications requires different knowledge than that required for testing the information systems that are so familiar to us. This makes it very difficult to assign all tests to the development team. It often does not have all the required knowledge.

The project manager that I had to advise wanted all the customer journeys to continue to work. Integration or chain tests will be needed, but how to deal with cross-team responsibilities? In practice, it does not work to put this responsibility entirely onto the teams. We need to support and facilitate them. While many test challenges transcend the responsibility of a single team, some even exceed the IT-domain. Organizations are increasingly realizing that a minimally viable product only yields business value if the organization can actually deliver the service, sell the product and support the process. It therefore seems logical to conduct business validation tests also. Instead of testing integrated systems, we embed the systems in the organization. Organizational readiness becomes a test topic, a business plot on the before mentioned map.

A Valuable guideline

Considering the above, I advised the project manager to take care of a view things. First to create a map and define the plots on it. This so we know what must have been tested in before we decide to launch the product with confidence.

Then to ensure that we do our tests as quickly as possible and that we can collect the results.

In addition, we can make the team responsible for the tests, but we may also have to help them if it appears that they lack knowledge or experience. “To this end”, I suggested, we can take control measures, for example by conducting audits to identify any areas for improvement. We can help the teams through hands-on coaching, and we can actively link them to domain experts and specialists for substantive questions.

We need to position the end-to-end and business tests. Either with the teams or with a special test team. In the first case be aware that this will have an impact on the progress within the teams and in the second case there will be dependencies that we have to manage.

I looked at the project manager and checked whether she understood the implication. “There is much to be done. Little chance that things will go well automatically,” I said, “The teams are in the lead, but a test strategy can help them to coordinate the activities, to be able address responsibilities properly and not to let the team-transcending tests fall between two stools. We do not need a thick and bulky document, as long as it mentions the right things. Finally, we will set up governance so that we gain insight into the progress of the tests and the project.”

The meeting ended with the decision to set-up and implement an agile test strategy together with the teams. During the project, this strategy proved to be a valuable guideline through which activities could be deployed, and thus an important success factor.