Testautomation architecture?

Why is it still a challenge to set up good test automation? In practice I notice that test automation is still seen as a separate competency. Nothing could be further from the truth, because it belongs to a competence that has existed for many years: software development.

As with software development, the consequences of poor architecture can also be significant in test automation. Some of the symptoms I often hear are:

• Long lead time for the development of test automation and a low ROI because the solution is not flexible enough to adapt certain things;
• A lot of incorrect and unreliable information due to a high instability of the test output;
• No possibility to parallelise the test case output.

If you want to combat these symptoms, you need to take a few steps back and reconsider the situation. Only then can you see opportunities for improvement and take action. Conducting ongoing research into a good foundation for test automation based on software architecture principles has helped many software development teams. In this article I summarise the most important lessons.

What is Architecture?

Architecture can be compared to a construction drawing. The architecture describes which technical components are used in which way and how those components are connected to each other. An architecture document is often a living document that describes the desired situation and in which possible risks are quickly identified as well as possible solutions. Such a document will help to clearly explain the further development of software solutions in the bigger picture.

A sound architecture influences the quality, stability and extensibility of the software solution. It is seen as a guide to the software development process.

It should be clear that architecture is developed iteratively. This means you have to regularly check whether the architecture still corresponds to reality. Adjustments can, for example, relate to a reorganisation, new functionality or changing customer requirements. What many don’t realise is that changes in the foundation (read architecture) always take time and involve costs. Here too, the (further) development of software does not differ from the construction of a house. If the foundation or construction is not heavy enough to support a new extension or superstructure, these ‘invisible’ investments must first be made in order to successfully integrate the new construction.

How do we set up the architecture for test automation?

In an architecture document you record the following things:

• Needs and interests of the customer. Think of the requirements that the software must meet. In addition to business and functional requirements, you also specify technical requirements such as security and performance.
• The organisation of the technical components. What technical components or programming languages ​​are there? How do they communicate with each other? What dependencies should we take into account? How can we clearly describe and visualise this?
• Risks and mitigation. What risks do we see in architecture and how are we going to mitigate them?

On the basis of these themes, you then draw up the architectural principles, containing core values ​​or qualities that a principle should meet. These principles safeguard the wishes and interests of the customer and the organisation.

I use the following principles for test automation architecture setup:

1. Operation: Implementing and executing the test automation.
   Possible core values ​​are: user-friendliness from the integrated development environment (IDE), error sensitivity during output, future-proofing, integration with CI/CD, support by the team, scalability and security.
2. Testing: Setting up and developing quality controls in test automation.
   Possible core values ​​are: designing or modelling test cases, maintenance, parallelisation, test data, test coverage, test case independence, speed and testability of the systems under test (SUT).
3. Development: the software development to be able to perform quality checks.
   Possible core values ​​are: user-friendliness in verifications, mocking or stubbing, versioning, debugging, refactoring, coding standards and practices such as KISS, DRY, YAGNI and SOLID
4. Feedback: reporting quality checks and interpreting the outcome. Possible core values ​​are: understanding the test results, logging, extent to which the report is informative and the speed of test execution.

The layering of the test automation architecture

In addition to the principles and core values ​​of test automation, it is important to apply a good layering in test automation in order to be able to use the above principles.

I distinguish the following questions that will help to formalise a suitable solution.

1. Why do we need to check? What is the need?
   I often refer to this as the specification or test layer . This includes:
   a. Functional test descriptions in which the scenarios are made clear;
   b. Translation of functional description into executable steps.
2. What should we check? What steps do we take?
   I often refer to this as the actions and verification layer . Think of:
   a . Technical objects that we need to perform actions on (such as object repositories or page objects);
   b. Technical steps we need to perform in order to perform verification.
3. How are we going to check? What means do we use to check?
   I call this the driver or control layer . Think of:
   a. Technical components to control systems (such as Web, REST, message queues or database libraries);
   b. Technical components to realise test data or conversions, for example.
4. How are we going to perform and report the tests? What resources do we use to provide information?
   I call this the output and reporting layer. This includes:
   a. Technical components to run tests (such as test runners, CI/CD);
   b. Technical components to document the test output (such as screenshots, videos, error reporting, etc.).

Architecture pays off

By regularly discussing and structuring the interfaces with architecture in test automation, I notice that many of the problems mentioned have been solved with customers. Below is a small selection of my observations:

• The turnaround time for test automation is reduced and the ROI increases because there is more flexibility in adapting and expanding the test automation through the right layering in the architecture.
• Due to this faster turnaround time, problems are discovered earlier and the costs of finding and solving those problems remain low.
• There is less and less false information (false positives or false negatives) because stability is created in the test results.
• There are many possibilities to scale the test automation to your own needs.
• The layering in the architecture makes it clear whether sufficient quality checks are performed on high-risk applications.

What does your ideal test automation solution look like? How are your architecture principles in today’s test automation? If you would like an answer to these questions, please contact me for a Test Automation Assessment, based on the test automation architecture principles and core values. This way you will find the improvements to set up a good foundation for test automation.