What is the Regression Suite Automation Tool

Test automation on the rise!

The increasingly important role of automation in software testing

Software testing has established itself as a central element in software development, and in the course of this development, automated support for software testing has also become more and more important. Time and cost savings are strong motivations for this, and test automation has become an indispensable part of today's IT landscape.

The use of automation tools is already planned in the conception phase of a test project so that the tools support a test project as effectively as possible, because the range of tools and their areas of application are large. Applications range from State-of-the-art software development methods how Test-Driven Development above automated integration tests up to automated system and system integration tests via graphical and other user interfaces.

Experience has shown that there are also less successful automation projects that cause disproportionately high costs due to high maintenance costs and thus annoyance and frustration. This article shows some reasons for the failure of such projects and is intended to raise awareness of the possibilities, but also the limits of test automation and the tools associated with it.

How test automation works

Test automation was or is a niche in software development. In the case of automation via the user interface, a specialized development environment is generally used for a script language or an environment linked to a common programming language with prefabricated activity and action modules. The tools access the application's interfaces. In many cases these are the components of the user interface (buttons, text fields, trees, tables, etc.).

A selection of similar test cases that follow a common process is abstracted into a general scheme or a set of processes that consist of some key activities.

These schemes and processes are now implemented in scripts, often with the aid of the recording functionality of the automation tools, which can be controlled using spreadsheets or a database. In other words: test case variants that only differ in terms of the data can be automated in a data-controlled manner. The test team supplies control data that contain the specific test cases for these scripts, which can be carried out automatically with the help of these scripts.

But what are the benefits of automated tests?

In the developer test, the automation enables the development team to quickly make a statement about the function of the individual software modules. This safety net also allows major changes to be made without fear of unexpected deterioration.

Automation is also used by the specialized test team: apart from an additional safety net, it offers the possibility of releasing the testers from repetitive tasks. In iterative-incremental process models such as Scrum, for example, this can ensure that test specialists are free from tiresome regression tests and can concentrate on the currently relevant functionalities. So the purpose of automation here is not to find bugs. Rather, it is iYour task to confirm that the already completed functionality will not be affected by changes!

Last but not least, test automation also benefits the project client or project manager, since regression tests can be carried out more frequently, more cost-effectively and, above all, more reproducibly than by manual testers. This in turn helps reduce the number of errors after the release.

Many fail at the beginning - or even before that!

So why is there not automated test support in every project? In the component test, it is good form to have automated at least some unsystematically created test cases - but in the integration test and at the latest in the automated system test, the proportion of successfully used test automation is significantly lower. There are several reasons for this discrepancy: First of all, test automation means an investmentthat only brings the corresponding return over a longer period of time. The automation of a test case is more complex than a single execution and an effective benefit can therefore only be achieved after several runs. The resources needed to get started can obscure the long-term benefit.

Another factor concerns a widespread and fundamentally wrong expectation, especially with regard to commercial tools: Tools are purchased in the hope of gaining a sufficient basis for successful test automation. Once a tool has been purchased, it will Automateas a sideline seen by an inexperienced (and often overwhelmed) tester. This leads to tinkering in the proverbial "quiet little room" without a clear target or structure. In this way, apart from development and manual testing, one grows in unmaintainable, hardly feasible and difficult to analyze patchworkthat mixes technical and professional aspects. Test runs are conducted ad libitum and the results are not clearly communicated. In these cases, the tool is often blamed for the failure, and both the tool and the automation “gather dust”.

Special qualifications for test automation engineers are essential

Even if many tool manufacturers present their tools as "record and play" tools with which even people without any programming experience can immediately create automation suites, in real use it usually requires some experience in the field of software development. Many of the automation tools embed their scripts in a graphical user interface that defines processes using lists and trees, for example. However, it should be noted that these representations are basically nothing more than a visual preparation of the actual program code. This means that the same problems and questions arise with this representation as well, such as What happens if an error occurs during the test run? " and What happens if this error is not in the application but in the script? ".

A common misjudgment when deciding in favor of automated test support is that training as a tester is sufficient for the role of test automation specialist. Knowledge of processes and methodology in the test area is important for the development of test automation systems, but is usually rarely sufficient. In order to design and implement a test automation system, knowledge and experience in the development of software systems are necessary. This is especially true if this system is to be used flexibly over the long term. The lack of appropriate know-how manifests itself, as it is already well known from the field of software development: Unstable, unstable, inefficient and poorly scaling code is created. Experience with development, testing and the tools used are therefore not only an advantage, but also to a large extent decisive for the success of such a project.

The tool has to fit the user and the goals

A good selection of tools is necessary, but not enough on its own for success. When deciding on a tool, not only the technical compatibility and the price should be decisive: Another criterion that is at least as important is the compatibility of the objective, the tool and the user.

In one project it may be possible to use tools that require programming knowledge in specific languages, in others a graphic design of the test cases is absolutely necessary. Therefore, a test run on a living object is recommended before making a decision - ideally in the form of a proof-of-concept with the future users and based on the defined goals.

Another aspect is the introduction of tools and the associated processes: What is the input for the automation and what is the expected output? Who formulates the test cases, who implements them? When and how often are the automated test cases carried out? How are results communicated? All of these questions should be deliberately examined so that the resulting structures are not left to chance.

The course must be set correctly!

As with all investments in software development projects - and these include test automation projects, even if test cases can be created without programming knowledge - the course for test automation must also be set right from the start. But on the other hand, it should be monitored and readjusted again and again during ongoing use in order to be able to harvest the hoped-for yield - and possibly a little more - in the long term. A clear concept, a structured approach and well-considered embedding in the general software development process can help test automation specialists to focus their activities in a profitable manner.

When it comes to the technical implementation, a good structuring of the automation elements, e.g. with the help of keyword-driven test automation, and a look at the maintainability of the automated test cases are essential. Last but not least, the testability of the target application plays an important role: This ranges from clear identification features on the user interface to general aids, such as resetting a test environment to an initial state. In order to determine the testability requirements for the system to be tested in the long term, recording these requirements right from the start is suitable. Ideally, this consideration is coupled with the earliest possible test use of test automation (proof of concept, pilot) in order to be able to supplement requirements that have not yet been recorded and that are shown in the real environment in good time. On the other hand, the point in time for such a deployment should not be premature either: The system to be tested should have already reached a degree of maturity corresponding to the approach and the test level in order to save automation from aiming for a "moving target".

Inadequate planning and design of test automation projects

Many problems in test automation exist or arise during the planning phase:

  • Architecture decisions of the automation system and a well-considered selection of test cases, the automation of which is the goal, are special in test automation critical.
  • Lack of support by other project participants: management, developers, testers and departments must first be convinced of the usefulness of automation.
  • The necessary resources, in particular time, should be sufficient, as otherwise it would be difficult for the automation engineer to deal professionally and technically with the application to be tested with sufficient depth.
  • Analysis of the requirements or specifications for the automated system absence often completely, if one disregards the test cases whose automation is planned.
  • The effort involved must not be disregarded, otherwise there is a risk that The purpose and scope of the automation system were misunderstood and thus the development of the automation system bypasses the actual needs of the test.
  • Experience has shown that, as is also known from software development, the The complexity of the project itself is often underestimated. What starts out as a simple system with a few proof-of-concept test cases often grows very quickly, and parts of the system are reused in other areas.

Test automation systems require their expansion: Errors found in automatic or manual tests or even in productive use then require new test cases that are also to be automated. In order to be able to scale and expand the system sufficiently, a well-considered architecture is therefore necessary.

In the area of ​​test automation, there are a few approaches that should be considered. Methods and Techniques such as data, keyword or behavior driven development are now supported by numerous tools and can be used with almost any technology. The consistent use of these methods can largely avoid high maintenance costs and ensure the scalability of the automation system.

Another essential aspect that is now taken into account by some tools is parallel execution of test cases. In this context, not only the technical division of the test execution is a challenge, but above all the related content issues such as test data management (especially test data with technical state models behind it) and test configuration (such as test cases that the system to be tested under certain parameters have to test).

Lack of integration in the development process

An essential aspect for the sustainable success of test automation projects is Integration into a development process. If this point is not observed, this often leads to isolated automation solutions, the benefits of which are not fully realized or unrealistic expectations are placed on them. To prevent this, there are different starting points, depending on the development process used:

  • Continuous integration / continuous delivery
  • Operation (e.g. provisioning of the test environment)
  • Error management planning, prioritization and estimation

The integration should go so far that, for example, not just one test automation engineer is able to start a test run. Cooperation works optimally when any project team member can start a test run in dedicated test environments without first asking a large number of employees for permission or support. In addition to developers, the company, department employees and, above all, the customer also benefit from this option.

Agile - not without test automation

A high degree of automation is a prerequisite for project success, especially in the currently widespread agile process models. Due to the short iteration time, frequent releases and frequent changes even in advanced project phases, a detailed regression test is necessary. If this quality assurance measure is not used, there is a risk that new changes will affect existing functionality and errors may induce errors. However, since these changes are made very frequently, the effort for manual regression tests in these projects are often not viable and should therefore be automated or partially automated.

If continuous delivery (continuous integration & continuous provision on the productive system) is practiced, test automation is even more important and should therefore be pushed even more.

Failure to meet the requirements for test automation in the application

As already described, there are three main connection points for automatic test execution:

  • Access of the tool to interfaces the application to be tested
  • an effective one Test approach
  • a certain degree of maturity the application to be tested and its environment

At first glance, these points look quite insensitive: In our case, the interface is the user interface, which is expected to be operable on the one hand and also logical and consistent on the other. Schemas in the course of the test cases are another usability-supporting property that should be fulfilled by many applications. Unfortunately, not all that glitters is gold here: On the one hand, tools are very sensitive in these areas, on the other hand, they are the focus of further developments of software. Graphic redesigns of applications that do not change anything in terms of the actual functionality, cause maintenance costs for automation engineers or extensions to include interfaces for automation by developers. The same applies to the redesign of the processes in the application to be tested. If the automation suite is to be used sustainably and over a longer period of time, considerable maintenance costs can arise here.
Violations of the third point usually occur together with violations of the first two points, but an application whose design and process design can be regarded as finished can also start insufficient degree of maturity Suffer. As a result, the logs of automatic test runs are not very meaningful, as the errors prevent many test cases from being executed. Another consequence is that the occurrence of the faults shown is too frequent for a meaningful evaluation or cannot be reproduced.

Test Automation - The Wrong Approach?

In test automation, as in many other areas, the following principle must be observed: Just because something can be automated does not necessarily mean that it has to be automated.

As already described, the automation of test cases is not in every situation recommendable or sometimes not feasible at all. Automated UI tests on mobile devices is a current example of this.

In this case, you may have to resort to other approaches in order to keep the test effort within a manageable framework. An alternative is, for example Crowd testing. A large number of different devices can be tested in a relatively short time and at low cost. In addition, in this way one receives efficient feedback on non-functional properties such as usability or reliability. Nevertheless, crowdtesting should Not be seen as a pure alternative to test automation projects, but support them.

It is not easy. But effective!

If these aspects are taken into account, the chances are good to create stable test automation and to use it efficiently. However, it is not easy. As is so often the case, the experience of others can also be used here: Test automators from your own environment, external advice from experts, relevant reading material or forums and communities can help to take the important first steps in the right direction at the beginning.

First steps towards a worthwhile goal: The "free play" of testers who thereby focus on finding new and important bugs (instead of laboriously going through the same regression test cases over and over again), is probably the central quality feature of effective test automation, and represents an essential lever for quality improvement for project and product.

Test automation is a safety net

Finding errors within the various test types
(Source: Mark Fewster, ANECON expert breakfast, March 14, 2012)

At first glance, the error detection rate of 9% in the automated test may seem low, but the meaningful figure is the 58% in the exploratory test: Test automation is a safety net, but not a primary instrument for finding errors. What is automated has already worked properly - here errors are only identified as a side effect. The greater part of the errors is found in the manual, exploratory test, for which the tester has to spend a lot of time. Test automation creates space for the "creative" test, as the test team has more time and human creativity and test intelligence intervene precisely where the error occurs. The goal of test automation is therefore not primarily to save in the test team, but to create more time to devote to the methodical, "human", manual test in addition to the automatic regression.

Conclusion

Test automation is a sub-area of ​​software development and software testing that uses specialized tools and also a Link between the disciplines can represent. For a successful automation, additional effort is necessary at the beginning, which however, with good planning and competent implementation, leads to a significant improvement and increase in efficiency of the test. Professional test automatorswho have at least basic knowledge of software development and procedures and processes in testing are required for implementation. Ideally, you have extensive experience in the field of test automation. A level of maturity is a prerequisite for good test automation developmentthat corresponds to the framework of test automation. Support from management, development and test team are also important points, the importance of which should not be underestimated.