Table of Contents
1 Motivation
2 Concept of Automated Testing IT Software Development
3 Methodology for Demonstration of Benefits
4 Defining Objectives and KPIs associated with Automated Testing
5 Risks and Opportunities
6 Costs and Benefits of Introducing Automated Tests
7 The Sensitivity of Benefits to Variance in KPIs
8 Summary
1 Motivation
Majority of industries, including information and communication technologies, are undergoing swift-evolution involving automation of various repetitive activities. Significant public attention received automation of manufacturing processes with the replacement of tedious, humans-performed processes by robots and automatic production lines in various configurations. Less visible, but perhaps even more drastic, is automation in corporate offices. Variety of roles dealing with “pushing the paper” around are replaced by information systems, so that repetitive verification and validation activities are gradually taken out of the labor market. Software development is a similar case as it requires extensive testing prior to its release to the end-users. The growing list of features for a pSW package in time means increasing time and labor required to test functionalities as such, alongside with executing the set of regression tests.
The presented paper aims at clearing the mist concerning hard benefits of introducing automated testing in software development. Additionally, a rigorous methodology for determining the benefits is presented to support the case of automated testing in the decision making process of key stakeholders. Alongside hard benefits impacting profit and loss statement, multiple soft benefits might be associated with automated testing with the ultimate impact on long term performance of the business entity.
2 Concept of Automated Testing IT Software Development
Testing in general aims at the comparison of actual outcomes versus predicted outcomes. Automated testing utilizes software separate from the actual software under validation to control the execution of testing. Automated testing is a critical enablement factor for the introduction of continuous integration and delivery. It spreads across multiple levels of the project: Core of automated testing lies in unit tests that validate whether units of code match the expected functional requirements. The vast majority of automated tests fall into this category, accounting approximately 70% of all tests. Another group of tests focuses on functionality SW components, such as user account creation, user authorization or order processing reflecting on a typical e-shop application. In turn, integration testing ensures the individual components interacts as intended, e.g. customer data are transferred properly through the entire order fulfillment process. Individual areas of testing form a pyramid as illustrated in Figure 1.
Figure 1: Pyramid of automated testing1]
1]Available at www.quateslab.com
Followed by API tests, the very peak of the pyramid is formed by GUI tests that have typically the least share in the total amount of automated test cases. Expensive to implement, GUI tests are susceptible to even slight modifications of GUI layout or functionalities.
To fulfill its purpose, the automated testing suite shall meet the following attributes:
- Ability to detect faults
- Tests ought to be easy to create
- Tests should be executable automatically
- Test results are reported out automatically
- The testing framework is integrated with other tools in the development and deployment chain
- Evaluation of results is performed by the team
Introduction of automated testing in SW development and deployment process needs to be supported by appropriate infrastructure and tools. Additionally, the testing framework provides an integrated system defining rules of automation.
3 Methodology for Demonstration of Benefits
A wide array of inputs influence decision making whether to implement or to what extent, automated SW testing. A transparent methodology for determining benefits of automated testing, including associated risks and opportunities, is essential for qualified decision making. Figure 2 illustrates the workflow deployed in the assessment of automated testing for the purpose of allocating investments in the enterprise.
Figure 2: Workflow for decision making and evaluation of costs vs benefits associated with the introduction of automated testing
Based on projects objectives, the methodology steers the project team toward developing a set of KPIs for measuring the success of the projects. Both risks and opportunities might considerably influence the ability to achieve expected benefits. They represent valuable inputs to a sensitivity analysis meaning risks might reduce benefits, whilst opportunities might increase them. Thus decision-makers are presented a business case including not only the baseline scenario, but also a number of what-if scenarios taking into account the severity of risks, and magnitude of opportunities.
Benefits of introducing automated testing are going to be demonstrated at a representative