Chapter 3 - The Generic Test Automation Architecture

Generic Test Automation Architecture (gTAA)
1 Introduction to
Test Automation
2 Preparing for
Test Automation
3 gTAA
Software Testing - ISTQB Advance
Test Automation Engineer Exam Preparation
Chapter 3
Neeraj Kumar Singh
5 Reporting
& Metrics
6 Transitioning
Manual to Automation
7 Verifying TAS
8 Continuous
Improvement
4 Risks &
Contingencies

The Generic Test Automation Architecture
Contents
3.1 Introduction to gTAA
3.2 TAA Design
3.3 TAS Development
Neeraj Kumar Singh

Introduction to gTAA
A test automation engineer (TAE) has the role of designing, developing, implementing, and maintaining
test automation solutions (TASs). As each solution is developed, similar tasks need to be done, similar
questions need to be answered, and similar issues need to be addressed and prioritized. These
reoccurring concepts, steps, and approaches in automating testing become the basis of the generic test
automation architecture, called gTAA in short.
The gTAA presents the layers, components, and interfaces of a gTAA, which are then further redefined
into the concrete TAA for a particular TAS. It allows for a structured and modular approach to building a
test automation solution by:
 Defining the concept space, layers, services, and interfaces of a TAS to enable the realization of
TASs by in-house as well as by externally developed components
 Supporting simplified components for the effective and efficient development of test automation
 Re-using test automation components for different or evolving TASs for software product lines and
families and across software technologies and tools
 Easing the maintenance and evolution of TASs
 Defining the essential features for a user of a TAS
Neeraj Kumar Singh

Introduction to gTAA
A TAS consists of both the test environment (and its artifacts) and the test suites (a set of test cases including test
data). A test automation framework (TAF) can be used to realize a TAS. It provides support for the realization of
the test environment and provides tools, test harnesses, or supporting libraries.
It is recommended that the TAA of a TAS complies with the following principles that support easy development,
evolution, and maintenance of the TAS:
 Single responsibility: Every TAS component must have a single responsibility, and that responsibility must be
encapsulated entirely in the component. In other words, every component of a TAS should be in charge of
exactly one thing, e.g., generating keywords or data, creating test scenarios, executing test cases, logging
results, generating execution reports.
 Extension (see e.g., open/closed principle by B. Myer): Every TAS component must be open for extension, but
closed for modification. This principle means that it should be possible to modify or enrich the behavior of the
components without breaking the backward compatible functionality.
 Replacement (see e.g., substitution principle by B. Liskov): Every TAS component must be replaceable without
affecting the overall behavior of the TAS. The component can be replaced by one or more substituting
components but the exhibited behavior must be the same.
 Component segregation (see e.g., interfaces segregation principle by R.C. Martin): It is better to have more
specific components than a general, multi-purpose component. This makes substitution and maintenance
easier by eliminating unnecessary dependencies.
 Dependency inversion: The components of a TAS must depend on abstractions rather than on lowlevel details.
In other words, the components should not depend on specific automated test scenarios.
Neeraj Kumar Singh

Overview of gTAA
The gTAA is structured into horizontal layers for the following:
 Test generation
 Test definition
 Test execution
 Test adaptation
Neeraj Kumar Singh

Overview of gTAA
Neeraj Kumar Singh

Overview of gTAA
It is important to understand that these layers can be present or absent in any given TAS. For example:
 If the test execution is to be automated, the test execution and the test adaptation layers need to
be utilized. They do not need to be separated and could be realized together, e.g., in unit test
frameworks.
 If the test definition is to be automated, the test definition layer is required.
 If the test generation is to be automated, the test generation layer is required.
Most often, one would start with the implementation of a TAS from bottom to top, but other approaches
such as the automated test generation for manual tests can be useful as well. In general it is advised to
implement the TAS in incremental steps (e.g., in sprints) in order to use the TAS as soon as possible and
to prove the added value of the TAS. Also, proofs of concept are recommended as part of test
automation project.
Any test automation project needs to be understood, set up, and managed as a software development
project and requires dedicated project management. The project management for the TAF development
can be separated from the project management for the TAS.
Neeraj Kumar Singh

Test Generation Layer
The test generation layer consists of tool support for the following:
 Manually designing test cases
 Developing, capturing, or deriving test data
 Automatically generating test cases from models that define the SUT and/or its environment (i.e., automated
model-based testing)
The components in this layer are used to:
 Edit and navigate test suite structures
 Relate test cases to test objectives or SUT requirements
 Document the test design
For automated test generation the following capabilities may also be included:
 Ability to model the SUT, its environment, and/or the test system
 Ability to define test directives and to configure/parameterize test generation algorithms
 Ability to trace the generated tests back to the model (elements)
Neeraj Kumar Singh

Test Definition Layer
The test definition layer consists of tool support for the following:
 Specifying test cases (at a high and/or low level)
 Defining test data for low-level test cases
 Specifying test procedures for a test case or a set of test cases
 Defining test scripts for the execution of the test cases
 Providing access to test libraries as needed (for example in keyword-driven approaches)
The components in this layer are used to:
 Partition/constrain, parameterize or instantiate test data
 Specify test sequences or fully-fledged test behaviors (including control statements and expressions), to
parameterize and/or to group them
 Document the test data, test cases and/or test procedures
Neeraj Kumar Singh

Test Execution Layer
The test execution layer consists of tool support for the following:
 Executing test cases automatically
 Logging the test case executions
 Reporting the test results
The test execution layer may consist of components that provide the following capabilities:
 Set up and tear down the SUT for test execution
 Set up and tear down test suites (i.e., set of test cases including test data)
 Configure and parameterize the test setup
 Interpret both test data and test cases and transform them into executable scripts
 Instrument the test system and/or the SUT for (filtered) logging of test execution and/or for fault injection
 Analyze the SUT responses during test execution to steer subsequent test runs
 Validate the SUT responses (comparison of expected and actual results) for automated test case execution
results
 Control the automated test execution in time
Neeraj Kumar Singh

Test Adaptation Layer
The test adaptation layer consists of tool support for the following:
 Controlling the test harness
 Interacting with the SUT
 Monitoring the SUT
 Simulating or emulating the SUT environment
The test adaptation layer provides the following functionality:
 Mediating between the technology-neutral test definitions and the specific technology requirements of the SUT
and the test devices
 Applying different technology-specific adaptors to interact with the SUT
 Distributing the test execution across multiple test devices/test interfaces or executing tests locally
Neeraj Kumar Singh

Configuration Management of A TAS
Normally, a TAS is being developed in various iterations/versions and needs to be compatible with the
iterations/versions of the SUT. The configuration management of a TAS may need to include:
 Test models
 Test definitions/specifications including test data, test cases and libraries
 Test scripts
 Test execution engines and supplementary tools and components
 Test adaptors for the SUT
 Simulators and emulators for the SUT environment
 Test results and test reports
Neeraj Kumar Singh

Project Management of A TAS
 As any test automation project is a software project, it requires the same project management as any other
software project.
 A TAE needs to perform the tasks for all phases of the established SDLC methodology when developing the TAS.
 Also, a TAE needs to understand that the development environment of the TAS should be designed such that
status information (metrics) can be extracted easily or automatically reported to the project management of
the TAS.
Neeraj Kumar Singh

TAS Support for Test Management
 A TAS must support the test management for the SUT. Test reports including test logs and test results need to be
extracted easily or automatically provided to the test management (people or system) of the SUT.
Neeraj Kumar Singh

TAA Design
Capture requirements needed to define an appropriate TAA
The requirements for a test automation approach need to consider the following:
 Which activity or phase of the test process should be automated, e.g., test management, test design, test
generation, or test execution. Note that test automation refines the fundamental test process by inserting test
generation between test design and test implementation.
 Which test level should be supported, e.g., component level, integration level, system level
 Which type of test should be supported, e.g., functional testing, conformance testing, interoperability testing
 Which test role should be supported, e.g., test executor, test analyst, test architect, test manager
 Which software product, software product line, software product family should be supported, e.g., to define the
span and lifetime of the implemented TAS
 Which SUT technologies should be supported, e.g., to define the TAS in view of compatibility to the SUT
technologies
Neeraj Kumar Singh

TAA Design
Compare and contrast different design/architecture approaches
The TAE needs to analyze the pros and cons of different approaches when designing selected layers of the TAA.
These include but are not limited to:
Considerations for the test generation layer:
 Selection of manual or automated test generation
 Selection of for example requirements-based, data-based, scenario-based or behavior based test generation
 Selection of test generation strategies (e.g., model coverage such as classification trees for data-based
approaches, use case/exception case coverage for scenario-based approaches, transition/state/path coverage
for behavior-based approaches, etc.)
 Choosing of the test selection strategy. In practice, full combinatorial test generation is infeasible as it may lead
to test case explosion. Therefore, practical coverage criteria, weights, risk assessments, etc. should be used to
guide the test generation and subsequent test selection.
Neeraj Kumar Singh

Considerations for the test definition layer:
 Selection of data-driven, keyword-driven, pattern-based or model-driven test definition
 Selection of notation for test definition (e.g., tables, state-based notation, stochastic notation, dataflow
notation, business process notation, scenario-based notation, etc. by use of spread sheets, domain-specific test
languages, the Testing and Test Control Notation (TTCN3), the UML Testing Profile (UTP), etc.)
 Selection of style guides and guidelines for the definition of high quality tests
 Selection of test case repositories (spreadsheets, databases, files, etc.)
Considerations for the test execution layer:
 Selection of the test execution tool
 Selection of interpretation (by use of a virtual machine) or compilation approach for implementing test
procedures – this choice typically depends on the chosen test execution tool
 Selection of the implementation technology for implementing test procedures (imperative, such as C;
functional, such as Haskell or Erlang; object-oriented, such as C++, C#, Java; scripting, such as Python or Ruby,
or a tool-specific technology) – this choice is typically dependent on the chosen test execution tool
 Selection of helper libraries to ease test execution (e.g., test device libraries, encoding/decoding libraries, etc.)
Neeraj Kumar Singh
TAA Design

Considerations for the test adaptation layer:
 Selection of test interfaces to the SUT
 Selection of tools to stimulate and observe the test interfaces
 Selection of tools to monitor the SUT during test execution
 Selection of tools to trace test execution (e.g., including the timing of the test execution)
Neeraj Kumar Singh
TAA Design

TAA Design
Identify Areas where Abstraction can Deliver Benefits
 Abstraction in a TAA enables technology independence in that the same test suite can be used in different test
environments and on different target technologies. The portability of test artifacts is increased. In addition,
vendor-neutrality is assured which avoids lock-in effects for a TAS. Abstraction also improves maintainability and
adaptability to new or evolving SUT technologies.
 The TAE needs to discuss with the stakeholders in software development, quality assurance, and testing which
level of abstraction to use in which area of the TAS. For example, which interfaces of the test adaptation and/or
test execution layer need to be externalized, formally defined, and kept stable throughout the TAA lifetime? It
also needs to be discussed if an abstract test definition is being used or if the TAA uses a test execution layer
with test scripts only.
 The more abstraction is used for a TAA, the more flexible it is with respect to further evolution or transitioning
to new approaches or technologies. This comes at the cost of larger initial investments (e.g., more complex test
automation architecture and tools, higher skill set requirements, bigger learning curves), which delays the
initial breakeven but can pay off in the long run. It may also lead to lower performance of the TAS.
 While the detailed ROI (Return on Investment) considerations are the responsibility of the TAM, the TAE needs to
provide inputs to the ROI analysis by providing technical evaluations and comparisons of different test
automation architectures and approaches with respect to timing, costs, efforts, and benefits.
Neeraj Kumar Singh

TAA Design
Understand SUT Technologies and Interconnection with the TAS
The access to the test interfaces of the SUT is central to any automated test execution. The access can be available
at the following levels:
 Software level, e.g., SUT and test software are linked together
 API level, e.g., the TAS invokes the functions/operations/methods provided at a (remote) application
programming interface
 Protocol level, e.g., the TAS interacts with the SUT via HTTP, TCP, etc.
 Service level, e.g., the TAS interacts with the SUT services via web services, RESTful services, etc.
In addition, the TAE needs to decide about the paradigm of interaction of the TAA to be used for the interaction
between the TAS and SUT, whenever the TAS and SUT are separated by APIs, protocols or services. These paradigms
include the following:
 Event-driven paradigm, which drives the interaction via events being exchanged on an event bus
 Client-server paradigm, which drives the interaction via service invocation from service requestors to service
provider
 Peer-to-peer paradigm, which drives the interaction via service invocation from either peer
Neeraj Kumar Singh

TAA Design
Understand SUT Environment
An SUT can be standalone software or software that works only in relation to other software(e.g., systems of
systems), hardware (e.g., embedded systems), or environmental components (e.g., cyber-physical systems). A TAS
simulates or emulates the SUT environment as part of an automated test setup.
Examples of test environments and sample uses include the following:
 A computer with both the SUT and the TAS – useful for testing a software application
 Individual networked computers for an SUT and TAS respectively – useful for testing server software
 Additional test devices to stimulate and observe the technical interfaces of an SUT – useful for testing the
software for example on a set-top box
 Networked test devices to emulate the operational environment of the SUT –useful for testing the software of a
network router
 Simulators to simulate the physical environment of the SUT –useful for testing the software of an embedded
control unit
Neeraj Kumar Singh

TAA Design
Time and complexity for testware architecture implementation
While the effort estimation for a TAS project is the responsibility of a TAM, a TAE needs to support a TAM in this by
providing good estimates for the time and complexity of a TAA design. Methods for estimations and examples include
the following:
 Analogy-based estimation such as such as functions points, three-point estimation, wideband delphi, and expert
estimation
 Estimation by use of work breakdown structures such as those found in management software or project
templates
 Parametric estimation such as Constructive Cost Model (COCOMO)
 Size-based estimations such as Function Point Analysis, Story Point Analysis, or Use Case Analysis
 Group estimations such as Planning Poker
Neeraj Kumar Singh

TAA Design
Ease of use for a given testware architecture implementation
In addition to the functionality of the TAS, its compatibility with the SUT, its long-term stability and evolvability, its
effort requirements, and ROI considerations, a TAE has the specific responsibility to address usability issues for a
TAS. This includes, but is not limited to:
 Tester-oriented design
 Ease of use of the TAS
 TAS support for other roles in the software development, quality assurance, and project management
 Effective organization, navigation, and search in/with the TAS
 Useful documentation, manuals, and help text for the TAS
 Practical reporting by and about the TAS
 Iterative designs to address TAS feedback and empirical insights
Neeraj Kumar Singh

TAA Design
Approaches for Automating Test Cases
Test cases need to be translated into sequences of actions which are executed against an SUT. That sequence of
actions can be documented in a test procedure and/or can be implemented in a test script. Besides actions, the
automated test cases should also define test data for the interaction with the SUT and include verification steps to
verify that the expected result was achieved by the SUT. A number of approaches can be used to create the
sequence of actions:
1. The TAE implements test cases directly into automated test scripts. This option is the least recommended as it
lacks abstraction and increases the maintenance load.
2. The TAE designs test procedures, and transforms them into automated test scripts. This option has abstraction
but lacks automation to generate the test scripts.
3. The TAE uses a tool to translate test procedures into automated test scripts. This option combines both
abstraction and automated script generation.
4. The TAE uses a tool that generates automated test procedures and/or translates the test scripts directly from
models. This option has the highest degree of automation.
Well-established approaches for automating test cases include:
 Capture/playback approach, which can be used for option 1
 Structured scripting approach, data-driven approach, and keyword-driven approach, which can be used for
option 2 or 3
 Model-based testing (including the process-driven approach), which can be used for option 4Neeraj Kumar Singh

TAA Design
Capture/Playback Approach
Principal concept
In capture/playback approaches, tools are used to capture interactions with the SUT while performing the sequence
of actions as defined by a test procedure. Inputs are captured; outputs may also be recorded for later checks.
During the replay of events, there are various manual and automated output checking possibilities:
 Manual: the tester has to watch the SUT outputs for anomalies
 Complete: all system outputs that were recorded during capture must be reproduced by the SUT
 Exact: all system outputs that were recorded during capture must be reproduced by the SUT to the level of
detail of the recording
 Checkpoints: only selected system outputs are checked at certain points for specified values
Pros
 The capture/playback approach can be used for SUTs on the GUI and/or API level. Initially, it is easy to setup
and use.
Cons
 Capture/playback scripts are hard to maintain and evolve because the captured SUT execution depends strongly
on the SUT version from which the capture has been taken.
Neeraj Kumar Singh

TAA Design
Linear Scripting
Principal concept
As with all scripting techniques, linear scripting starts with some manual test procedures. Note though that these
may not be written documents–the knowledge about what tests to run and how to run them may be ‘known’ by one
or more Test Analysts. Each test is run manually while the test tool records the sequence of actions and in some
cases captures the visible output from the SUT to the screen. This generally results in one (typically large) script for
each test procedure.
Pros
 The advantages of linear scripts focus on the fact that there is little or no preparation work required before you
can start automating. Once you have learned to use the tool it is simply a matter of recording a manual test and
replaying it.
Cons
 The disadvantages of linear scripts are numerous. The amount of effort required to automate any given test
procedure will be mostly dependent on the size (number of steps or actions) required to perform it.
 Thus, the 1000th test procedure to be automated will take a similarly proportional amount of effort as the
100th test procedure. In other words, there is not much scope for decreasing the cost of building new
automated tests.
Neeraj Kumar Singh

TAA Design
Structured Scripting
Principal Concept
The major difference between the structured scripting technique and the linear scripting technique is the
introduction of a script library. This contains reusable scripts that perform sequences of instructions that are
commonly required across a number of tests. Good examples of such scripts are those that interface, e.g., to the
operations of SUT interfaces.
Pros
 Benefits of this approach include a significant reduction in the maintenance changes required and the reduced
cost of automating new tests.
 The advantages of structured scripting are largely attained through the reuse of scripts. More tests can be
automated without having to create the volume of scripts that a linear scripting approach would require.
Cons
 The initial effort to create the shared scripts can be seen as a disadvantage but this initial investment should
pay big dividends if approached properly.
 Programming skills will be required to create all the scripts as simple recording alone will not be sufficient.
 The script library must be well managed
Neeraj Kumar Singh

TAA Design
Data-Driven Scripting
Principal Concept
The data-driven scripting technique builds on the structured scripting technique. The most significant difference is
how the test inputs are handled. The inputs are extracted from the scripts and put into one or more separate files.
Typically the ‘reusable’ main test script is called a ‘control’ script. The control script contains the sequence of
instructions necessary to perform the tests but reads the input data from a data file.
Pros
 The cost of adding new automated tests can be significantly reduced by this scripting technique.
 This technique is used to automate many variations of a useful test, giving deeper testing in a specific area and
may increase test coverage.
Cons
 The need to manage data files and make sure they are readable by TAS is a disadvantage but can be approached
properly.
 Also, important negative test cases may be missed. Negative tests are a combination of test procedures and test
data.
Neeraj Kumar Singh

TAA Design
Keyword-Driven Scripting
Principal Concept
The keyword-driven scripting technique builds on the data-driven scripting technique. There are two main
differences:
(1) the data files are now called ‘test definition’ files or something similar (e.g., action word files); and
(2) there is only one control script.
Pros
 Once the controlling script and supporting scripts for the keywords have been written, the cost of adding new
automated tests will be much reduced by this scripting technique.
 The benefit of the keyword-driven approach over the data-driven approach in this regard is the use of the
keywords. Each keyword should represent a sequence of detailed actions that produce some meaningful result.
Cons
 Implementing the keywords remains a big task for test automation engineers, particularly if using a tool that
offers no support for this scripting technique.
 Care needs to be taken to ensure that the correct keywords are implemented. Good keywords will be used often
with many different tests whereas poor keywords are likely to be used just once or only a few times.
Neeraj Kumar Singh

TAA Design
Process Driven Scripting
Principal Concept
The process-driven approach builds on the keyword-driven scripting technique with the difference that scenarios –
representing uses cases of the SUT and variants thereof – constitute the scripts which are parameterized with test
data or combined into higher-level test definitions. Such test definitions are easier to cope with as the logical
relation between actions, e.g., ‘check order status’ after ‘place order’ in feature testing or ‘check order status’
without previous ‘place order’ in robustness testing, can be determined.
Pros
 The use of process-like, scenario-based definition of test cases allows the test procedures to be defined from a
workflow perspective.
 The aim of the process-driven approach is to implement these high-level workflows by using test libraries that
represent the detailed test steps.
Cons
 Processes of an SUT may not be easy to comprehend by a Technical Test Analyst – and so is the implementation
of the process-oriented scripts, particularly if no business process logic is supported by the tool.
 Care needs also to be taken to ensure that the correct processes, by use of correct keywords, are implemented.
Good processes will be referenced by other processes and result in many relevant tests whereas poor processes
will not pay off in terms of relevance, error-detection capability, etc.
Neeraj Kumar Singh

TAA Design
Model Based Testing
Principal Concept
Model-based testing refers to the automated generation of test cases (see also the Model-Based Tester Syllabus by
ISTQB) – as opposed to the automated execution of test cases – by use of capture/playback, linear scripting,
structured scripting, data-driven scripting or process-driven scripting. Model-based testing uses (semi-) formal
models which abstract from the scripting technologies of the TAA. Different test generation methods can be used to
derive tests for any of the scripting frameworks discussed before.
Pros
 Model-based testing allows by abstraction to concentrate on the essence of testing (in terms of business logic,
data, scenarios, configurations, etc. to be tested).
 It also allows generating tests for different target systems and targeted technologies, so that the models used
for test generation constitute a future-safe representation of testware which can be reused and maintained as
the technology evolves.
Cons
 Modeling expertise is required to run a model-based testing approach effectively. The task of modeling by
abstracting an SUT’s interfaces, data and/or behavior can be difficult.
 In addition, modeling and model-based testing tools are not yet main stream, but are maturing. Model-based
testing approaches require adjustments in the test processes.
Neeraj Kumar Singh

Contents
3.2 TAA Design
3.2.3 Technical Considerations of SUT
3.3 TAS Development
Neeraj Kumar Singh

TAA Design
Technical Considerations of The SUT
Interfaces of the SUT
 An SUT has internal interfaces (inside the system) and external interfaces (to the system environment and its
users or by exposed components).
 A TAA needs to be able to control and/or observe all those interfaces of the SUT which are potentially affected
by the test procedures (i.e., interfaces need to be testable).
 Test focus (e.g., a test) is needed at the beginning of the project during architecture definition to verify the
availability of the necessary test interfaces or test facilities required for the SUT to be testable.
SUT Data
 An SUT uses configuration data to control its instantiation, configuration, administration, etc. Furthermore, it
uses user data which it processes.
 An SUT also may use external data from other systems to complete its tasks. Depending on the test procedures
for an SUT, all these types of data need to be definable, configurable and capable of instantiation by the TAA.
 The specific way of coping with the SUT data is decided in the TAA design. Depending on the approach, data may
be handled as parameters, test data sheets, test databases, real data, etc.
Neeraj Kumar Singh

TAA Design
SUT Configurations
 An SUT may be deployed in different configurations, for example on different operating systems, on different
target devices, or with different language settings.
 Depending on the test procedures, different SUT configurations may have to be addressed by the TAA. The test
procedures may require different test setups (in a lab) or virtual test setups (in the cloud) of the TAA in
combination with a given SUT configuration.
 It may also require adding simulators and/or emulators of selected SUT components for selected SUT aspects.
SUT Standards and Legal Settings
 In addition to the technical aspects of an SUT, the TAA design may need to respect legal and/or standards
requirements so as to design the TAA in a compatible manner.
 Examples include privacy requirements for the test data or confidentiality requirements that impact the logging
and reporting capabilities of the TAA.
Neeraj Kumar Singh

TAA Design
Tools and tool environments used to develop the SUT
 Along with the development of an SUT, different tools may be used for the requirements engineering, design and
modeling, coding, integration and deployment of the SUT.
 The TAA together with its own tools should take the SUT tool landscape into account in order to enable tool
compatibility, traceability and/or reuse of artifacts.
Test interfaces in the software product
 It is strongly recommended not to remove all the test interfaces prior to the product release. In most cases,
these interfaces can be left in the SUT without causing issues with the final product.
 When left in place, the interfaces can be used by service and support engineers for problem diagnosis as well as
for testing maintenance releases. It is important to verify that the interfaces will pose no security risks. If
necessary, developers usually can disable these test interfaces such that they cannot be used outside the
development department.
Neeraj Kumar Singh

Contents
3.2 TAA Design
3.2.4 Considerations for Development/QA Process
3.3 TAS Development
Neeraj Kumar Singh

TAA Design
Considerations for Development/QA Process
The aspects of the development and quality assurance processes of an SUT should be considered when designing a
TAA. Some of these are discussed below although this is not a complete list but should serve as a sample of the
important aspects.
Test execution control requirements
 Depending on the level of automation required by the TAA, interactive test execution, batch mode test
execution or fully automated test execution may need to be supported by the TAA.
Reporting requirements
 Depending on the reporting requirements including types of reports and their structures, the TAA needs to be
able to support fixed, parameterized or defined test reports in different formats and layouts.
Role and access rights
 Depending on the security requirements, the TAA may be required to provide a role and access rights system.
Established tool landscape
 SUT project management, test management, code and test repository, defect tracking, incident management,
risk analysis, etc., may all be supported by tools composing the established tool landscape. The TAA is also
supported by a tool or tool set which needs to seamlessly integrate with the other tools in the landscape..
Neeraj Kumar Singh

Contents
3.2 TAA Design
3.3 TAS Development
3.3.1 Introduction to TAS Development
Neeraj Kumar Singh

TAS Development
Introduction to TAS Development
 Development of a TAS is comparable to other software development projects. It can follow the same procedures
and processes including peer reviews by developers and testers.
 Specific to a TAS are its compatibility and synchronization with the SUT. These require consideration in the TAA
design and in the TAS development. Also, the SUT is impacted by the test strategy, e.g., having to make test
interfaces available to the TAS.
 This section uses the software development lifecycle (SDLC) for explaining the TAS development process and the
process-related aspects of compatibility and synchronization to the SUT.
 These aspects are likewise important for any other development process that has been chosen or is in place for
the SUT and/or TAS development – they need to be adapted accordingly.
Neeraj Kumar Singh

TAS Development
Introduction to TAS Development
 The set of requirements for a TAS needs to be analyzed and collected.
 The requirements guide the design of the TAS as defined by its TAA. The design is
turned into software by software engineering approaches. Please note that a TAS may
also use dedicated test device hardware, which is outside of consideration for this
syllabus.
 Like any other software, a TAS needs to be tested. This is typically done by basic
capability tests for the TAS which are followed by an interplay between the TAS and
SUT.
 After deployment and use of a TAS, often a TAS evolution is needed to add more test
capability, change tests or to update the TAS to match the changing SUT. The TAS
evolution requires a new round of TAS development according to the SDLC.
 Please also note that the SDLC does not show the backup, archiving and teardown of
a TAS. As with the TAS development, these procedures should follow established
methods in an organization.
Neeraj Kumar Singh

Contents
3.2 TAA Design
3.3 TAS Development
3.3.2 Compatibility Between TAS and SUT
Neeraj Kumar Singh

TAS Development
Compatibility Between TAS and SUT
Process compatibility
Testing of an SUT should be synchronized with its development – and, in the case of test automation, synchronized
with the TAS development. Therefore, it is advantageous to coordinate the processes for SUT development, TAS
development and for testing. A large gain can be achieved when the SUT and TAS development are compatible in
terms of process structure, process management and tool support.
Team compatibility
Team compatibility is another aspect of compatibility between TAS and SUT development. If a compatible mindset is
used to approach and manage the TAS and the SUT development, both teams will benefit by reviewing each other’s
requirements, designs and/or development artifacts, by discussing issues, and by finding compatible solutions. Team
compatibility also helps in the communication and interaction with each other.
Neeraj Kumar Singh

Contents
3.2 TAA Design
3.3 TAS Development
3.3.3 Synchronization between TAS and SUT
Neeraj Kumar Singh

TAS Development
Synchronization between TAS and SUT
Synchronization of requirements
After requirements elicitation, both SUT and TAS requirements are to be developed. TAS requirements can be
grouped into two main groups of requirements: (1) requirements that address the development of the TAS as a
software-based system, such as requirements for the TAS features for test design, test specification, test result
analysis, etc. and (2) requirements that address the testing of the SUT by means of the TAS. These so called testing
requirements correspond to the SUT requirements and reflect all those SUT features and properties which are to be
tested by the TAS. Whenever the SUT or the TAS requirements are updated, it is important to verify the consistency
between the two and to check that all SUT requirements that are to be tested by the TAS have defined test
requirements.
Synchronization of development phases
In order to have the TAS ready when needed for testing the SUT, the development phases need to be coordinated. It
is most efficient when the SUT and TAS requirements, designs, specifications, and implementations are
synchronized.
Synchronization of defect tracking
Defects can relate to the SUT, to the TAS or to the requirements/designs/specifications. Because of the relationship
between the two projects, whenever a defect is corrected within one, the corrective action may impact the other.
Defect tracking and confirmation testing have to address both the TAS and the SUT.
Neeraj Kumar Singh

TAS Development
Synchronization of SUT and TAS evolution
Both the SUT and the TAS can evolve to accommodate new features or disable features, to correct defects, or to
address changes in their environment (including changes to the SUT and TAS respectively as one is an environment
component for the other). Any change applied to an SUT or to a TAS may impact the other so the management of
these changes should address both the SUT and TAS.
Two synchronization approaches between the SUT and TAS development processes are depicted in Figures below.
Neeraj Kumar Singh
Figure shows an approach where the two SDLC processes
for the SUT and the TAS are mainly synchronized in two
phases:
(1) the TAS analysis is based on the SUT design, which
itself is based on the SUT analysis and
(2) the testing of the SUT makes use of the deployed TAS.

TAS Development
Figure shows a hybrid approach with both manual and automated
testing. Whenever manual tests are used before the tests are
automated or whenever manual and automated tests are used
together, the TAS analysis should be based both on the SUT design
and the manual tests. In this way, the TAS is synchronized with
both.
The second major synchronization point for such an approach is as
before: the SUT testing requires deployed tests, which in the case
of manual tests could just be the manual test procedures to be
followed.
Neeraj Kumar Singh

Contents
3.2 TAA Design
3.3 TAS Development
3.3.4 Building Reuse into the TAS
Neeraj Kumar Singh

TAS Development
Building Reuse into the TAS
Reuse of a TAS refers to the reuse of TAS artifacts (from any level of its architecture) across product lines, product
frameworks, product domains and/or project families.
Requirements for reuse result from the relevance of TAS artifacts for the other product variants, products and/or
projects. Reusable TAS artifacts can include:
 (Parts of) test models of test goals, test scenarios, test components or test data
 (Parts of) test cases, test data, test procedures or test libraries themselves
 The test engine and/or test report framework
 The adaptors to the SUT components and/or interfaces
While reuse aspects are already settled when the TAA is defined, the TAS can help increase the ability for reuse by:
 Following the TAA or revising and updating it whenever needed
 Documenting the TAS artifacts so that they are easily understood and can be incorporated into new contexts
 Ensuring the correctness of any TAS artifact so that the usage in new contexts is supported by its high quality
It is important to note that while design for reuse is mainly a matter for the TAA, the maintenance and
improvements for reuse are a concern throughout the TAS lifecycle.
Neeraj Kumar Singh

Contents
3.2 TAA Design
3.3 TAS Development
3.3.5 Support for a Variety of Target Systems
Neeraj Kumar Singh

TAS Development
Support for a Variety of Target Systems
TAS support for a variety of target systems refers to the ability of a TAS to test different configurations of a software
product. Different configurations refer to any of the following:
 Number and interconnection of SUT components
 Environments (both software and hardware) on which the SUT components run
 Technologies, programming languages or operating systems used to implement the SUT components
 Libraries and packages the SUT components are using
 Tools used to implement the SUT components
While the first four aspects impact the TAS on any test level, the last one applies mainly to component-level and
integration-level testing.
The ability of a TAS to test different software product configurations is determined when the TAA is defined.
However, the TAS has to implement the ability to handle the technical variance and has to enable the management
of the TAS features and components needed for different configurations of a software product.
Neeraj Kumar Singh

TAS Development
Support for a Variety of Target Systems
The handling of the TAS variety in relation to the variety of the software product can be dealt with differently:
 Version/configuration management for the TAS and SUT can be used to provide the respective versions and
configurations of the TAS and SUT that fit to each other
 TAS parameterization can be used to adjust a TAS to an SUT configuration
It is important to note that while design for TAS variability is mainly a matter for the TAA, the maintenance of and
improvements for variability are a concern throughout the TAS life cycle. It requires continuous consideration and
efforts to revise, add and even remove options and forms of variability.
Neeraj Kumar Singh

Sample Questions
1. Why is it important for the TAA to separate the test definition from the test execution?
Answer Set
a. Separating the two increases the speed of execution
b. The test definition can be completed without knowledge of the tool that will be used for execution
c. The execution will be able to add test cases to the test definition as needed during execution
d. The test definition layer will supply the adaptation necessary for the tests to run with various tools
and interfaces
Neeraj Kumar Singh

Sample Questions
2. You are working on a test automation project that will automate business scenarios for the user
acceptance testers to use when conducting UAT. The business scenarios are well-defined and
frequently repeated during UAT. The goal is to also be able to test these same scenarios as part of
regression testing by executing the test automation.
Structured scripting has already been used to develop the library of functions that can be used by
the test automation. What scripting technique should be used to build upon the structured scripting
that has already been done and to implement this test automation to meet the business goals?
Answer Set
a. Scenario-driven scripting
b. Keyword-driven scripting
c. Process-driven scripting
d. Linear Scripting
Neeraj Kumar Singh

Sample Questions
3. Which of the following is true about reuse?
Answer Set
a. Reuse is built into the TAA and maintained and improved in the TAS
b. Reuse is built in both the TAA and TAS and maintained in the gTAA
c. Reuse is only applicable for the gTAA
d. Reuse is built into the TAS and maintained and improved in the TAA
Neeraj Kumar Singh

Chapter 3 - The Generic Test Automation Architecture

More Related Content

What's hot (20)

Similar to Chapter 3 - The Generic Test Automation Architecture (20)

More from Neeraj Kumar Singh (20)

Recently uploaded (20)

Chapter 3 - The Generic Test Automation Architecture

Editor's Notes