Automated Integration Regression Testing

Automated
Integration
Regression Testing
Michal Oravec

2
 Testing Layers
 Test Projects
 Jenkins/Docker Runs
 Typical Test
 Challenges
Content

3
Testing Layers
 Gherkin meta-language (behave Python library)
 Python code using (Germanium API Python library)
 Germanium itself handling browsers (includes Selenium
Python library and web-drivers)

4
Testing Layers
‒ Basically almost English free text test case executable by human.
‒ Simple text files are used called features. The feature can contain one
or more scenarios (test cases) and each scenario may be executed one
or more times if there are more test data sets available.
‒ The scenarios are self-contained integration “tests”.
‒ behave: http://pythonhosted.org/behave/
Single test step (sentence)
Single Python
function gets
called

5
Testing Layers
‒ We tend to have one scenario per one feature (best practice).
‒ We store the feature files in repository along the Python testing code.
‒ The sentences (test steps) are read by a Python library called “behave”
that links them with their implementation and executes them one by
one.

6
Testing Layers
‒ There is very little structure – feature name and scenario name +
keywords Given / When / Then.
‒ “When” marks action and “Then” marks assertion (not necessary – just
best practice)

7
Testing Layers
‒ Note convention for providing data used for reusing steps (=
sentences) in more tests.
‒ Same convention enables the same test to run with multiple data sets
– note part “Examples”. Each line represents a test run.

8
Testing Layers
‒ Each sentence (test step) is mapped to one Python function.
‒ These may use other functions, mappings of elements (selectors of
buttons, dialogs, parts of text…) and generally anything doable in
Python and/or Germanium API.
‒ This layer is simply code in Python.
‒ We use and depend fully on the API
https://germaniumhq.com/documentation/

9
Testing Layers
‒ Example:

10
Testing Layers
 Germanium itself handling browsers
(includes/integrates Selenium Python library and
web-drivers)
‒ Basically that’s it because we do not directly use Selenium.
‒ We write / customize stuff only for Gherkin/behave and
Germanium/Python, this layer just does the job in background if
all goes well.

12
 Example test projects
for our apps called
Workplace and
Administrator
Test Projects
12

14
Jenkins/Docker Runs
‒ Jenkins part of running tests is more technical and/or complicated.
‒ In our case there are two Jenkins jobs that run the whole tests suite
for the two tested apps against whatever is currently deployed on test
machine.
‒ These jobs are triggered at certain time generally every night
regardless of other jobs/builds/deployments.
‒ Typically this time is when we expect or suppose that the build and
deployment to test machine was already done.

15
Jenkins/Docker Runs
‒ The how/where of the test run is complicated/technical. Somehow a
virtual environment (Docker) is created with the specified browser, all
the tests are fetched from repository and executed against .90 test
server running the recently built and deployed web-application.
‒ There is very little configuration in the Jenkins GUI – most is hidden in
the linked Jenkinsfile stored in the repository

16
Jenkins/Docker Runs
‒ More necessary details how to create the environment from which the
tests will be run are hidden inside the “Dockerfile” (in repository):

17
Jenkins/Docker Runs
‒ The output (log) is available in simple text form (very messy, not
structured) in the Jenkins job output to potentially analyze by a
person.
‒ One way to analyze failures is to rerun failed tests locally (from Idea)
and/or manually (by hand).

19
Typical Test
‒ The test cases are generally rather short and simple.
‒ Generally it is a series of actions (click link/button) and assertions
(expect certain text is on the page / in a dialog or expect certain page
to load or dialog to show up)
‒ Example for our Workplace application: Login -> check status and
version if it is even reasonable to continue -> run Business Case ->
check if Human Task opened correctly and if it has all the expected
content -> continue to Debug Human Task and check content ->
finish Business Case -> Logout.
‒ Challenge: It is difficult, error-prone and time consuming to check that
certain actions cause a new page to load with a random content when
it takes a random relatively long time.

20
Typical Test
‒ Having short and simple and static TCs seems to be best practice -
one would argue that this is good / sufficient approach.
‒ Best practice: One should expect fixed / static texts and values and
never compute expected values (never replicate features of the tested
app).
‒ Automated integration regression testing should be like a sanity check
that really basic feature still work and that stuff generally opens and
loads as expected every day after nightly deployment.
‒ You don’t do it to find bugs – more like to:
 avoid testing the same basic stuff every day
 have some confidence in the morning that generally stuff works
 free real people to do more investigative testing

22
Challenges
1. Maintenance.
2. Random technical failures in Jenkins/Docker run not replicable
locally.
3. Jenkins run output is not structured – it is very difficult to analyze
and offers almost no help in determining which tests failed and why.
4. Running the same tests on other browsers (Firefox, recent IE)
5. Marking tests that fail because known real failures - bugs.
6. Found bugs are ignored long-term – ”fix” the tests?
7. Tests are dependent on constant state of “public” and constantly
changing environment (application settings, content/resource
storage)
8. Tests too slow. Faster execution causes random failures.
9. Difficult/complicated to configure Jenkins job/run to execute only a
subset of all tests in a project.

23
Conclusion
‒ Integration level of automated regression testing in web-frontend
area is useful but benefits depend on many factors
‒ It is always an open and valid question how to automate, how much
to automate, who should automate and also if to automate at all
Thank you for your attention.

Automated Integration Regression Testing

More Related Content

What's hot (20)

Similar to Automated Integration Regression Testing (20)

Recently uploaded (20)

Automated Integration Regression Testing