Test Automation Pyramid

Editor's Notes

• #2: My name is Alexander, and I’m usually one floor above, working on Quality Assurance stuff, although I am here quite often as well I want to give an introduction to the test automation pyramid, a concept which I think it is useful for everyone involved in the development process to know about, so hopefully we have many different roles in the audience Im going to start off with some observations about test automation in general
• #3: When you boil it all down, we usually do test automation because it allows us to test more with less effort You invest some time into designing, implementing and maintaining your automated tests, and from that investment you get certain benefits: Don’t need to spend time and effort to do manual testing, at least not regression testing We can test things that are not possible or practical to do manually In some ways, automated tests are more accurate and more reliable Run tests all the time, continuous verification and quick feedback on that your service works as expect while we are constantly changing it When more and more of your test effort is automated, you can release in a shorter amount of time, and also more frequently which should be a goal in itself Result hopefully, is higher software quality Test automation should be done as a natural part of the development process, and not as an afterthought or something to be done when you have time If you do that, you will accumulate technical debt, that you will suffer for later A story is not done until appropriate automated tests are passing What is appropriate? That depends on many different factors, which is why it should be discussed in the team prior to implementation of a user story In that discussion you can use the test automation pyramid
• #4: Shows different technical levels on which you can have automated tests: Unit Tests, Component Integration Tests, ... An important thing to mention is that this is a model. It is a thinking tool, something to be used as a reference, and it is imperfect by definition. You can create a pyramid like this in many different ways, and I will show you some examples of that later. As you move up in the pyramid, the scope of the tests increase, and their execution time, and the effort needed to create/run/maintain them Model argues that you should probably have more unit tests than component integration tests, more component integration tests than API tests, more UI tests than API tests, and so on. We’ll get into why as we go through the layers.
• #5: With unit tests you want to test the smallest possible parts of the system in isolation To achieve that you replace dependencies with test doubles Dummies, stubs, spies, fakes, mocks An example for those who are not developers: If you’re creating a unit test for unit A, and unit A depends on unit B. In your unit test, you replace unit B with a test double that you create and control. So your unit test for unit A is not relying on unit B at all. Because we have replaced the dependencies that to unit B, our unit test has a very small scope. If it fails, we know exactly where the problem is - in unit A. The small scope is also why unit tests have the best performance in the pyramid. When you replace all the dependencies, so you are not writing to hard drive, talking to a database or integrating with an external system, and so on But even with a great unit test suite, you need another line of defense
• #6: We already know that our units work as we expect them to, in isolation, now we want to make sure that they work in combination Which is why we don’t use test doubles, at least not for the integrations that we are interesting in testing So in my previous example, we would test unit A and B together, without using a test double You can have small component integration test like that, testing only two components, or you have bigger component integration tests, involving a bigger part of the system So the scope of these tests vary, but in general the performance is still pretty good, even though unit tests are faster
• #7: Most services expose a lot of important functionality through APIs As an example, you might have a REST API that is used by your front-end, used by your mobile app, and used by external systems integrating with your system Now, you could consider API tests as being very large component integration tests, because the API relies on many different components on lower levels But when you are testing APIs, you have a different focus You are testing the system from more of a business and user perspective, thinking about how your API will be used You want to guarantee that your API can be used in a certain way, and that it behaves as it should Generally, API tests have a large scope, but they are still pretty fast, at least much faster than UI Tests If you know that your UI relies on a solid API, you don’t need a huge number of UI tests, which brings us to the next level
• #8: Here I want to stress that I mean using the UI to test the application end-to-end, possibly with integrations to other systems. We are not about unit tests or component integration tests in JavaScript - that would be the bottom layer. By definition, these end-to-end tests have the largest scope compared to the other layers, which is Good because they can catch many different types of problems But you need to spend a little bit of time to figure out where the error actually is The large scope is also one reason for UI tests having the worst performance As an example, let's say you are using Selenium WebDriver which allows you to test web applications by automating the browser. In that case you automatically start a browser, download resources, execute JavaScript, the browser makes multiple calls to your REST API, then the backend makes multiple calls to the database, and possibly to other systems But to improve the performance of your UI tests you can Prepare or clean up test data with other means that perform better Run the tests in parallel, but doing that can be a big challenge in itself (tests, application, test infrastructure) Brittle, flaky and a lot of work to maintain: Because the scope of the test is so large. Brittle, meaning that they fail easily. Many things can impact the test, even though you didn’t intend to. Flakey, meaning that they fail inconsistently, for instance because you have a bigger test infrastructure involved that could randomly crash Rant: A flakey test is worse than having no test at all. If a test can fail when nothing is wrong, you will stop taking that test seriously. You will also waste a lot of time, running and re-running the test, investigating the results and so on. If you have flakey tests, address the root cause, and if you can't, delete the test, or switch to another technology. To get back on track: To avoid having too many UI tests you want to identify the most important user workflows in your application, and create tests that emulate a user using going through those workflows In general: Very valuable, but costly (for the same reason)
• #9: Not a part of the pyramid itself, because it is manual testing but not regression testing Combine experience with the system, and experience as a tester, with creativity to explore the system, looking at the system from new angles, discovering new defects, and using what you learn to improve your automated tests Often, exploratory testing is based on a mission, test charter or persona <some examples> Example: Let’s say you act as this impatient manager, not filling out all the input fields you should and double and triple clicking all the buttons, and you discover a problem when clicking the Save button repeatedly Then you can investigate that, see if you have similar problems elsewhere and if you should have automated tests for catching these problems.
• #10: So we have covered all the layers, but I am sure someone has been thinking where is this and that form of testing? The pyramid concerns itself with different technical levels of testing, while these examples here are either different motivations for testing, or certain properties you want to test for, and can be done on all or several of these technical levels
• #12: I wanted to illustrate that there is no silver bullet. You can’t use one single tool for test automation. If you do, you might get this →
• #13: This is what you don’t want. Ice cream cone: Mostly manual testing, started with Selenium WebDriver. Too many UI tests, so you start with SoapUI to create API tests. Completely opposite of what you want. Hour-glass: Developers write unit tests. Other people who write UI tests. No cooperation. Something to be cognizant of, and to avoid.
• #14: Here are the test automation pyramids I promised you earlier. As you can see, there are many ways of creating a test automation pyramid. Maybe your system doesn’t have a UI. Or maybe you have some good arguments for why you should have more API tests than component integration tests, which then of course would be fine. It’s not really possible to have one ideal test automation pyramid for everyone, which is why there is a special Testing Week task for this
• #15: Go to this page on Confluence. Use my example, or create your own pyramid. Make sure you have a common understanding within your team. Try figure how your test effort is distributed today, and how you can improve
• #16: I want to talk about where the test automation pyramid is relevant in the development process Example workflow from someone having an idea until the idea has been implemented and deployed to Production QP: Where the team builds quality into the product DP: Where you verify quality QP: Quality increases, maximum quality when you push to mainline DP: Confidence increases, maximum confidence when you deploy to Production
• #17: When the PO or BA is designing a user story they might draft some acceptance criteria - when A, B, and C are true, I consider the story to be fully implemented
• #18: Then, during refinement/grooming/breakout, different roles collaborate closely, sometimes referred to as the 3 amigos Of course you can have more than 3 people, but all roles should be represented. So you might have a PO/BA, the person with the vision for the user story, a developer, and a QA/tester They make sure they have a common understanding of the story, they finalize the acceptance criteria, they finalize the acceptance tests, and they discuss risks and how to mitigate them In that discussion they plan what should be tested, on which level, and the concept of the pyramid can be very useful
• #19: If you have done a good job in the refinement phase, developers should have a pretty clear picture of what is expected both in terms of functionality and automated tests Automated tests are implemented as part of the user story
• #20: Assuming we have a good process for creating automated tests, where and when do we execute the tests? Typically unit tests and component integration tests are executed by developers during implementation, and before they push their changes to the mainline Then, you run the same tests on a CI server, in case someone forgot to run them locally, ‘ to make sure the tests pass on other machines as well, and as the last line of defence before you deploy that new change to the first test environment But then, which tests do you run in the different environments In this example we have 3 test environments, and they have different objectives In the internal test environment, all integrations are mocked, so our system is completely isolated. Because all the integrations are mocked, we are able to run all our API tests and our UI tests, verifying that this version works as expected and won’t cause problems when we promote it to the User Acceptance environment which is where we have real integrations In the User Acceptance environment can run the API and UI tests again, to make sure the system still works with but now with real integrations In the Production environment you probably only run a smoke test Exploratory testing In all environments Developers can do exploratory testing during/after implementation in their development environment QA/testers, or PO/BA can do it in any test environment. You can put a manual step anywhere in this automated delivery pipeline, so it is really up to you to see what fits you Some people are even doing exploratory testing in Production, using feature toggles to turn on a new feature only for internal users who do exploratory testing in Production
• #21: A quick summary, to wrap up. The test automation pyramid is a useful thinking tool and discussion reference Make conscious decisions about what to test on which level, and also where you run which tests Try to distribute your tests optimally High quality High velocity Medium effort Discuss with your team Do the Testing Week task That’s it for me, do you have any questions or comments?
• #22: REPEAT THE QUESTION