Avoiding test hell

Editor's Notes

• #3: I don’t have a silver bullet for everything The advice I am to give is more practical rather than theoretical. Its easy to talk about ideal world solutions but unfortunately, the projects I worked on weren’t perfect. We had things called deadlines and constraints on budgets which meant we had to make compromises.
• #6: A Junit could be written as what you see here. The developer probably only had a design document on how the class should behave and what methods they had to write. So then they would write a test like that. So what would happen if this junit would fail? What would you know if you looked at a junit report that said SelectPrice failed? Problems – if the test failed, how do you have to spend time investigating what particular functionality failed. You would have to go into the test and see what it really tested. Whether you absolutely have to fix it depends on which requirement it reflects. How its not so easy to determine which requirement it is for. The wording does not reflect the requirements or the desired behaviour of the system So BDD attempts to address this focusing on the behaviour of the system. We went from having requirements the system will do something to something like As a user, I want to be able to find out the highest price for a security so that I charge the most for my portfolio. The tests then we organised in the given some precondition, when some action is called, then I expect something to happen and the method name is something a bit more meaningful so that if you looked at a junit report, when it fails, I know have a better indicator what failed.
• #7: FitNesse is an Acceptance Testing Framework with a wiki as the front end, some fixtures or glue code that invokes your system / functions or what we call the System Under Test. There are a couple of other good testing frameworks such as Cucumber, Jbehave, Concordion and so on. We choose FitNesse because it was the easiest for our BAs to use. But I have to admit, some of the other tools might be closer to BDD than FitNesse. Given some prices, when we compare prices and generate the report, then we expect no differences with a predefined report.
• #8: Your tests are there to save you money, but they aren’t free. Tests come at a cost. If the cost is greater than what they save you, are they worth it? It costs you money to write tests, maintain them, debug and fix them. Even looking at them costs you money. What if your client whats a piece of functionality in production next week? Can you do it? What about tomorrow? What can you confidently say you’ve tested everything? Are your tests holding you back? If you have a test that fails, do you believe it has failed immediately or do you rerun it first? That rerun is wasted effort. If a test passes, do you believe that functionality works or do you have a poke around at the GUI first? Then there is the people side of things. You ask someone to investigate a test failure, do they groan? Can a new joiner into the team start investigating and writing their own tests relatively quickly?
• #9: Is it cheaper to fix the problem before or after its released into production? What if you hold up production? Costs of breaking contractual agreements. You should also ask why didn’t your tests pick up the problem earlier. If a bug is found so late in the cycle, will it costs more to fix before releasing to production due to costs? Time to fix it? How long to retest? If you have not got a release process clearly defined, then it will take you a while to release from DEV to SIT and then UAT. Are the creation of your release packages for different test environments standardized? You need to ensure that what you deploy to each environment, from your build servers to SIT and UAT is the same. So you need to ensure everything is scripted and theres no manual copying of files anywhere. There is a whole lot of things you can do around DevOps to help you here but is beyond the topic today. The project I’m on used to take 2 to 4 hours to deploy to an environment. Anything could go wrong, locking out accounts to copying the wrong set of files and manual copying and unzipping files. We were copying files across continents – from UK to Hong Kong. Now we have it all automated to deploy to 2 Linux servers and 4 Window servers across Hong Kong in under 5 minutes.
• #10: One indicator that your tests are costing you is what do you after they’ve failed. Do you trust your reporting well enough to verify it’s a real break? If it is a real break, how long does it take you to get started to look into it? Are there sufficient debug statements and logs? We will see some more examples of things that take up time investigating broken tests And then, was the test broken because of some environment issue? Is it connecting to a database or MQ which happened to be down? Then the test is just creating noise as you don’t know whether the actual function being tested broken or not. A lot of computation heavy methods take time. Did you introduce a wait or delay into your test? What if one day, the environment is a little bit slower than normal? Most of the time, you don’t plan for tests to be ran in parallel. Whoever wrote the first couple of Junits in the system just wrote tests. They probably didn’t think about running tests in parallel or other performance issues. So its possible that some data might be reused across tests. If you’re running tests in parallel, would you have the problem of over writing each others data?
• #11: So now lets look at examples of what I just talked about. From the test, how long does it take for you to just look at where the failures are? If its hidden in a file, you have to go the file system and check the reports. In Fitnesse, you can reuse pages so you can have a page with just the test data so that if the data needs to change, you can just change that one test page.
• #12: Discuss the use of valuation_date and the wait for 1 minute What if other pages also use 1st march? How are you going to run them in parallel? Are you going to setup VMs? Separate folders per process? The simplest thing you could at least do is agree with others on what date you will use so that you don’t over lap. Changes are, that’s going to be a real quick way to solve some of your issues. The wait for 1 minute, what if the compare prices takes longer than 1 minute? What are things you can try? 90 seconds? 2 minutes? What if the operation really did fail? Then you have to wait 2 minutes. What if you poll? Say 3 seconds for 30 rounds? If it passes, then you’ve sped up the test. But what if it fails? Then you are still waiting a long time. Can you change to event driven architecture? That is most ideal solution, but probably not feasible.
• #13: Continuing from the previous example, most of the time, large processes and engines are often put into threads which makes it asynchronous. How many times do you need to test the actual engine or can you extract the logic inside the thread. In the ideal situation, you will have unit tests covering each function and perhaps each class and then a small number of FitNesse tests that represent integration tests.
• #14: How well do you trust the coverage %? What does it mean? If you only cover unit tests, then you are missing out on the integration. Still need ways to unsure that testing covers everything? Can have Excel or tools like QC that people put in test cases. If an automatic test suite passes, is that sufficient or do you still have a whole lot of manual testing to go?
• #15: If this is a problem, then you have a problem. Is it easy for people to add new tests? Often the first thing most people do is look for a working example and then copy and paste it and then change the values they need. But how do you know whether they are copying a test that is up to date with the methods being called? What usually happens is that people write some tests that passes. Then some people add some new utilities or things that make the tests work better but the old tests don’t get updated. You need to come up with a system that ensures people are copying the most up to date tests. I have not got a silver bullet for this issue. In an ideal world, all tests are up to date. But in reality, we have code reviews and owners for test suites who know which test are up to date. Not only do we need to keep the code up to date, the documentation needs to be there and kept up to date. I always follow the DRY – Don’t Repeat Yourself principle. Ideally everyone will remove duplication but in reality, it doesn’t always happen. You always build up on debt.
• #16: The solution is easy – invest time to clean it up. Lets say you package the structure a little better and created a report package and put the report related classes inside. Then inside the documentation for those classes, you don’t need to explicitly say those are reporting classes. Simply by being in the package, you know it’s a reporting related class. You’ve made life easier for yourself. And then same if you refactor methods as well. You’ve reduced the number of things that need documentation.
• #17: Mocking allows you to simulate behaviour. Rarely does your class work alone (unless it’s a really bad class) or it’s a toy. But when you are testing just one class or component, you really need to be able to simulate it communicating with other classes and interfaces. You don’t want to connect to the database or file system everytime you test because its slow and you are introducing more variables into the test environment. And you also need to be able to repeat the behaviour over and over again and make sure everytime the inputs into the test are the same. But Mocking itself is not the answer to everything. Frameworks can easily be abused and you will end up with a mess. Often testing frameworks require you to mock out each method call and potentially there could be nested mock objects.
• #18: First example – create the objects required for testing inside the test. You can see it can get quite complicated, what if your objects have dependencies you aren’t testing? Your tests can get quite long, full of code that isn’t actually needed for the test but required for the setup Second example – an object mother – huge improvement by using factories. Hides all the details. But if the test fails, you have to dig into the object mother to see what the object looks like (wasting time on figuring out what is wrong). What if you have a slightly different object? Then you need a new method Third example – test object builders. The test is more verbose but more elegant. You can hide dependencies that you don’t care about inside the .with() methods or upon initialisation of the builder object, set a whole lot of default fields.
• #19: (skip this slide if not enough time) Chances are the vendor software is doing something complex, if it was simple, you may not have purchased it. And therefore it will have complex data requirements and messages flowing in and out of the system. If you are programming in a language that compiles like Java, then using the Vendor’s API will give you some clue that changed in a version upgrade.
• #20: Everything I have talked about so far are a symptom, not the cause of why you are having trouble with your tests. The root cause in my opinion is the people. All the software packages like FitNesse and Mocking and theory like test boundaries I have talked about ultimately get used by people. And it is how your people choose to use these things that ultimately get you into a mess. What if your people don’t believe in testing? Have they seen the benefits? Or are they just feeling the pain of cleaning up a messy test suite? If the benefit is all at the management level, why would you expect at the individual contributors to fully embrace testing? Gamification – could back fire if moral is low. Example – 2 teams, manager publishes some statistics that he wants to measure and introduces a little competition against each other. But if moral is low, or the incentives are low, then no one cares. Everyone will want to beat the game. Eg ratio of emergency releases into production to releases is a low percentage. I don’t have a solution for people problems – but are you even aware of it?