Moving from Ad Hoc Testing to Continuous Test Data with FitNesse

Editor's Notes

• #2: This is an experience story about automating the creation of test data by making use of FitNesse. It was done in a project for the e-commerce operation of the largest supermarket chain in the Netherlands; Albert Heijn Online.
• #6: This is the current website. However, from my perspective it is not much more than a collection of buttons, text and images.
• #7: This is the distribution centre in Almere that was built in 2012 to serve, among other places, the city of Amsterdam. It measures about 13.000 m2. Most of the things that are physically in this building have their representation in our database: the batch pickers, the products, the shelves, the loading bays, the carts, the crates etc…. All of the processes that are used in this building have their representation in our database.
• #8: E-commerce back-end: handling everything: placement of an order allocating delivery slots scheduling delivery runs maintaining stock generating financial data. track employee productivity replenishment operational data Business logic is scattered all over the code. It is like a huge glass vase that your drop on a tile floor and it breaks into a million pieces. Oracle databases, Oracle forms order influences reports production forecast replenishment payment
• #11: Wen I started in the project I compared the system to a desert. No data was flowing through the system. As a result it was impossible to observe system behavior, since system behavior can only be observed when the system is processing data. I am not writing tests. I am writing scripts that generate data and push data through the system.
• #12: What do I mean by ad hoc testing? A test that is not scheduled is not a test! Code was scattered all across FitNesse scripts. When you need to do refactoring FitNesse is not the best place to do it. Development tools: Eclipse, version control (subversion)
• #13: I introduced the use of Eclipse, the use of JUnit, the use of version control, automatic build and deployment of the code.
• #17: Much of what FitNesse is able to do depends on what is done in the Java layer. I investigated GUI automation using OpenScript, but it seems that this solution would pose serious diffficulties.
• #18: Gradually a building block evolved to which I supplied a functional description and a list of input and output parameters. This made it easier to connect with this building block and use it in a script. The purpose of making building blocks is to keep refactoring in mind. So that if the Java code changes there is a very limited (and easily traceable number of places) where the scripts have to change.
• #19: This is an example of a building block. As you can see, I added a functional description.
• #20: Make it impossible to use data that is not configured. Search for the use of data with ‘Where used’ in FitNesse. Data configuration is a major part in the testing effort. If the data is incorrect, somewhere along the line the results that you expect may not be there.
• #21: Symbolic links allow you to run the same group of tests against different environments. Tags are very handy because you can build large test suites and run only selection of those based on tags.
• #22: The flexibility of FitNesse depends on what happens in the Java layer. The fixtures are a sort of API against the back end system.
• #23: The team consisted of PL / SQL developers, so developing the Java fixtures was up to the testers. Programming can only be learned by doing it. There is no course you can take to make you a better programmer faster. There are books that are helpful. After almost a year I wrote the first piece of what I believe to be efficient code.
• #24: JUnit allows you to explore what you built. It takes about 30 seconds to build and deploy the code for the fixtures. It runs automatically. Build & deploy is something we do not have to think about, it just happens. Eclipse makes life easier in so many ways. It can generate much of the code, gives suggestions, has useful plugins etc… Starting with OpenJPA (object-relational mapping software) was a risk. It requires an investment. In return it gives you (much) better control over database access.
• #26: Time spent on test automation cannot be spent on anything else. There are user stories in a sprint that need to be tested. On the other hand there is the apparent need for proper test data that can only be generated using test automation. I had to make choices in this area more than once and each time it is a difficult decision, especially when your choices impact the quality of the code that goes into production.
• #27: Test automation doesn’t do the root cause analysis for you. If the root cause analysis takes a couple of hours and there is a failure every day then automated checks consume quite a lot of time. This may negatively impact the value of automated checks. It is hard to write tests that give you effective and reliable information about the cause of failure.
• #28: Test automation should be a team effort; automation code should be maintained by the whole team. But placing it in the definition of done does not make it a team effort. People on the team have different thoughts about test automation; who should do it, how much effort should be put into it, in which language it should be written. Test automation can only become a team effort once you start discussing it with your team members.
• #30: These are some of the outcomes of the test automation effort.