Automation is not the answer... unless you WANT it to be

Editor's Notes

• #2: Welcome everyone to “Automation is not the answer… unless you want it to be”... The goal of this session will be to give you a high level view of automated testing and explain why most people abandon it, and give some tips for those who don’t so that you too can be utilizing automation to it’s fullest potential
• #3: As Joel mentioned, I am Mike Sparks, I am the qa architect at Grant Street Group and also the product lead on a testing application known as Tellurium. Over the past few years I’ve assisted a number of companies with their automated testing efforts, and I’ve included a number of the lessons learned in this talk. In a shocking twist of events, I’m actually going to start by talking about how great automated testing is.
• #4: But Mike, doesn’t the title say that automated testing is NOT the solution? Yes, bare with me, we’ll get there. Let’s start with a story:
• #5: This was me, 8 years ago. Not the kid or the cat, the other guy. Doesn’t FEEL that long ago, but sadly it was. I was asked to start up a QA department for the company that I worked for. Neither they nor I knew anything about QA, so I said “what the heck?” As I started researching testing as part of QA in general I stumbled upon the mythical being known as
• #6: “automated testing”. Now at this point I was an army of 1 trying to assemble test plans and get help wherever I could find it to test this massively complex system. Here comes “automated testing” and it sounded like it was going to be the magical elixir that would take one of me, and make multiples of me that could help me accomplish all of the testing that needed to happen. Specifically automated testing promised 3 main advantages…
• #7: Saves time You can get a machine to do automatically what you’ve been doing manually, duh. Allows you to increase your test coverage If I have automated tests for this stuff over here, then I no longer need to do it manually. Therefore we can focus on these other areas. Instills confidence in your releases More tests mean that we know that more of our application is working, which means we feel better about pushing out changes. Huzzah! You can do things like automated regression testing, automated smoke testing, automated load testing, automated stress testing, automated spike testing, and the list goes on and on. All three of these things are great and extremely good selling points for automated testing, and to be honest are all true.
• #8: Today we have a team of people managing thousands of tests that are all running automatically, catching bugs early in the development process, and everything is great. BUT….
• #9: … there is a big gap between what happened 8 years ago and today. How we got from there to here made us go down paths that we never thought we’d need to travel based on those initial 3 selling points. It’s been my experience, working with tons of different companies, that that path can lead you to either great success, OR massive time loss and ultimately failure. So that’s why the title of this discussion is “Automated testing is NOT the answer” because for most people and companies it’s not. BUT, there is hope...
• #10: “... unless you want it to be.” It all comes down to how badly you and your company WANT it. If you think it’s going to be easy, it’s not. If you think it’s going to save you time, it won’t (at least not until you reach a certain threshold). Right now you’re thinking “Jeeze Mike, you’re really convincing me that I want to do this.” I know.
• #11: Fear not. I am here to help fill in the gap, let you know that you aren’t alone in all of this, tons of other people just like you have gone through this process and come through the other end with something that works for them. And today I’m going to give you the real reason why you shouldn’t trust the initial pitch of automation, and knowing the TRUTH will help you make a better decision about using it as a tool in your arsenal.
• #12: Let’s sidestep the hype and start with the truth about these 3 selling points. After that, after you’ve distilled that truth and determined that automated testing is still for you, then I’ll give you some tips and tricks for crossing that chasm and making it to the automated testing holy land. The Truth Saves time Yes, in the LONG term automated tests will save you time. But not YOUR personal time.
• #13: What you end up saving is what I call “theoretical time”. You’ll end up with tests that run for thousands of hours/month, which you can equate to management as “our automated tests run for 3,000 hours which is the equivalent of 15 people you didn’t have to pay!” That’s great, but in all likelihood management wasn’t going to hire 15 more people to do all of that manual testing anyway. It’s testing that otherwise just wouldn’t have gotten done if you didn’t have automated testing.
• #14: But what about the argument that “If I automate all of the stuff I do manually now, then that’s less time I need to spend manually testing.”? In reality, most individual testers get into automated testing to replace the work their doing now so that they can focus on other things. The thought is that automated testing will make you more efficient and save you time and unfortunately that’s not the case.
• #15: Someone needs to spend time creating all of those tests, maintaining all of those tests, running all of those tests, reviewing the results of those tests, checking for false failures, and reporting the findings. These aren’t negligible tasks, they all take time and resources to complete. Even one automated test adds overhead to your testing process. Not to mention that not all of your tests will likely be automatable. One of the unique things about testing is that no two companies really do it the same because no two companies are building the exact same applications in the exact same way. Therefore, there is no be-all-end-all automated testing tool that’s going to allow you to test 100% of the things you want to test. Especially if you’re trying to use something like a record/playback tool.
• #16: Regardless, it’s been my experience that people DRASTICALLY underestimate the time it takes to create, manage, maintain, run, and report on their automated tests. Not to mention that if your app changes, you need to go and fix all of your tests. Again, this isn’t as big of a problem long term, but it’s a trap that many testers new to automation fall into. “I can do this in conjunction with my manual testing and still handle everything myself” Unfortunately there just aren’t enough hours in the day, and most people learn this after it’s too late and get discouraged.
• #17: Don’t get discouraged, now you know. The benefits long term are massive if you can get over this hurdle, you just need to understand and persevere.
• #18: So will automated testing save you time? Yes, long term. But immediately it’ll add more work, much more work to your existing testing efforts. Next up, Automated testing allows you to increase your test coverage
• #19: Yes, automated testing does allow you to increase your test coverage… a lot. But it takes time, a lot, to create all of those tests. Our current test suite saves us over 5,000 hours of testing each month. That’s the equivalent of 25 full-time testers doing nothing but testing every hour of every workday. It’s a substantial amount of testing, and something that our company greatly values. But if you’re trying to do it solo while also doing manual functional testing, manual regression testing, and all of the other things that testers are used to doing, then you’re never going to get to the point where you have a large enough library that you’re likely hoping to get to someday. In most cases, people starting in automation actually spend MORE time testing just to keep up! Notice the “-5 hours”. That’s being generous, it’s likely much worse than that.
• #20: To get past this, you’ll likely need to either focus on automated testing full time, or hire someone else to focus on it full time while you continue to do your current tasks. You CAN get there, it’s just never as easy as it seems on the surface. You need to set realistic expectations, which we’ll talk about in a minute.
• #21: So, can you increase your test coverage with automated testing? Absolutely. But are you going to see substantial changes in the first few weeks, months, or year if you’re attempting it on your own? Probably not. Finally we have “Instills confidence in your releases” The idea being, if we have more tests, testing more areas of our application, then we’ll have more confidence that our app is doing what it’s supposed to do. Yes and no. One big warning, and I’ve seen this in multiple companies, is a false sense of equating pass percentage with app reliability. And this applies to not just automated testing, but manual testing as well.
• #22: For example, Let’s say that Company A and B produce similar applications and each have 3 main features, searching, a shopping cart, and payment processing. Company A has 100 automated tests that focus on all 3 of those main features. Company B has 500 automated tests (!) but they only focus on searching and the shopping cart. They have no automated tests for payment processing. Both Companies run their automated test suites and both come back passing at 100%. Management for both companies may look at that number and feel great, but as a tester, which one do you feel more confident in? It’s a trick question, one that we can’t answer without knowing more about the tests themselves. What are they doing? It’s all about context, and doing the work up front to build a test plan that sufficiently covers your application, which you’d want to do regardless of whether you’re doing automation or not.
• #23: But the trap that I’ve seen companies fall into is a false sense of app stability. “Our tests are passing at 100%, we’re good”. Not if your tests aren’t covering your entire application, which most automated testing suites don’t. Does 100% give you the warm and fuzzies? Sure. But it also makes people overlook other areas that shouldn’t be overlooked. Adam talked a bit about this during his Risk session yesterday.
• #24: Can you overcome this? Absolutely, by setting expectations early that automated testing is just one part of the larger testing process, and without proper planning could result in a whole lot of tests for one piece of functionality, which may give you confidence in THAT FUNCTIONALITY, but not the application as a whole.
• #25: Ok, so we’ve established that all of these points are valid as it relates to automated testing, but none of them are benefits that you’re going to experience right out of the gate. It’s going to take you time and resources to get there. Hopefully I haven’t scared you half to death. If not, then great, because now I’m going to give you 5 automation secrets so that if mentally you can get past the items above, you can be successful using automation.
• #26: #1. Don’t dive in head first. In this case the fox is actually winning, but the image is too good to pass up. Most people take on automation and in their pure excitement try to make as many tests as they can. Don’t do that. Plan, plan, plan. Identify the most important areas of your application. The ones that would spell doomsday if they failed. Think about the business processes behind your applications. What are the things that bring your applications to a screeching halt if they weren’t functioning properly.
• #27: For example, let’s look at Amazon. Their top 3 doomsday features might be: Searching Adding items to the cart Checking out. If any of those three things go down, then Amazon can’t do what Amazon is known for: selling products. Next, think about what features are used the most often. Sometimes they’ll be the same as your doomsday features, but often times they aren’t. So for Amazon, we could argue that their most used features are: Searching Reading reviews And viewing more details about the products Once you have your two lists, you want to compare them to determine what you should prioritize with your testing. Look for where you have crossover. That’s a good place to start planning your regression testing and figuring out what tests you should start automating first. But you don’t just need to plan for WHAT you’ll be testing, but also HOW you’ll be testing. What you don’t want to do is create hundreds of automated tests only to realize you need to go back and recreate them because you didn’t create them in the most efficient way for the long haul. I’ve worked with more than 1 company that’s happened to, where they’ve had a massive suite of tests but essentially had to spend months starting over to make themselves more agile and more efficient.
• #28: Instead you want your tests to be as flexible as possible and share processes as much as possible. For example, say you have 50 tests and they all start with logging into your application. Typically testers will just include the steps to login as part of every test. The problem with that is if the login process ever changes, then you need to update all 50 tests individually, which can take a massive amount of time. Instead you should use something like a phrase or a subroutine. A phrase or a subroutine is a small piece of your test whose process is defined outside of your test. For example something like logging in, searching for an account, checking out. These processes all contain repeatable steps that could be used within a multitude of your automated tests. So let’s look at those same 50 tests, but instead of the login process being written in all of the tests, we save the login process somewhere else as a phrase or subroutine and reference it in our test. Now if your login process changes, rather than updating all 50 tests, all you have to do is update the phrase or subroutine where you defined the login process and all of your tests will update automatically. It saves you a MASSIVE amount of time, and writing your tests this way from the start will save you a ton of headaches down the road. Not using something like phrases or subroutines has lead teams to completely abandon their automated tests and automated testing practices because the work to go back and add them in could seem insurmountable
• #29: Using something like phrases or subroutines will help you substantially cut down on how many tests you need to update when a particular process changes. This is an example of what a login phrase could look like in our automated testing tool, Tellurium. Things like phrases and subroutines, or even just planning what tests you should automate first are all things that you need to consider BEFORE you start creating your tests since they will save you massive amounts of time long term and could be the difference between automated testing working for you, or not.
• #30: #2. Keep calm and Do not automate everything Some things are much more important to automate than others. For example, what’s the first test that everyone wants to automate? Logging in.
• #31: Sure, go ahead and create a test for logging in if you’re going to use it for smoke testing. But odds are that you’re going to test logging in while you’re in the process of testing other features in your application.
• #32: For example: Say you create an automated test to test logging in. Now say that you’re manually testing what happens when you search for a specific item within your application. Odds are, you’re going to have to login in order to manually test that functionality. So you’d manually be duplicating exactly what the automated test is doing. Not only that, but you’d be maintaining that automated test, you’d be running that automated test, you’d be checking the results, all for a process that you already know is working because of the manual testing you’re doing. That’s the opposite of what we’re trying to accomplish with automation. Instead you want to identify the features within your application that are best suited for automation, and which ones you’ll test manually, either intentionally or not. Don’t do both, you’ll just be making more work for yourself. Keep in mind, there are some things that you simply will not be able to automate OR they’ll be so difficult to automate and maintain that you’re better off testing it manually.
• #33: For example: Say you have an application that prints documents that are then mailed. And say that the document needs to have everything aligned just right otherwise the address won’t appear in the envelope's window. You COULD build an automated test to check the positioning of the text, but not all automation tools support that. And even if they do, those can be very complicated tests to try to build and maintain. Instead, it may be better to consider a different workflow: have your automated test simply print out one of the forms and then have someone manually spot check the print out. Are you creating additional work for someone? Yes. But is the amount of time required to pickup a form from the printer and verify that everything is aligned LESS than what is required to create, manage, maintain, and review the results of automated test that does the same thing? More than likely, Yes. If you’re going to be testing it regardless, you want to find the efficiencies in the process and automation is NOT always the most efficient way to test something. So don’t feel like you need to automate everything, an efficient process purposely balances both manual and automated testing.
• #34: And that segways into our next point. This is from a movie called “Office Space” where they brought in consultants to essentially make everyone in the office justify their continued employment. Often testers see automated tests as their replacement, and ultimately making themselves obsolete. Not so! <click> #3. Automation does NOT mean the end of manual testing
• #35: Pixar ran into a lot of opposition early on when they started making computer animated movies. Until Toy Story, the vast majority of animated moves were produced by traditional animators using pen and paper. When it was originally introduced, computer animation appeared to be this breakthrough technology that could make more realistic films and cut down on the production time. But traditional animators saw computer animation as a threat to their jobs, concerned that long term it was going to force them out of work. What they didn’t realize was that the skills they’d developed related to understanding movement, scale, character design, conveying emotion, focus… all of those things would need to be translated and carried over into computer animation. Their knowledge was and still is crucial to make computer animation successful. The parallels between animation and automated testing are hard to ignore.
• #36: Automation is simply a tool to help testers accomplish what they need to accomplish. Sure, it’s great. But automated tests can only test exactly what you tell them to test. No more, no less. So they’re only as effective as the people creating the automated tests. There will always be a need for manually testing a new piece of functionality and the skillset of great testers will always be in demand. Testers have the experience to know tips and tricks on how to “break” their application, and those tips are sometimes more valuable than the potential for automated testing. One of the big intangibles that makes testers so valuable is their approach to testing and the testing mentality.
• #37: For example, say you’re supposed to test an input field. Most people would just type in that field, submit the form, and say it works. As we know, Testers look at that field and think things like: What happens if I DON’T type in that field? Will I get an error message? What happens if I type an invalid value in that field? Will the form still submit? Will it throw an error message? And what constitutes an invalid value? If you’re supposed to enter a number between 1 and 100, what happens if you enter a negative number? We think beyond the obvious because we’re curious and want to uncover what secrets our apps are hiding. That curiosity is something that’s extremely valuable and is something that other employees covet in the changing app development landscape. Those are all things that you COULD automate, but most people don’t think to. So that experience and knowledge that testers have is valuable in teaching people HOW to test applications. Considering that apps and app development are constantly changing there will always be manual testing that occurs in some capacity. How large of a tool it will be in your tool belt is up to you.
• #38: Tip #4. Set realistic expectations for management Implementing automation, correctly, is not just an add on for an overworked tester or team. As we’ve discussed, it requires dedicated resources to create, manage, modify, run, and report on the results. Eventually the time spent by those resources will greatly be dwarfed by the hours of automated testing you’ll be able to accomplish, but that won’t be the case to start. Don’t oversell what you’ll be able to accomplish with automation in the first few months just because of the long term benefits. Be up front with management, making it clear that it’s going to be slow going and there will be ramp up time to get to the point where you want to be. It’s like Rob said in yesterday’s keynote, it’s all about communication. Automation seems like this mystical thing. The less mystical and grounded you can make it with management, the more interested and engaged they’ll be in wanting to see it succeed. And why is this important? Why do you need management’s buy-in?
• #39: Because to really make automation work it’s going to require a change in process, a cultural shift within your company. It’s very difficult to just “kind of” implement it. Everyone needs to grab hold and move forward together or it’s not going to stick.
• #40: Example of GSG and Tellurium Tellurium is a testing tool that we developed within the larger company, Grant Street Group. We built it to help our testing efforts with one specific application within the company. That app was too complex for any other testing tools available, so we had to build something powerful enough to meet our needs. As other teams and applications heard about what was possible they loved the idea of using Tellurium for their testing as well. The problem was just that, they everyone loved the IDEA of automated testing. Unfortunately they weren’t willing to change their business processes and put in the time and energy to implement automated testing properly. Eventually we had to sell the idea to upper management, who then mandated that automated testing be adopted company wide with the understanding that it would require a change in business processes. As soon as that mandate was passed down everyone took hold and automated testing has become a reliable part of our business processes.
• #41: The “A” stands for “Automation”. Most people don’t know that, now you do. As part of your planning figure out the right number of people you’re going to need to make it work and make that known to management up front. A good method for this is to show them all of the amazing things that automation can do for you, and then make it clear that you won’t get anywhere near there without X number of people and buy in from the whole company. The advantages of automation are substantial enough that you’ll at least get SOME consideration, even if they don’t sign off and approve the hiring of all the people you need to implement your grand plan. Baby steps. If the minimum you can get is a mandate that it be adopted company wide, then you’re headed in the right direction.
• #42: If you’re having a hard time getting buy in, try automating just a little bit yourself. You know it won’t be sustainable long term, but if you can show the company results as part of a proof of concept, then you’re more likely to get buy in for the larger cultural shift required to get everyone on board.
• #43: Tip #5. Be collaborative Look how happy they all look...
• #44: If you have a testing tool that only you can use then: A) The onus all falls on you to make it work, and B) You’re never really going to get buy in from anyone else. You’ll be the one in the top right, talking about how great automation is, but no one else will care since they really aren’t dealing with it. This can be a problem since a lot of automated testing tools don’t allow for easy collaboration.
• #45: Most have to be installed on just one computer which means that all tests must be created, modified, and run on that one particular machine. This substantially limits your ability to collaborate and will make it substantially more difficult for you to grow automation within your organization. Instead what you want is an automation tool that allows developers and testers to work together to build the test suite.
• #46: If you only have one person that can work on the tests then they become the stopgap and progress will be slow and can easily be abandoned. But if you have more people sharing ideas about what tests should be considered, what efficiencies could be gained, and what other resources could be committed to doing something like load testing or spike testing, then your efforts become substantially more valuable more quickly. Example from GSG: Once it was mandated that everyone use it, then everyone started asking more questions, sharing ideas, and collaborating on building, running, and maintaining our test suites. Different people could run the test suite for different purposes, whenever they needed to. Developers could run the tests as soon as they added their changes to the Test Environment Dev Ops could run the tests as soon as they did a promotion to make sure the applications were working as expected Testers could run the tests as part of their overall regression efforts ITS can use some of the tests to do load and spike testing And the list goes on and on We NEVER could have gotten to where we are today if it wasn’t for that collaborative process. And this isn’t an isolated example. I’ve talked and worked with MANY companies who have had similar experiences
• #47: So there you have it, 5 tips for succeeding with automated testing once you truly understand and determine whether it’s right for you in the first place. [Read through them] Remember, if you’re just starting out in automation it’s not going to be easy or seem like it’s what you signed up for. But, if you know that going in and follow these 5 tips, then your chances of succeeding will be substantially greater than those who have come before you. You CAN do it…. If you WANT to.
• #48: If you’re looking for automated testing tool for web applications that includes things like phrases and collaboration I’d recommend checking out Tellurium. It’s free, there’s nothing to install, and you’ll learn very quickly whether it’s something that will meet your needs.
• #49: Thank you