Why Apps Succeed: 4 Keys to Winning the Digital Quality Game

Editor's Notes

• #4: Without further ado, I’d like to introduce our guests for today’s webinar: Maheema Raghunath is the Mobile Quality Manager for Cigna Health and has years of experience managing software quality teams in multiple locations around the world. Maheema (she’s sitting right next to me here), how are you today? And also we have Paul Dumas, Directory of API platform at Shop.com. Paul has been integral to numerous project in his past related to Digital Promotions and now works as a product owner and manager over the APIs that serve internal product groups and external partners. Paul, welcome. Wish you could be with the two of us over here, but glad you could join today’s conversation. How are you? Also, since there are two Paul’s here, we can all refer to me today as “Bruce” for simplicity’s sake.
• #5: So today I want to cover some of the basics of our report and have that serve as an opportunity to have a discussion about how technical teams are dealing with shifts in what and how we deliver digital experiences. We’ll start with...
• #6: So just as background for the data you’ll see today and in the report, it’s available online… We had some demographic questions to help segment in deeper analysis. Questions about how frequently they released software in 2015 and how they see those expectations will change in 2016. What their development and testing environments look like, their strategies for testing. The importance of user experience across all digital channels in their organization. Their challenges, team structure, and then a few questions related specifically to Perfecto’s line of business over how they manage their testing lab.
• #7: Based on the demographic questions, we can get a sense of who is represented in the findings. We had a lot of people from the high tech and financial services industries. Also, telecoms, retail, and healthcare responded. From that we also see that just about half of the respondents were from the enterprise segment, though there were plenty of others from startups on through to mid-size organizations too.
• #8: Now I want to take a moment to define a term that we use in the report and in today’s webinar. “Digital experience”
• #9: Everyone has a mobile phone and everyone ends up using its browser. We also use our desktops more than ever. This is not news. What is new is that we don’t use them exclusively for any one task. As consumers, we also expect that the voice of a brand is the same regardless of what device we use. The fact is, “and” is the normal state, not “or” when it comes to how we experience things.
• #10: A practical example is Finn. Finn is a gadget lover and loves watching documentaries in high-def. …This is a perfect example of an engaging digital experience built to be great from the customer’s perspective.
• #11: That’s why “digital experience” is such a powerful concept for organizations to adopt right now. It is aligned with how people actually use what we provide.The characterizing aspect of a “digital experience” mindset is that it’s all about engaging users from their perspective. … And it has an impact on the organization… @Maheema @Paul: any color you want to add?
• #12: And beyond that, it’s important to understand how classical thinking over channels (the delivery of your organization’s message) versus devices. Interactions with a consumer don’t just happen one device at a time; they happen through all the devices to varying degrees, purposely designed to do so.Through the conversations we have with our customers, we also see that doing this successfully requires a unified strategy across teams and devices. @Paul: you’re no stranger to aligning teams and resources to business strategy. How have you seen the way organizations adapt with new strategies for engaging customers over multiple channels?
• #14: So let’s take a look at two of the findings in the report.
• #16: When asked, “How important is it to your organization to deliver a reliable, consistent, and engaging user experience across all digital channels?”, a combined 80% of respondents said that it was a matter of survival or critical to their competitive positioning. This is very interesting because I’d expect leadership (people who are setting the vision) to understand this, but I’m surprised that so many people from various lines of business understand its competitive importance.@Maheema: You’ve covered projects across devices; how have you seen the quality landscape change now that we’re unifying multiple devices and channels in the customer’s digital experience? @Paul: In your work with APIs, how do you see “digital experience” thinking impact the business lines your APIs serve?
• #17: From the report, we also see very few respondents say they have just mobile only or web only environments, and that a mix of the two is by far the majority. We also see that a majority of organizations have one team for both mobile and desktop web testing. Makes sense now that responsive web design has hit it’s stride technologically. Looking into the data further, we see that 80% of small orgs and 50% of enterprises have a single team for their web presence on both mobile devices and in web browsers. If I had to guess, I’d imagine that their web team’s mobile testing strategy has considerable overlap with their native mobile app’s testing strategy in terms of reserving device time and synchronizing design resources and branding. @Maheema: from your point of view, what are some advantages and disadvantage if there’s one team for mobile devices and one for desktop browser environments? @Paul: where does the back-end team fit? How have you seen enterprises evolve testing to accommodate rapid API development used on the front-end?
• #18: We also see that many respondents consider providing a reliable and consistent experience quite a challenge. Two-third say that it is a major challenge. Of the small sliver of people who don’t think it’s a challenge, there were small but potentially relevant differences. But really, I want to leverage the experience of our guests in helping to shed light on why this is a challenge. @Paul: What are some things you’ve seen get in the way of delivering quickly and consistently?@Maheema: How about from your vantage point: what are some ways that consistency and reliability can be hard to achieve at the same time?
• #19: Also in the report, we identified four main challenges that organizations currently face when delivering excellent digital experiences.
• #20: The first is quite frankly, growth. People are having a hard time keeping up with releasing functionality faster, and the software business never sleeps. We asked the question “How many product releases…?” both for 2015 and what people estimate 2016 will look like for them. As you can see, from 21% to 31%, a 10 point difference represents 50% growth YoY. That’s huge expectations on technical teams, considering all that goes into a release. I know some shops have tried to go Continuous Delivery, most of the enterprises we talk to are heavily agile, but have had some difficulty going to CD. @Maheema: what have you seen in the teams you’ve worked with in the past? Are expectations that high? @Paul: What do you think is driving this kind of growth in expectations?
• #21: At first it seems like a “duh” moment. Everyone knows that automation is required to accelerate development and delivery. But when I dove deeper into the data, I found that while a majority of people seem to be on board, really only around half of all organizations have automated more than 40% of testing. Both in what people say they need to do to speed up and what they don’t quite have yet, it’s clear that automated testing is a top requirement to speed up. @Paul: At a management level, where have you seen teams get stuck when they don’t have proper automation in place? @Maheema: How have you seen testing teams adapt to go from manual to automated test technologies?
• #22: When we asked people what percentage of automated testing they perform, there’s considerable differences in the trend between mobile and web testing. Looks like the difficulty starts earlier with mobile testing than with web. @Maheema: as our resident QA expert, what can you tell us about why mobile testing is such a challenge? Why don’t we see more of it?
• #24: And speaking of speed, I have to mention as a side note, when digging in deeper and segmenting by release frequency, I did notice a difference related to responsive web teams which lead me to a thought process around the technology you’re delivering. Does it lend itself to fast deployment? Does it require the user to accept updates, or is the new version seamlessly delivered? Do you as the vendor control all the resources required for a “successful” deployment of new versions, or are we waiting for things like the app store to accept our new version? With things like responsive websites and APIs, you have the authority to update them without prior notification. @Paul: as someone who owns the decisions around back-end technologies like APIs, how do you see the teams (like an API team vs. a mobile or web team) making different decisions around releases? What’s the coordination look like between teams that depend on each other for functionality?
• #25: Another challenge we identified was that while a good portion of people test with other apps running in the background and test network switching, other real-world conditions we experience every day like interruptions outside the app, low memory and battery conditions, and performance problems aren’t as prevalent in the testing strategy. @Maheema: Why is this? What do you focus on in your testing strategy? How do you know if and when to incorporate greater realism into your tests? @Paul: I wanted to ask you, specifically related to performance testing, where does that fit into the overall strategy for quality in your products?
• #26: So at Perfecto, what we see in our engagements with enterprises is that when they are transitioning to a digital experience, it’s important to have a framework around what that means. Across multiple devices, personas become really important; having a concrete understanding of who you’re designing for really translates into the overall design and testing strategy. Integrating traits of the real world into your strategy makes measuring success along the customer journey much easier, not to mention detecting failure or regression proactively. And though there were a lot of U.S. companies who responded to our survey, it’s the end-user whose geographic region matters. It’s their experience, and if you’re not thinking about that in your simulations, you may want to check with marketing or ops analytics to see how much of the world you’re actually serving versus what you should be covering in automated tests. The last thing I’ll mention here is that diagnosing issues as they arise really needs reporting to contain things like visuals, steps to reproduce, and what conditions mobile devices were under when failures happen. This dramatically simplifies and speeds hand-off between testing experts, developers, and anyone else involved.
• #27: Obviously there’s lots of devices out there, but it’s the ones that your customers are using that matters. A digital experience is only as strong as its weakest link, so you need late-breaking support for the latest OS updates to test that when your customers go through that process, they’ll be happy. That means keeping up with major hardware and software vendor cycles, OS updates, could even mean that you’ll get pushed to test more based on what the development team uses for a software platform, i.e. Xamarin/Cordova Importantly though, supporting your apps and sites on multiple platforms means that you need a solid understanding of what’s coming out and when. @Maheema: what’s it like going through an OS update or certifying new devices? Where have you seen testing teams work through these challenges well? What slows this process down?
• #28: Additionally, I’d suggest that visual validation (much like I mentioned around reporting) dramatically simplifies the hand-offs and communications between teams and actors when troubleshooting and resolving issues. Everyone in mobile generally seems to agree that navigation is pain to test, and even the cutting-edge app providers like Facebook and Google require major architectural changes to avoid this with stateless, argument-based workflows. For us normal people, we have apps that have navigation like buttons, menus, and input elements. If you are struggling to speed automated testing, you may want to look into how efficient your script and testing framework is around navigation. The more efficient your tests are and the more relevant information you get out of them, the easier it will be for your organization to incorporate them into other workflows. Developers especially need to cycle fast when they’re implementing code, and your testing strategy needs to reflect this. Do as much automated unit testing as you can provided it doesn’t unnecessarily impede development time, and split off long-running tests into a build or nightly cycle.
• #29: Near to our heart at Perfecto is how people are still trying to manage in-house testing labs. We provide a cloud-based solution for that, so of course we had to ask some questions specifically about it. Once again, we see that the top challenge is keeping up with new browser versions and devices. We also see that for those who have an on-prem lab, it take staff to run it. @Maheema: in your experience, how does a cloud-based device lab enable organizations to ship better software? How does near-real-time availability of latest devices and versions help technical teams keep up with demand to deliver faster? @Paul: in your experience, how have you seen transitions to cloud-based services help software teams re-focus on their core competencies?
• #30: I want to drill into that point for a moment because it’s at the core of what Perfecto does, this is where I come in with my one major value proposition. We see that delivering really great digital experiences place considerable demand on organizations to provide real devices for testing from two directions: 1) external influences like hardware and framework updates from the market, and 2) internal teams who are trying to speed delivery through parallel (hence more) testing. Doing this yourself is like a science in and of itself. It’s really hard, and takes constant maintenance.
• #35: Questions:Prashant: How can QA get more unit testing from developers? => Why are developers not shipping unit tests? Low coverage or no coverage? Dev testing is all about TTM slippage risk; untested code takes longer to resolve downstream. PoCs are tough to make the case for testing unless part of a strategic motion. Xien (Jen): What’s the best tools for testing cross web site and mobile app? => depends on the skills and team; open source is good but needs to have support to fit into the enterprise; flexibility is also important along the toolchain; standards help John: DevOps is hard because everyone wants to use different tools. How can we fix this? => less about tools, more about attitude and sesing the need to change; fragmented toolchain can hinder automation; pilot how successful DevOps is in one or two projects and use that as your own case study