Guerilla testing for the IoT / REST jungle

Editor's Notes

• #2: If you are in this audience, congratulations, you’re responsible for the future of mankind, no pressure
• #4: This is the goal I want to impress on you today
• #5: Problem: How do you do that in a world that’s constantly changing faster than we can keep up, and where you know less about the way people will use your products than ever before?
• #6: We have arrived at the life of a guerilla tester, because… Testing isn’t about what you know, it’s about how you explore what you don’t.
• #7: Both IoT and Microservices break the world up into smaller bits, and for good reason
• #8: The physical world is complicated, requires lots of detangling / parsing or instrumentation
• #9: The information space, data and code, can get out of hand surprisingly fast (spaghetti code)
• #10: Lots of things. Traditional: laptops, mobile phones, tablets, POS stations, scanners, ATMs Non-consumer stuff: medical and scientific instruments, automotive, robotics IT: Servers, Gateways, Switches, Balancers, Firewalls, Filters, UPC, HVAC, Lighting, Entertainment Infra: Buildings, bridges, tunnels, dams
• #11: There are lots of “not things” Your website, a hub of information you can’t life without, but it’s not a “thingy” Your API is a purely digital product. There is no real-life counterpart to your API. The concept of APIs is a purely information-age concept. AWS Lambda, IronWorker, StackStorm, IFTTT
• #12: What’s different in a world inundated by connected bits? “bits” demand composability, your “things” MUST be recomposed in ways you didn’t intend
• #13: Data on context (locality, identity, capability) will be even more important than ever before Metadata is often more important than data; social connections, not status updates
• #14: Failures before more real, tangible, tactile, and personal (J I N D O) HVAC malfunction in the middle of winter, alarms, medical misinformation, car brakes
• #15: Jindo Haenam bridge outfitted with structural sensors, when’s the best time to take advantage of that system?
• #16: Our testing principals will have to adapt to the shifting emphasis on “orchestration” of things and security What I’ve been calling “orchestration testing”, includes unit/service/workflow testing, but also concepts like consumer-driven contract testing, API and device virtualization, and necessarily asynchronous testing processes to simulate how the real world situation plays out. Testing beyond the “happy path” scenarios becomes just as necessary as getting the obvious business cases out of the way. In fact, letting testers identify SPOFs and poorly performing 3rd parties will become vital to rapid-release dev cycles.
• #17: Why do I call this a jungle? A densely packed ecosystem with lots of unrestrained growth A lack of standards, MQTT and CoAP are working on this SDKs run rampant, ultra-proprietary technology impedes innovation and growth There is a food chain, and big players look to devour
• #18: The good news: some things are easier to test than others; what makes something easy to test? How simple is it? How many moving parts? How many interaction points does it have? Does it require physical interaction?
• #19: Is it well documented (for people AND machines)? Toolchain / ecosystem support Interoperability, standard protocols
• #20: What does a mixed-environment test look like in tooling? How does MQTT fit alongside our existing knowledge?
• #21: Why does this matter to testers? If it can’t be tested (easily), it needs to go back to the drawing board; get up, stand up Testing processes will move from narrow constraints to few constraints with lots of things “Orchestration testing” will require architect-like thinking over automation process Things like Docker will be invented to solve the “lots of bits” problem, and affect our workflow
• #22: How can we make sure the IoT doesn’t run off the rails? Iterate, iterate, iterate; you’ll never know if you don’t try; small, anticipated failure teaches Earlier testing; reduces downstream cost, proves feasibility Contribute to open standards, don’t just use them; devs and testers alike Think about the whole picture; how will my devices and services be used and misused?
• #23: And that’s all I had. Thank you for letting me come here and speak!