![Using the ACTS Combinatorial Tool:
Example
18
Parameters:
Andriod AppPlatform
[Device 1, Device 2, Device 3, Device 4, Device 6,
Device 7, Device 8, Device 9, Device 10]
IoTProtocolHome [true, false]
IoT Devices
[Refrig, Stove, mircrowave, TV, front door, Garage
door, Home gaurd, Stereo, Temp Control, Lights,
Drapes, Water Heater, window openers]
Routers [0, 1, 2, 3, 4, 5]
Comm providers
[Cell1, Broadband, cable, Cell 2, Space based,
Vendor godzilla]
Data [1, 0, -1, 99999, -99999, 100, -200]
Test Case# Andriod AppPlatform IoTsHome IoTDevices Routers
Comm
providers Data
0 Device 1 false Refrig 1 Broadband 0
1 Device 2 true Refrig 2 cable -1
2 Device 3 false Refrig 3 Cell 2 99999
3 Device 4 true Refrig 4
Space
based -99999
4 Device 6 false Refrig 5
Vendor
godzilla 100
5 Device 7 true Refrig 0 Cell1 -200
119 Tests
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"](https://image.slidesharecdn.com/t49-160708143716/85/Use-Combinatorial-Testing-for-Mobile-Device-Fragmentation-20-320.jpg)
![Other Statistical Tools to Consider
General Technique
Concept Tool Examples (Note 1)
Examples of where technique
can be used
Specific sub- technique
examples
Combinatorial Testing
ACT [4], Hexawise[5]
rdExpert [6]
PICT[7]
Medical, Automotive,
Aerospace, Information Tech,
avionics, controls, User
interfaces
Pairwise, orthogonal arrays, 3-
way, and up to 6 way pairing are
now available
Design of Experiments
(DOE)
DOE ProXL[8]
DOE++ [9]
JMP [10]
Hardware, systems, and
software testing where there
are "unknowns" needing to
be evaluated
Taguchi [12]
DOE
Random Testing
Random number
generator feature used
from most systems or
languages
Chip makers, manufacturing
quality control in hardware
selection
Testing with randomly
generated numbers includes:
fuzzing and use in model-based
simulations
Statistical Sampling SAS [10]
Most sciences, engineering
experiments, hardware
testing, and manufacturing
Numerous statistical methods
are included with most
statistical tools
Software Black box
Domain Testing
Mostly used in manual
test design, though
some tools are now
coming available [11]
All environments and types of
software tests. These are
“classic” test techniques, but
still underused
Equivalence Class, Boundary
Value Analysis, decision tables
(Note 2)
Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"](https://image.slidesharecdn.com/t49-160708143716/85/Use-Combinatorial-Testing-for-Mobile-Device-Fragmentation-21-320.jpg)
Use Combinatorial Testing for Mobile Device Fragmentation
- 1. T4 Mobile Testing 5/5/16 9:45 Use Combinatorial Testing for Mobile Device Fragmentation Presented by: Jon Hagar Grand Software Testing Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.stareast.techwell.com/
- 2. Jon Hagar Grand Software Testing Jon Hagar is a systems software engineer and testing consultant, supporting software product integrity and verification and validation (V&V), with a specialization in mobile and embedded software system testing. For more than thirty years, Jon has worked in software testing and engineering projects. He authored Software Test Attacks to Break Mobile and Embedded Devices; consults, presents, teaches, and writes regularly in many forums on software testing and V&V; and is lead editor/author on committees including OMG UTP model-‐based test standard, IEEE 1012 V&V plans, and ISO/IEEE/IEC 29119 software test standard. Contact Jon at [email protected].
- 3. Use Combinatorial Testing for Mobile Device Fragmentation Jon D. Hagar, Consultant, Grand Software Testing [email protected] 1 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 4. • Scary stories • It only takes a few minutes of using an App before users like or hate it • Worse than that. . . – Many users will post a poor social media review of the app or device – You may be on the nightly news (bad press is not good) – A question I get a lot, “how do we deal with fragmentation?” • So You want to be – Part of the billions of devices » You want to be GREAT 2 The Mobile Opportunity Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 5. What We Will Cover • Introduction and definitions • A combinatorial test attack pattern • Some Combinatorial (CT) Tools – Demo • Wrap up 3 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 6. Basic Definitions • Test – the act of conducting experiments on something to determine the quality (ies) and provide information – Many methods, techniques, approaches, levels, context – Considerations: input, environment, output, instrumentation • Quality (ies) – Value to someone (that they will pay for) – Functional – Non-functional – It “works” – Does no harm • Are there (critical) bugs? 4 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 7. The Mobile-IoT-Embedded Space 5 Embedded IoT Mobile-Smart Personal Computers Big Iron Cloud Many Options Huge Numbers of Devices (billions) Numbers of Devices (millions) Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 8. • Embedded – Software contained in “specialized” hardware… • Mobile and handheld devices—small, held in the hand, connected to communication networks, including – Cell and smart phones – apps – Tablets – Medical devices • IoT – Internet of Things are traditional devices with software and comms added • Mobile, Handheld, IoT typically have: – Many of the problems of classic embedded systems – The power of PCs/IT – More user interfaces than classic embedded systems – Fast and frequent updates • Devices are “evolving” with more power, resources, apps, etc. • Mobile and IoT are (currently) the “hot” area of computers/software You know what they are. . . Right? Embedded, IoT, Mobile and Handheld? Test Brakes What’s this? Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 9. • Embedded – Software contained in “specialized” hardware… – Minimal networking-communication PLUS • Mobile and handheld smart devices—small, held in the hand, highly connected (web, cloud, servers,….) • IoT – Internet of Things are “traditional” embedded and new devices with software and communication added What is a Mobile (and IoT) device? Test Brakes Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 10. Defining Software Capabilities • James Whittaker defines 4 fundamental capabilities that all software possesses 1. Software accepts inputs from its environment 2. Software produces output and transmits it to its environment 3. Software stores data internally in one or more data structures 4. Software performs computations using input or stored data • To this, we expand and refine based on an mobile context: – Within time – Using specialized hardware (as sub of items 1 and 2 above) and control – Security and privacy – Different development lifecycle constraints 8 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 11. Attack-based Testing Patterns What is an attack? • A pattern (of testing) based on a common mode of failure seen over and over – Some see this as a negative, when it is really a positive – Attacks seek the “bugs” that may be in the software – May include or use classic test techniques and test concepts • Lee Copeland’s book on test design • Many other good books • A Pattern (more than a process) which must be modified for the context at hand to do the testing • Testers learn mental attack patterns when working over the years in a specific domain Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 12. Example Attacks (from “Software Test Attacks to Break Mobile and Embedded Devices”) • Attack 1: Static Code Analysis • Attack 2: Finding White–Box Data Computation Bugs • Attack 3: White–Box Structural Logic Flow Coverage • Attack 4: Finding Hardware–System Unhandled Uses in Software • Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs • Attack 6: Long Duration Control Attack Runs • Attack 7: Breaking Software Logic and/or Control Laws • Attack 8: Forcing the Unusual Bug Cases • Attack 9 Breaking Software with Hardware and System Operations • 9.1 Sub–Attack: Breaking Battery Power • Attack 10: Finding Bugs in Hardware–Software Communications • Attack 11: Breaking Software Error Recovery • Attack 12: Interface and Integration Testing • 12.1 Sub–Attack: Configuration Integration Evaluation • Attack 13: Finding Problems in Software–System Fault Tolerance • Attack 14: Breaking Digital Software Communications • Attack 15: Finding Bugs in the Data • Attack 16: Bugs in System–Software Computation • Attack 17: Using Simulation and Stimulation to Drive Software Attacks • Attack 18: Bugs in Timing Interrupts and Priority Inversion • Attack 19: Finding Time Related Bugs • Attack 20: Time Related Scenarios, Stories and Tours • Attack 21: Performance Testing Introduction • Attack 22: Finding Supporting (User) Documentation Problems • Sub–Attack 22.1: Confirming Install–ability • Attack 23: Finding Missing or Wrong Alarms • Attack 24: Finding Bugs in Help Files • Attack 25: Finding Bugs in Apps • Attack 26: Testing Mobile and Embedded Games • Attack 27: Attacking App–Cloud Dependencies • Attack 28 Penetration Attack Test • Attack 28.1 Penetration Sub–Attacks: Authentication — Password Attack • Attack 28.2 Sub–Attack Fuzz Test • Attack 29: Information Theft—Stealing Device Data • Attack 29.1 Sub Attack –Identity Social Engineering • Attack 30: Spoofing Attacks • Attack 30.1 Location and/or User Profile Spoof Sub–Attack • Attack 30.2 GPS Spoof Sub–Attack • Attack 31: Attacking Viruses on the Run in Factories or PLCs • Attack 32: Using Combinatorial Tests • Attack 33: Attacking Functional Bugs Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 13. In Mobile and IoT Many Example Combinations: Standards, Interfaces, Protocols, Platforms, Software, and Data Patterns 11 Network-Comm Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices" Many of these Combinations Will Need Testing
- 14. Exercise: How should we test these? (How do you do it now?) 12 How many tests are needed? Coverage of combinations? How do we find errors? Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 15. Combinatorial Testing (CT) Math Offers Solutions • CT has long history of Usage • CT uses many tools • CT is still underused • CT has some cool possibilities • CT should be one of the attack techniques used • Find out how CT can help your testing 13Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 16. Math-based Testing Testing is a sampling problem: How can Math aide testing? • Test systematically the numbers of devices, configurations, networks, etc. • Use sampling in environments and quality control • Use sampling of data from the input domain space • Help use Big Data Analytics to feed testing 14Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 17. Pattern Attack 32: Combinatorial Tests 15 • When to apply this attack? – There are numerous related variables and variable values which interact – Validation Analysis Upfront – Testing throughout the life cycle and in Maintenance Mode • What faults make this attack successful? – Untested configuration combinations – Data “bugs” • Who conducts this attack? – Tester, analyst • Where is this attack conducted? – Tool running in the lab or field • How to determine if the attack exposes failures? – A test fails to meet success criteria – Hard crash - NIST Data Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 18. Attack 32: Combinatorial Test Patterns 16 • How to conduct this attack – basic pattern – Identify combinatorial situation – Identify combinatorial tool – Identify variables – Identify values – Identify constraints on values – Enter variables and values into tool with constraints – Exercise resulting combinations in usage scenario tests or automated tests – Look for failures – Repeat and refine as needed Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 19. • Android or Other OS Example Usage: Numbers of data choices, devices and configurations 17 • Hardware • Connected devices • Data • Routers • Home Protocols How many Tests? 10 x 2 x 13 x 6 x 6 x 7 = 65,520 tests! Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 20. Using the ACTS Combinatorial Tool: Example 18 Parameters: Andriod AppPlatform [Device 1, Device 2, Device 3, Device 4, Device 6, Device 7, Device 8, Device 9, Device 10] IoTProtocolHome [true, false] IoT Devices [Refrig, Stove, mircrowave, TV, front door, Garage door, Home gaurd, Stereo, Temp Control, Lights, Drapes, Water Heater, window openers] Routers [0, 1, 2, 3, 4, 5] Comm providers [Cell1, Broadband, cable, Cell 2, Space based, Vendor godzilla] Data [1, 0, -1, 99999, -99999, 100, -200] Test Case# Andriod AppPlatform IoTsHome IoTDevices Routers Comm providers Data 0 Device 1 false Refrig 1 Broadband 0 1 Device 2 true Refrig 2 cable -1 2 Device 3 false Refrig 3 Cell 2 99999 3 Device 4 true Refrig 4 Space based -99999 4 Device 6 false Refrig 5 Vendor godzilla 100 5 Device 7 true Refrig 0 Cell1 -200 119 Tests Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 21. Other Statistical Tools to Consider General Technique Concept Tool Examples (Note 1) Examples of where technique can be used Specific sub- technique examples Combinatorial Testing ACT [4], Hexawise[5] rdExpert [6] PICT[7] Medical, Automotive, Aerospace, Information Tech, avionics, controls, User interfaces Pairwise, orthogonal arrays, 3- way, and up to 6 way pairing are now available Design of Experiments (DOE) DOE ProXL[8] DOE++ [9] JMP [10] Hardware, systems, and software testing where there are "unknowns" needing to be evaluated Taguchi [12] DOE Random Testing Random number generator feature used from most systems or languages Chip makers, manufacturing quality control in hardware selection Testing with randomly generated numbers includes: fuzzing and use in model-based simulations Statistical Sampling SAS [10] Most sciences, engineering experiments, hardware testing, and manufacturing Numerous statistical methods are included with most statistical tools Software Black box Domain Testing Mostly used in manual test design, though some tools are now coming available [11] All environments and types of software tests. These are “classic” test techniques, but still underused Equivalence Class, Boundary Value Analysis, decision tables (Note 2) Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 22. Many Variables and Choices 20 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices" And the ACTS tool in real-time (be on the high wire)
- 23. Tool Demo Link To ACTS Tools 21 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 24. Expanding Combinatorial Testing • For Dev-Ops – Sampling user data – Model-based testing – Advanced data selection • Support domain testing • Do test without an Oracle – NIST Study – Combine with automation – Run 4-to-6 way combo’s – Look for Major Crashes 22 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 25. Summary Common Mobile Problem • Data selection • Dealing with numbers of configurations – Hardware, Software, Protocol, etc. • Testing within time and budget Overlooked Solutions • Data analysis with sampling – Classic testing • Combinatorial Testing with tools – Test Automation (not a requirement) • Reduce combinations to fit within budget and schedule 23 Copyright 2016 Jon D. Hagar – "Software Test Attacks to Break Mobile and Embedded Devices"
- 26. • • • • • • • • • • Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
- 27. Book List • “Software Test Attacks to Break Mobile and Embedded Devices” Jon D. Hagar, 2013 • “How to Break Software” James Whittaker, 2003 – And his other “How To Break…” books • “A Practitioner’s Guide to Software Test Design” Copeland, 2004 • “Introduction to Combinatorial Testing” D. Richard Kuhn Raghu N. Kacker Yu Lei , 2013 • Copyright 2016 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”