SlideShare a Scribd company logo

Efficient Dependency Detection for Safe Java Test Acceleration

J
jon_bell

The document discusses efficient dependency detection for safe Java test acceleration, focusing on the challenges and techniques for minimizing build times dominated by testing. Key methods include regression test selection, test suite minimization, and utilizing ElectricTest for identifying test dependencies to enhance speed while ensuring safety. Results indicate that the proposed approach significantly reduces runtime overhead compared to traditional methods, improving the efficiency of Java testing processes.

Technology
1 of 56
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
@_jon_bell_ESEC/FSE September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan Bell, Gail Kaiser, Eric Melski and Mohan Dattatreya Columbia University & Electric Cloud, Inc
@_jon_bell_ESEC/FSE September 4, 2015 Simplified Software Lifecycle Make changes to code Build & test Commit How long is too long of a build? 1 day? 6 hours? 10 minutes?
@_jon_bell_ESEC/FSE September 4, 2015 Simplified Software Lifecycle • Compile sources • Generate documentation • Run tests • Package Make changes to code Build & test Commit
@_jon_bell_ESEC/FSE September 4, 2015 Testing Dominates Build Times 20% 38% 41% 351 projects from GitHub Testing Other Compiling
@_jon_bell_ESEC/FSE September 4, 2015 Testing Dominates Build Times 14% 26% 60% Projects taking > 10 minutes to build (69) Testing Other Compiling
@_jon_bell_ESEC/FSE September 4, 2015 Testing Dominates Build Times 2%8% 90% Projects taking > 1 hour to build (8) Testing OtherCompiling
Faster tests = Faster builds
Test acceleration is well studied
@_jon_bell_ESEC/FSE September 4, 2015 Regression Test Selection Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01] Changeset
@_jon_bell_ESEC/FSE September 4, 2015 Regression Test Selection Test 1 Test 3 Test 8 Test 4 Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01] Changeset Tests not relevant to changeset: skipped
@_jon_bell_ESEC/FSE September 4, 2015 Test Suite Minimization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 < /> Code Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et al. [IST ’98]; Wong et al. [ICSE ’95] and more
@_jon_bell_ESEC/FSE September 4, 2015 Test Suite Minimization Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 13 Test 14 < /> Code Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et al. [IST ’98]; Wong et al. [ICSE ’95] and more Redundant tests: removed
@_jon_bell_ESEC/FSE September 4, 2015 Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
@_jon_bell_ESEC/FSE September 4, 2015 Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
@_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code
@_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External Factors
@_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External Factors Test 1
@_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External Factors Test 1Test 2
@_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External Factors Test 1Test 2Test 3
Not sound in practice
@_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Write, Value “A” Test 4 Read Write, Value “B” Value: B Test 3 Read
@_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies Test 1 Test 2 Test 3Test 4 Shared File
@_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies Test 1 Test 2 Test 3Test 4Test 1 Test 2 Test 3 Shared File Write, Value “A” Test 4 Write, Value “B” Read, Expect Value “A” Value: B Read
@_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies Test 1 Test 2 Test 3Test 4Test 1 Test 2 Test 3 Shared File Write, Value “A” Test 4 Write, Value “B” Read, Expect Value “A” Value: B A manifest test dependency Read
@_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies: A Clear and Present Danger • Really exist in practice (Zhang et al. found 96, Luo et al. found 14) • Hard to specify - if we could specify, would be safe to accelerate • Rarely: isolated (especially not in long building projects) • Existing technique to detect: combinatorially run tests [Zhang, et al ’14]
@_jon_bell_ESEC/FSE September 4, 2015 Brute Force Dependency Detection • Looked at feasibility on 10 large open source test suites • Exhaustive approach: > 10300 years to find all dependencies • Pairwise approach: Average 31,882 executions of the entire test suite to find (incomplete) dependencies • Problem: How do we safely accelerate test suites in the presence of unknown dependencies?
@_jon_bell_ESEC/FSE September 4, 2015 Manifest Test Dependencies • Definition: a data dependence between tests T1, T2 that results in the outcome of T2 changing • All manifest dependencies are data dependencies • Not all data dependencies are manifest dependencies
@_jon_bell_ESEC/FSE September 4, 2015 Data Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Write, Value “A” Test 4 Read Write, Value “B” Test 3 Read Present Dependencies: Test 1 must run before 2 and 3 Test 4 must run after 2 and 3
Key Insight: Dependencies don’t need to be precise, but must be sound
@_jon_bell_ESEC/FSE September 4, 2015 Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Idle extra capacity
@_jon_bell_ESEC/FSE September 4, 2015 Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Idle extra capacity A lot of dependencies, but still a 2x speedup
@_jon_bell_ESEC/FSE September 4, 2015 ElectricTest - Detecting Data Dependencies in Java • Tracks in-memory dependencies (JVMTI plugin) • Tracks file and network dependencies (IO-Trace agent) • Implemented entirely within the Oracle or OpenJDK JVM, no specialized drivers, etc required • Captures stack traces when dependencies occur to support debugging • Generates dependency trees to enable sound test acceleration
@_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies After each test, garbage collect; traverse heap to map objects back to static fields. Class A End of test 1
@_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies After each test, garbage collect; traverse heap to map objects back to static fields. Class A End of test 1
@_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies After each test, garbage collect; traverse heap to map objects back to static fields. Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield End of test 1
@_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies During test execution, monitor accesses to existing objects Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield During Test 2
@_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies During test execution, monitor accesses to existing objects Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield W2 W2 Write! During Test 2
@_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies During test execution, monitor accesses to existing objects Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield W2 W2 W1 Read! During Test 2 Dependency!
@_jon_bell_ESEC/FSE September 4, 2015 Identifying External Dependencies Application under test Network Filesystem Log remote host address Log path
ElectricTest enables sound exploitation of existing test acceleration techniques
@_jon_bell_ESEC/FSE September 4, 2015 Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test 11 Test 12 Test 13 Test 14 Test 15
@_jon_bell_ESEC/FSE September 4, 2015 Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14
@_jon_bell_ESEC/FSE September 4, 2015 Safe Test Selection Test 15 Single test selected to be executed
@_jon_bell_ESEC/FSE September 4, 2015 Safe Test Selection Test 15Test 1 Test 2 Test 3 Single test selected to be executed with its dependencies
@_jon_bell_ESEC/FSE September 4, 2015 Understanding Dependencies • What should a developer do about test dependencies? • Might be intentional (e.g. cache shared state) • Might be unintentional but OK (e.g. loggers) • Might be unintentional and bad (e.g. bug)
@_jon_bell_ESEC/FSE September 4, 2015 Assisting Debugging Debugging information reported by the previous technique Test 3 Test 1 Depends on
@_jon_bell_ESEC/FSE September 4, 2015 Assisting Debugging Exception*in*thread*"main"* edu.columbia.cs.psl.testdepends.DependencyException:*Static*Field* ClassA.FieldA*member*was*previously*written*by*Test*1,*read*here.* ! at!edu.columbia.cs.psl.testdepends.test.Example$NestedExample.dragons(Example.java:20)! ! at!edu.columbia.cs.psl.testdepends.test.Example.moreMagic(Example.java:12)! ! at!edu.columbia.cs.psl.testdepends.test.Example.magic(Example.java:8)! ! at!edu.columbia.cs.psl.testdepends.test.Example.main(Example.java:15)! Test that wrote value Stack trace shows use Value that is read
@_jon_bell_ESEC/FSE September 4, 2015 Evaluation • RQ1: Recall (accuracy) • RQ2: Runtime overhead • RQ3: Impact on acceleration
@_jon_bell_ESEC/FSE September 4, 2015 RQ1: Recall Project Joda XMLSecurity Crystal Synoptic Dependencies Detected Ground Truth Ground Truth 2 4 18 1 ElectricTest Writers 15 3 15 10 Readers 121 103 39 117
@_jon_bell_ESEC/FSE September 4, 2015 RQ1: Recall Project Joda XMLSecurity Crystal Synoptic Dependencies Detected Ground Truth Ground Truth 2 4 18 1 ElectricTest Writers 15 3 15 10 Readers 121 103 39 117 ElectricTest Shared Resource Locations App 39 3 4 3 Library 12 15 19 14
@_jon_bell_ESEC/FSE September 4, 2015 RQ2: Overhead • Selected 10 projects with > 10 minutes of tests • Also included projects studied by Zhang et al, averaging < 10 seconds of testing • Previous exhaustive approach slowdown: >10300X • Previous heuristic approach slowdown: 31,882X • ElectricTest slowdown: 36X (885X faster than previous approach)
@_jon_bell_ESEC/FSE September 4, 2015 0X 1,000X 2,000X 3,000X 4,000X 5,000X 6,000X 7,000X 8,000X 9,000X 10,000X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- ElectricTest Slowdown Pairwise Slowdown *418,000X RQ2: Overhead On average, ElectricTest is 885X faster than running all tests pairwise Slowdown relative to a single test suite execution (lower is better)
@_jon_bell_ESEC/FSE September 4, 2015 0X 50X 100X 150X 200X 250X 300X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- RQ2: Overhead Average 36X A lot of fast running tests: Runtime dominated by pauses between tests (gc) Slowdown relative to a single test suite execution (lower is better)
@_jon_bell_ESEC/FSE September 4, 2015 0X 5X 10X 15X 20X 25X 30X camel& crunch& hazelcast& je/y.project& mongo5java5driver& mule& ne/y& spring5data5mongodb& tachyon& :tan& Safe Unsafe Speedup (higher is better) RQ3: Impact on Acceleration Average (Unsafe) 19x Average (Safe) 7x
@_jon_bell_ESEC/FSE September 4, 2015
@_jon_bell_ESEC/FSE September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan Bell, Gail Kaiser, Eric Melski and Mohan Dattatreya Columbia University & Electric Cloud, Inc

More Related Content

What's hot (14)

PDF
Test Drive Development
satya sudheer
 
PPTX
QUICKAR: Automatic Query Reformulation for Concept Location Using Crowdsource...
Masud Rahman
 
ODP
Java code coverage with JCov. Implementation details and use cases.
Alexandre (Shura) Iline
 
ODP
Agile analysis development
setitesuk
 
PPTX
Entaggle: an Agile Software Development Case Study
Elisabeth Hendrickson
 
PDF
Hack in Paris - Agnitio
Security Ninja
 
PPTX
DevSecCon Tel Aviv 2018 - End2End containers SSDLC by Vitaly Davidoff
DevSecCon
 
PPTX
Actor Concurrency Bugs: A Comprehensive Study on Symptoms, Root Causes, API U...
Raffi Khatchadourian
 
PDF
Static Analysis For Security and DevOps Happiness w/ Justin Collins
Sonatype
 
PDF
Test Driven Development
Dhaval Dalal
 
KEY
Php com con-2011
LB Denker
 
PDF
Agile Testing Overview
Elisabeth Hendrickson
 
PPTX
Seeding a Tree in a Gherkin
Paul Rohorzka
 
PPTX
DevSecCon Tel Aviv 2018 - Security learns to sprint by Tanya Janca
DevSecCon
 
Test Drive Development
satya sudheer
 
QUICKAR: Automatic Query Reformulation for Concept Location Using Crowdsource...
Masud Rahman
 
Java code coverage with JCov. Implementation details and use cases.
Alexandre (Shura) Iline
 
Agile analysis development
setitesuk
 
Entaggle: an Agile Software Development Case Study
Elisabeth Hendrickson
 
Hack in Paris - Agnitio
Security Ninja
 
DevSecCon Tel Aviv 2018 - End2End containers SSDLC by Vitaly Davidoff
DevSecCon
 
Actor Concurrency Bugs: A Comprehensive Study on Symptoms, Root Causes, API U...
Raffi Khatchadourian
 
Static Analysis For Security and DevOps Happiness w/ Justin Collins
Sonatype
 
Test Driven Development
Dhaval Dalal
 
Php com con-2011
LB Denker
 
Agile Testing Overview
Elisabeth Hendrickson
 
Seeding a Tree in a Gherkin
Paul Rohorzka
 
DevSecCon Tel Aviv 2018 - Security learns to sprint by Tanya Janca
DevSecCon
 

Similar to Efficient Dependency Detection for Safe Java Test Acceleration (20)

PDF
Test Dependencies and the Future of Build Acceleration
New York City College of Technology Computer Systems Technology Colloquium
 
ODP
Test antipatterns
Jiří Kiml
 
PDF
May: Automated Developer Testing: Achievements and Challenges
TriTAUG
 
PPTX
Advances in Unit Testing: Theory and Practice
Tao Xie
 
PDF
Unit testing [4] - Software Testing Techniques (CIS640)
Venkatesh Prasad Ranganath
 
PDF
Agile Software Testing the Agilogy Way
Jordi Pradel
 
PPTX
Principles and patterns for test driven development
Stephen Fuqua
 
PDF
(automatic) Testing: from business to university and back
David Rodenas
 
PDF
Testing, Learning and Professionalism — 20171214
David Rodenas
 
PDF
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Kevin Brockhoff
 
PPTX
The Test way
Mikhail Grinfeld
 
PDF
GTAC 2014: What lurks in test suites?
Patrick Lam
 
PPTX
ICST2016 Keynote - What got us here won't get us there: Trends and challenges...
Adam Porter
 
PDF
TDD Workshop UTN 2012
Facundo Farias
 
PPTX
TDD Training
Manuela Grindei
 
PPTX
RIA 06 & 07 - Unit Testing in Detail
Johannes Hoppe
 
PPTX
Automatic for the People
Andy Zaidman
 
PDF
Unit testing and scaffolding
Valerio Maggio
 
PDF
Unit testing - 9 design hints
Victor Rentea
 
PDF
Testing: ¿what, how, why?
David Rodenas
 
Test Dependencies and the Future of Build Acceleration
New York City College of Technology Computer Systems Technology Colloquium
 
Test antipatterns
Jiří Kiml
 
May: Automated Developer Testing: Achievements and Challenges
TriTAUG
 
Advances in Unit Testing: Theory and Practice
Tao Xie
 
Unit testing [4] - Software Testing Techniques (CIS640)
Venkatesh Prasad Ranganath
 
Agile Software Testing the Agilogy Way
Jordi Pradel
 
Principles and patterns for test driven development
Stephen Fuqua
 
(automatic) Testing: from business to university and back
David Rodenas
 
Testing, Learning and Professionalism — 20171214
David Rodenas
 
Deliver Faster with BDD/TDD - Designing Automated Tests That Don't Suck
Kevin Brockhoff
 
The Test way
Mikhail Grinfeld
 
GTAC 2014: What lurks in test suites?
Patrick Lam
 
ICST2016 Keynote - What got us here won't get us there: Trends and challenges...
Adam Porter
 
TDD Workshop UTN 2012
Facundo Farias
 
TDD Training
Manuela Grindei
 
RIA 06 & 07 - Unit Testing in Detail
Johannes Hoppe
 
Automatic for the People
Andy Zaidman
 
Unit testing and scaffolding
Valerio Maggio
 
Unit testing - 9 design hints
Victor Rentea
 
Testing: ¿what, how, why?
David Rodenas
 
Ad

More from jon_bell (7)

PDF
Replay without Recording of Production Bugs for Service Oriented Applications
jon_bell
 
PDF
A Large-Scale Study of Test Coverage Evolution
jon_bell
 
PDF
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
jon_bell
 
PDF
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMs
jon_bell
 
PDF
Unit Test Virtualization: Optimizing Testing Time
jon_bell
 
PDF
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
jon_bell
 
PDF
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
jon_bell
 
Replay without Recording of Production Bugs for Service Oriented Applications
jon_bell
 
A Large-Scale Study of Test Coverage Evolution
jon_bell
 
CROCHET - Checkpoint Rollback in JVM (ECOOP 2018)
jon_bell
 
Phosphor: Illuminating Dynamic Data Flow in Commodity JVMs
jon_bell
 
Unit Test Virtualization: Optimizing Testing Time
jon_bell
 
Chronicler: Lightweight Recording to Reproduce Field Failures (Presented at I...
jon_bell
 
A Large-Scale, Longitudinal Study of User Profiles in World of Warcraft
jon_bell
 
Ad

Recently uploaded (20)

PDF
Windsurf Meetup Ottawa 2025-07-12 - Planning Mode at Reliza.pdf
Pavel Shukhman
 
PDF
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
PDF
HCIP-Data Center Facility Deployment V2.0 Training Material (Without Remarks ...
mcastillo49
 
PDF
July Patch Tuesday
Ivanti
 
PPTX
✨Unleashing Collaboration: Salesforce Channels & Community Power in Patna!✨
SanjeetMishra29
 
PDF
Persuasive AI: risks and opportunities in the age of digital debate
Speck&Tech
 
PDF
Why Orbit Edge Tech is a Top Next JS Development Company in 2025
mahendraalaska08
 
PDF
Empowering Cloud Providers with Apache CloudStack and Stackbill
ShapeBlue
 
PDF
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PPTX
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
PDF
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
PDF
Women in Automation Presents: Reinventing Yourself — Bold Career Pivots That ...
DianaGray10
 
PPTX
MSP360 Backup Scheduling and Retention Best Practices.pptx
MSP360
 
PPTX
Building a Production-Ready Barts Health Secure Data Environment Tooling, Acc...
Barts Health
 
PDF
Chris Elwell Woburn, MA - Passionate About IT Innovation
Chris Elwell Woburn, MA
 
PDF
Apache CloudStack 201: Let's Design & Build an IaaS Cloud
ShapeBlue
 
PDF
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
PPTX
Top Managed Service Providers in Los Angeles
Captain IT
 
PDF
Français Patch Tuesday - Juillet
Ivanti
 
Windsurf Meetup Ottawa 2025-07-12 - Planning Mode at Reliza.pdf
Pavel Shukhman
 
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
HCIP-Data Center Facility Deployment V2.0 Training Material (Without Remarks ...
mcastillo49
 
July Patch Tuesday
Ivanti
 
✨Unleashing Collaboration: Salesforce Channels & Community Power in Patna!✨
SanjeetMishra29
 
Persuasive AI: risks and opportunities in the age of digital debate
Speck&Tech
 
Why Orbit Edge Tech is a Top Next JS Development Company in 2025
mahendraalaska08
 
Empowering Cloud Providers with Apache CloudStack and Stackbill
ShapeBlue
 
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
Women in Automation Presents: Reinventing Yourself — Bold Career Pivots That ...
DianaGray10
 
MSP360 Backup Scheduling and Retention Best Practices.pptx
MSP360
 
Building a Production-Ready Barts Health Secure Data Environment Tooling, Acc...
Barts Health
 
Chris Elwell Woburn, MA - Passionate About IT Innovation
Chris Elwell Woburn, MA
 
Apache CloudStack 201: Let's Design & Build an IaaS Cloud
ShapeBlue
 
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
Top Managed Service Providers in Los Angeles
Captain IT
 
Français Patch Tuesday - Juillet
Ivanti
 

Efficient Dependency Detection for Safe Java Test Acceleration

  • 1. @_jon_bell_ESEC/FSE September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan Bell, Gail Kaiser, Eric Melski and Mohan Dattatreya Columbia University & Electric Cloud, Inc
  • 2. @_jon_bell_ESEC/FSE September 4, 2015 Simplified Software Lifecycle Make changes to code Build & test Commit How long is too long of a build? 1 day? 6 hours? 10 minutes?
  • 3. @_jon_bell_ESEC/FSE September 4, 2015 Simplified Software Lifecycle • Compile sources • Generate documentation • Run tests • Package Make changes to code Build & test Commit
  • 4. @_jon_bell_ESEC/FSE September 4, 2015 Testing Dominates Build Times 20% 38% 41% 351 projects from GitHub Testing Other Compiling
  • 5. @_jon_bell_ESEC/FSE September 4, 2015 Testing Dominates Build Times 14% 26% 60% Projects taking > 10 minutes to build (69) Testing Other Compiling
  • 6. @_jon_bell_ESEC/FSE September 4, 2015 Testing Dominates Build Times 2%8% 90% Projects taking > 1 hour to build (8) Testing OtherCompiling
  • 7. Faster tests = Faster builds
  • 9. @_jon_bell_ESEC/FSE September 4, 2015 Regression Test Selection Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01] Changeset
  • 10. @_jon_bell_ESEC/FSE September 4, 2015 Regression Test Selection Test 1 Test 3 Test 8 Test 4 Gligoric et al. [ISSTA ’15], Orso et al. [FSE ’04], Harrold et al. [OOPSLA ’01] Changeset Tests not relevant to changeset: skipped
  • 11. @_jon_bell_ESEC/FSE September 4, 2015 Test Suite Minimization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 < /> Code Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et al. [IST ’98]; Wong et al. [ICSE ’95] and more
  • 12. @_jon_bell_ESEC/FSE September 4, 2015 Test Suite Minimization Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 13 Test 14 < /> Code Hao et al. [ICSE ’12]; Orso et al. [ICSE ’09]; Jeffrey et al. [TSE ’07]; Tallam et al. [PASTE ’05]; Jones et al. [TOSEM ’03]; Harrold et al. [TOSEM ’93]; Chen et al. [IST ’98]; Wong et al. [ICSE ’95] and more Redundant tests: removed
  • 13. @_jon_bell_ESEC/FSE September 4, 2015 Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
  • 14. @_jon_bell_ESEC/FSE September 4, 2015 Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14
  • 15. @_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code
  • 16. @_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External Factors
  • 17. @_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External Factors Test 1
  • 18. @_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External Factors Test 1Test 2
  • 19. @_jon_bell_ESEC/FSE September 4, 2015 Controlled Regression Testing Assumption Tests </> Code External Factors External Factors External Factors External Factors Test 1Test 2Test 3
  • 20. Not sound in practice
  • 21. @_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Write, Value “A” Test 4 Read Write, Value “B” Value: B Test 3 Read
  • 22. @_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies Test 1 Test 2 Test 3Test 4 Shared File
  • 23. @_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies Test 1 Test 2 Test 3Test 4Test 1 Test 2 Test 3 Shared File Write, Value “A” Test 4 Write, Value “B” Read, Expect Value “A” Value: B Read
  • 24. @_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies Test 1 Test 2 Test 3Test 4Test 1 Test 2 Test 3 Shared File Write, Value “A” Test 4 Write, Value “B” Read, Expect Value “A” Value: B A manifest test dependency Read
  • 25. @_jon_bell_ESEC/FSE September 4, 2015 Test Dependencies: A Clear and Present Danger • Really exist in practice (Zhang et al. found 96, Luo et al. found 14) • Hard to specify - if we could specify, would be safe to accelerate • Rarely: isolated (especially not in long building projects) • Existing technique to detect: combinatorially run tests [Zhang, et al ’14]
  • 26. @_jon_bell_ESEC/FSE September 4, 2015 Brute Force Dependency Detection • Looked at feasibility on 10 large open source test suites • Exhaustive approach: > 10300 years to find all dependencies • Pairwise approach: Average 31,882 executions of the entire test suite to find (incomplete) dependencies • Problem: How do we safely accelerate test suites in the presence of unknown dependencies?
  • 27. @_jon_bell_ESEC/FSE September 4, 2015 Manifest Test Dependencies • Definition: a data dependence between tests T1, T2 that results in the outcome of T2 changing • All manifest dependencies are data dependencies • Not all data dependencies are manifest dependencies
  • 28. @_jon_bell_ESEC/FSE September 4, 2015 Data Dependencies Test 1 Test 2 Test 3 Test 4Test 1 Test 2 Shared File Write, Value “A” Test 4 Read Write, Value “B” Test 3 Read Present Dependencies: Test 1 must run before 2 and 3 Test 4 must run after 2 and 3
  • 29. Key Insight: Dependencies don’t need to be precise, but must be sound
  • 30. @_jon_bell_ESEC/FSE September 4, 2015 Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Idle extra capacity
  • 31. @_jon_bell_ESEC/FSE September 4, 2015 Intuition Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Idle extra capacity A lot of dependencies, but still a 2x speedup
  • 32. @_jon_bell_ESEC/FSE September 4, 2015 ElectricTest - Detecting Data Dependencies in Java • Tracks in-memory dependencies (JVMTI plugin) • Tracks file and network dependencies (IO-Trace agent) • Implemented entirely within the Oracle or OpenJDK JVM, no specialized drivers, etc required • Captures stack traces when dependencies occur to support debugging • Generates dependency trees to enable sound test acceleration
  • 33. @_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies After each test, garbage collect; traverse heap to map objects back to static fields. Class A End of test 1
  • 34. @_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies After each test, garbage collect; traverse heap to map objects back to static fields. Class A End of test 1
  • 35. @_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies After each test, garbage collect; traverse heap to map objects back to static fields. Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield End of test 1
  • 36. @_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies During test execution, monitor accesses to existing objects Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield During Test 2
  • 37. @_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies During test execution, monitor accesses to existing objects Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield W2 W2 Write! During Test 2
  • 38. @_jon_bell_ESEC/FSE September 4, 2015 Identifying Heap Dependencies During test execution, monitor accesses to existing objects Class A W1 W1 W1 W1 W1 W1 W1 W1 W1 static field static field static field staticfield W2 W2 W1 Read! During Test 2 Dependency!
  • 39. @_jon_bell_ESEC/FSE September 4, 2015 Identifying External Dependencies Application under test Network Filesystem Log remote host address Log path
  • 40. ElectricTest enables sound exploitation of existing test acceleration techniques
  • 41. @_jon_bell_ESEC/FSE September 4, 2015 Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13 Test 14 Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Test 9 Test 10 Test 11 Test 12 Test 13 Test 14 Test 15
  • 42. @_jon_bell_ESEC/FSE September 4, 2015 Safe Test Parallelization Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14 Test 1 Test 2 Test 3 Test 8 Test 9 Test 10 Test 15 Test 4 Test 5 Test 6 Test 7 Test 11 Test 12 Test 13Test 14
  • 43. @_jon_bell_ESEC/FSE September 4, 2015 Safe Test Selection Test 15 Single test selected to be executed
  • 44. @_jon_bell_ESEC/FSE September 4, 2015 Safe Test Selection Test 15Test 1 Test 2 Test 3 Single test selected to be executed with its dependencies
  • 45. @_jon_bell_ESEC/FSE September 4, 2015 Understanding Dependencies • What should a developer do about test dependencies? • Might be intentional (e.g. cache shared state) • Might be unintentional but OK (e.g. loggers) • Might be unintentional and bad (e.g. bug)
  • 46. @_jon_bell_ESEC/FSE September 4, 2015 Assisting Debugging Debugging information reported by the previous technique Test 3 Test 1 Depends on
  • 47. @_jon_bell_ESEC/FSE September 4, 2015 Assisting Debugging Exception*in*thread*"main"* edu.columbia.cs.psl.testdepends.DependencyException:*Static*Field* ClassA.FieldA*member*was*previously*written*by*Test*1,*read*here.* ! at!edu.columbia.cs.psl.testdepends.test.Example$NestedExample.dragons(Example.java:20)! ! at!edu.columbia.cs.psl.testdepends.test.Example.moreMagic(Example.java:12)! ! at!edu.columbia.cs.psl.testdepends.test.Example.magic(Example.java:8)! ! at!edu.columbia.cs.psl.testdepends.test.Example.main(Example.java:15)! Test that wrote value Stack trace shows use Value that is read
  • 48. @_jon_bell_ESEC/FSE September 4, 2015 Evaluation • RQ1: Recall (accuracy) • RQ2: Runtime overhead • RQ3: Impact on acceleration
  • 49. @_jon_bell_ESEC/FSE September 4, 2015 RQ1: Recall Project Joda XMLSecurity Crystal Synoptic Dependencies Detected Ground Truth Ground Truth 2 4 18 1 ElectricTest Writers 15 3 15 10 Readers 121 103 39 117
  • 50. @_jon_bell_ESEC/FSE September 4, 2015 RQ1: Recall Project Joda XMLSecurity Crystal Synoptic Dependencies Detected Ground Truth Ground Truth 2 4 18 1 ElectricTest Writers 15 3 15 10 Readers 121 103 39 117 ElectricTest Shared Resource Locations App 39 3 4 3 Library 12 15 19 14
  • 51. @_jon_bell_ESEC/FSE September 4, 2015 RQ2: Overhead • Selected 10 projects with > 10 minutes of tests • Also included projects studied by Zhang et al, averaging < 10 seconds of testing • Previous exhaustive approach slowdown: >10300X • Previous heuristic approach slowdown: 31,882X • ElectricTest slowdown: 36X (885X faster than previous approach)
  • 52. @_jon_bell_ESEC/FSE September 4, 2015 0X 1,000X 2,000X 3,000X 4,000X 5,000X 6,000X 7,000X 8,000X 9,000X 10,000X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- ElectricTest Slowdown Pairwise Slowdown *418,000X RQ2: Overhead On average, ElectricTest is 885X faster than running all tests pairwise Slowdown relative to a single test suite execution (lower is better)
  • 53. @_jon_bell_ESEC/FSE September 4, 2015 0X 50X 100X 150X 200X 250X 300X mongo%java%driver- tachyon- spring%data%mongodb- xml-security- ne8y- je8y.project- crystal- crunch- camel- :tan- synop:c- hazelcast- mule- joda%:me- RQ2: Overhead Average 36X A lot of fast running tests: Runtime dominated by pauses between tests (gc) Slowdown relative to a single test suite execution (lower is better)
  • 54. @_jon_bell_ESEC/FSE September 4, 2015 0X 5X 10X 15X 20X 25X 30X camel& crunch& hazelcast& je/y.project& mongo5java5driver& mule& ne/y& spring5data5mongodb& tachyon& :tan& Safe Unsafe Speedup (higher is better) RQ3: Impact on Acceleration Average (Unsafe) 19x Average (Safe) 7x
  • 56. @_jon_bell_ESEC/FSE September 4, 2015 Efficient Dependency Detection for Safe Java Test Acceleration Jonathan Bell, Gail Kaiser, Eric Melski and Mohan Dattatreya Columbia University & Electric Cloud, Inc