JUnit PowerUp
- 1. JUnit PowerUP Practical Testing Tips James McGivern
- 2. About James Likes cats Talks fast Technical Evangelist Hates Marmite Mathematician turned Computer Scientist Lives in London
- 4. One Busy Day On Concurrent Island... Intro
- 6. Concurrency? concurrent - adjective 1. occurring or existing simultaneously or side by side: concurrent attacks by land, sea, and air. 2. acting in conjunction; co-operating. 3. having equal authority or jurisdiction. 4. accordant or agreeing. 5. tending to or intersecting at the same point: four concurrent lines.
- 7. The behaviour of a single threaded application is deterministic A multithreaded application may appear stochastic Concurrency introduces new problems: deadlocks resource starvation (e.g livelocks) race conditions contention • • • •
- 8. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 9. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 10. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 11. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 12. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 13. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 14. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 15. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 16. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 17. But Why? Consider the old singleton pattern getInstance() method: public Singleton getInstance() { if(INSTANCE == null) { INSTANCE = new Singleton(); } return INSTANCE; } and two threads: Thread A, Thread B
- 18. Each statement is an atomic block Each thread executes a non-negative number of atomic blocks Threads take turns but order is not guaranteed Given a number of threads t, and a group of statements s, the number of execution order permutations is given by: interleavings(t, s) = (ts)! t (s!)
- 21. public class FibonacciSequenceTest { FibonacciSequence fib = new FibonacciSequence(); @Test public void shouldOutputZeroWhenInputIsZero() { assertThat(fib.get(0), is(0)) } @Test public void shouldOutputOneWhenInputIsOne() { assertThat(fib.get(1), is(1)); } @Test public void shouldCalculateNthNumber(){ for(int i = 2; i <= 10; i++) { int x = fib.get(i-1); int y = fib.get(i-2); int sum = x + y; assertThat(fib.calculate(i), is(sum); } } }
- 22. Cost/Benefit Ratio (TDD) Unit tests are executable specifications of the code Unit (+ integration tests) will never find all the bugs Writing tests takes time Time is limited Which tests should I write and which should I forgo?
- 23. JUnit Runners Bundled Suite, Parameterized • • Theories, Categories, Enclosed Popular 3rd party runners Mockito, PowerMock(ito) • Custom JBehave, Spock • • dynamic tests?
- 24. Custom Runner public abstract class Runner implements Describable { public Runner(Class<?> testClass){...} public abstract Description getDescription(); public abstract void run(RunNotifier n); public int testCount() { return getDescription().testCount(); } }
- 26. RunNotifier fireTestStarted(Description d) fireTestFinished(Description d) fireTestFailure(Failure f) fireTestIgnored(Description d) fireTestAssumptionFailed(Failure f)
- 27. Warning • A very coarse way of modifying test execution • Even when extending from BlockJUnit4ClassRunner there is a degree of complex code • Runners can not be composed e.g. @RunWith(MockitoJUnitRunner.class} @RunWith(SpringJUnit4ClassRunner.class) public class SomeTest {...} @RunWith({MockitoRunner.class, SpringJUnit4ClassRunner.class}) public class SomeTest {...}
- 29. Rules can: Read/Write test metadata Modify the test before execution Modify the result after execution Suite vs Class == @ClassRule vs @MethodRule
- 30. public interface TestRule { Statement apply(Statement base, Description description); } public class MockRule implements TestRule { private final Object target; public MockRule(Object target) { this.target = target; } public Statement apply(final Statement base, Description description) { return new Statement() { public void evaluate() throws Throwable { MockitoAnnotations.initMocks(target); base.evaluate(); } }; } }
- 33. static class VolatileInt { volatile int num = 0; } public void shouldIncrementCounter() { final int count = 32 * 1000; final int nThreads = 64; ExecutorService es = Executors.newFixedThreadPool(nThreads); final VolatileInt vi = new VolatileInt(); for (int i = 0; i < nThreads; i++) { es.submit(new Runnable() { public void run() { for (int j = 0; j < count; j += nThreads) vi.num++; } }); es.shutdown(); es.awaitTermination(10, TimeUnit.SECONDS); assertEquals(count, vi.num); } http://vanillajava.blogspot.co.uk/2011/08/why-testing-code-for-thread-safety-is.html
- 34. Weapons Guide • Java Concurrency in Practice - Brian Goetz • Java Performance - Charlie Hunt • Effective Unit Testing: A guide for Java developers Lasse Koskela • Programming Concurrency on the JVM -Venkat Subramaniam
- 39. A Simple DSL No more Thread.sleep(), or while loops await().until( new Callable<Boolean>() { public Boolean call() throws Exception { return userRepository.size() == 1; } }; )
- 40. The previous example was not particularly re-usable. Let’s fix that! await().until(sizeOf(repository), equalTo(1)); where private Callable<Integer> sizeOf(Collection c){ return new Callable<Integer>() { public Boolean call() throws Exception { return c.size(); } }; }
- 41. Advanced Waiting • Callable is still a lot of boiler plate... await().untilCall(to(repository).size(), equalTo(3)); • Using reflection await().until( fieldIn(repository).ofType(int.class) .andWithName(“size”), equalTo(3) ); • Polling with() .pollInterval(ONE_HUNDERED_MILLISECONDS) .and().with().pollDelay(20, MILLISECONDS) .await("user registration").until( userStatus(), equalTo(REGISTERED));
- 44. Race Conditions A race condition is a situation in which two or more threads or processes are reading or writing some shared data, and the final result depends on the timing of how the threads are scheduled. public class Counter { protected long count = 0; public void add(long value){ this.count = this.count + value; } }
- 46. public class ConcurrentHashMapTest { ConcurrentMap<String, Integer> map; @ThreadedBefore public void before() { map = new ConcurrentHashMap(); } @ThreadedMain public void mainThread() { map.putIfAbsent("A", 1); } @ThreadedSecondary public void secondThread() { map.putIfAbsent("A", 2); } } @ThreadedAfter public void after() { assertEquals(map.get(“A”), 1); }
- 47. Works by instrumenting bytecode at runtime Does not integrate with JUnit but can be embedded @Test public void testThreading() { new AnnotatedTestRunner() .runTests( this.getClass(), ConcurrentHashMapTest.class ); } Has fine-grained control over breakpoints/code position Documentation is ok but not a lot on in-depth material PROJECT STATUS UNKNOWN, MAYBE INACTIVE
- 50. A recurrent producer-consumer information processing network for anomaly detection
- 52. Java Path Finder http://babelfish.arc.nasa.gov/trac/jpf JPF created by NASA Open-sourced in 2005 Is a JVM written in Java that runs on the JVM Primarily used as a model checker for concurrent programs, e.g deadlocks, race conditions, NPEs Very powerful Not very usable (e.g. limited CI support)
- 53. Byteman PowerUP
- 54. Java Bytecode Manipulating Agent - Byteman Uses java.lang.intrument as an agent Can be used for: fault injection testing, tracing during test and in production, an alternative AOP (e.g. cross-cutting logging) Using instrumentation in tests means: fewer mocks less boilerplate to generate abnormal senarios
- 55. Why Byteman? AOP is powerful but complex: • code is targeted indirectly rather than at the class/ method declaration site • AOP definition languages can be a bit odd/hard to work with • Impact at runtime is sometimes significant to application performance due to pointcut definitions • not always trivial to turn-off, alter, or remove advice
- 56. EBCA Rules A Byteman rule consists of: • Event - the class/interface/method target • Binding - initialise rule values • Condition - a boolean expression (optional) • Action - some Java code that may return or throw (can not break contracts) Rules may have state, i.e 1st invocation do A, 2nd invocation do B, etc.
- 57. Fault Injection RULE simulate exception from Executor INTERFACE ^java.util.Executor METHOD execute AT ENTRY IF callerEquals("ServiceInstanceImpl.execute", true) DO traceln(“Throwing exception in execute”); THROW new java.util.concurrent.RejectedExecutionException(); ENDRULE
- 58. In-built Rules Tracing traceOpen, traceClose, traceln, traceStack Shared Rule State flag, clear, flagged, countDown, incrementCounter Synchronization waitFor, signalWake, rendezvous, delay Timing createTimer, getElapsedTime, resetTimer Call Stack Checking callerEquals, callerMatches Recursive Triggering disable/enableTriggering
- 59. BMUnit @RunWith(BMUnitRunner.class) @BMScript(value="traceRules", dir="scripts") class DBTests1 { @Test @BMRule( className="UserRepository”, methodName="findByName", condition= "$1.getName().contains("James")", action="THROW new UserNotFoundException") public void shouldErrorIfNoUserFound() { ... } }
- 62. ConcurrentUnit Website: https://github.com/jhalterman/concurrentunit JUnit addon library Comprised of: • A thread co-ordinator Waiter • Allows assertions to be made in worker threads • Waiter collects and returns outcomes of assertions
- 63. FindBugs Website: http://findbugs.sourceforge.net Integrates with Maven, Gradle, Sonar, etc Multithreaded correctness group: • Inconsistent synchronization • Field not guarded against concurrent access • Method calls Thread.sleep() with a lock held • Creation of ScheduledThreadPoolExecutor with zero core threads
- 64. Freud Website: https://github.com/LMAX-Exchange/freud A framework for writing static analysis tests Lacks good documentation except code examples Has rules for: ✦ ✦ ✦ ✦ Java sources Java class objects. (i.e analysing the java.lang.Class object) Java class files (i.e analysing the ".class" file) Spring framework XML configuration files
- 65. Summary Scores
- 67. Lv.0 - Testing Tutorial • TDD is your friend • Prefer rules over runners • Don’t be afraid to write custom rules and runners • Ensure you spend your time on the important tests
- 68. Lv.1 - The Thread Challenge • Modelling real world behaviour of your application is hard • Don’t assume your tests are exhaustively testing the scenarios • Behaviour depends heavily on the system (e.g. load) • Know your stuff - read read read
- 69. Lv2. Choreographer’s Workshop • It is essential to test the co-ordinated behaviour of threads • If threads are not sharing resources we don’t need to care about synchronisation between thread • Awaitility allows us to check that some end state if finally reached (within the given time)
- 70. Lv.3 - The Shared Resource Contest • When a resource is shared between threads new problem cases can arise, e.g. races • Often the problem can be simplified to the case of 2 threads • To test the orders of execution of 2 threads we can use ThreadWeaver
- 71. Lv.4 - Princess Protection HQ • Many problems can’t be reduced to a 2 thread model • JPF can provide some automated analysis of concurrency issues • Byteman allows us to take control of the whole application • Both are very heavy weight and should not be
- 74. Credits
- 79. JUnit PowerUP Practical Testing Tips James McGivern