Reducing Build Time

Editor's Notes

• #4: * Green icon represents a stable green CI build * Orange icon represents a CI build in progress * Red icon represents a broken CI build
• #7: * It is good to immediately know where you stand: whether all is good or something is broken * That’d enable you to fix something almost as soon as you break it
• #10: * High functional coverage implies that almost all functionality of the system is tested and any error introduced would be caught * Short CI build time implies that quick feedback is in place and any error introduced in the covered functionality would be immediately highlighted
• #16: * A website with content that comes from various sources, with a focus on creating an online community of people with similar interests
• #21: * Failures not caused by programmatic errors: possible reasons could include AJAX response delays, unpredictable browser responsiveness, network slowness, etc
• #22: * If a build is broken on what appears to be non-deterministic failure, someone will have to manually trigger another build in hope of the error not reoccurring * A retriggered build will take long again, further delaying feedback
• #24: * One main process will fork multiple sub-processes in parallel, each running only a subset of the original collection of test scenarios * When all sub-processes complete, the union of their results will be the final result of the entire test run * If the sub-processes are run on the same machine, a powerful multicore machine is desirable * Another approach is to run the sub-processes on different machines on a grid
• #26: * Chart of build times vs build numbers: the drastic drop is when parallelization was implemented * Slight variances in build times, including minor drops, also occur based on memory and CPU load at the time
• #28: * Used a Ruby library called parallel_tests to fork multiple Ruby processes, each running a subset of Cucumber features * This also involved coming up with custom Ruby rake tasks, including pre-steps and post-steps * Instead of using parallel_tests to parallelize on the same machine, Selenium Grid is an alternate option to leverage grid computing
• #29: * Capture each forked process’s results independently * Generate consolidated HTML report as after-step when all forked processes finish
• #30: * Rails’ database configuration file (database.yml) was modified to suffix process number to the name of the database
• #31: * ERB (Embedded Ruby) in YAML file to append an environment variable which has a different value for each forked process
• #32: * Sunspot is the Ruby library we used to interact with Solr, an open source search platform * Sunspot’s configuration file (sunspot.yml) was modified to suffix process number to the paths that Solr was to use * Monkey-patched Sunspot code to allow parameterization in sunspot.yml
• #33: * ERB (Embedded Ruby) in YAML file to append an environment variable which has a different value for each forked process
• #34: * Although each sub-process starts its independent Firefox instance, Selenium uses a shared set of ephemeral ports to communicate with the browsers * Contention occurs for these ephemeral ports which get locked when in use for a particular Firefox instance * Monkey-patched Capybara’s Selenium driver code to retry communicating with Firefox if an attempt fails
• #35: * A particular machine can only allow so much parallelization based on its hardware specifications * Beyond a certain number of sub-processes, the non-deterministic failures will increase * After a few tries, we stabilized on having 6 concurrent sub-processes on the machine we were using
• #36: * Cucumber provides a rerun mode that retries all failed tests one more time before deciding the final status of the test suite run * Customized a Ruby rake task to invoke Cucumber’s rerun sequentially, after the parallel results come in
• #38: * We stubbed some external calls to return known values to cut down on communication delays * Used a Ruby library called WebMock * Examples: authentication servers, email servers, Facebook, Twitter, etc
• #39: * It helps to reduce the number of independent tests without reducing the functionality coverage * Example: if during sign up, email and zipcode are mandatory, they don’t need independent tests; there can be one test that begins by leaving email and zipcode empty, verifies that error messages are displayed for email and zipcode, and then adds email and zipcode, and proceeds with valid sign up scenarios * The above reduces browser clearing steps, visiting a certain page repeatedly without it contributing to the test steps, and so on
• #41: * There are multiple benefits of having quick feedback; implementing parallelization is one approach to achieve that * A complementary approach is to introspect and work on improving performance and responsiveness of the app itself, reducing build time by reducing waiting time * Being aware of build times and improving the same continuously is a good habit; even with parallelization in place, as more functionality gets added, build times will creep up