![Snippet of HAR format{"log": {"version": "1.1", "creator": { "name": "Firebug", "version": "1.7.0" }, "browser": { "name": "Firefox", "version": "4.0" }, "pages": [ { "startedDateTime": "2011-03-31T16:56:50.455-07:00", "id": "page_62901", "title": "Google", "pageTimings": { "onContentLoad": 431, "onLoad": 3148 } } ],"entries": [ { "pageref": "page_62901", "startedDateTime": "2011-03-31T16:56:50.455-07:00", "time": 250, "request": { "method": "GET", "url": "http://www.google.com/", "httpVersion": "HTTP/1.1", "cookies": [ { "name": "PREF", "value": "ID" },](https://image.slidesharecdn.com/seleniumconference-upload-110415121343-phpapp01/85/Performance-Metrics-in-a-Day-with-Selenium-4-320.jpg)
Performance Metrics in a Day with Selenium
- 1. Performance Metrics In A DayShane HenderMark Watson
- 2. AgendaGoal is to show how you can enhance your automation tests to gather performance metricsKeep the short presentation as practical as possible with enough code snippets to give you a starting point
- 3. One more thing…HTTP Archive (HAR)What?UTF8 Text based JSON encoding to represent performance dataWhy?Many libraries to convert from JSON to objectsMore tools allowing import/export/view HARBecoming the standard format
- 4. Snippet of HAR format{"log": {"version": "1.1", "creator": { "name": "Firebug", "version": "1.7.0" }, "browser": { "name": "Firefox", "version": "4.0" }, "pages": [ { "startedDateTime": "2011-03-31T16:56:50.455-07:00", "id": "page_62901", "title": "Google", "pageTimings": { "onContentLoad": 431, "onLoad": 3148 } } ],"entries": [ { "pageref": "page_62901", "startedDateTime": "2011-03-31T16:56:50.455-07:00", "time": 250, "request": { "method": "GET", "url": "http://www.google.com/", "httpVersion": "HTTP/1.1", "cookies": [ { "name": "PREF", "value": "ID" },
- 5. HAR Viewershttp://www.softwareishard.com/har/viewer/Fiddler UI can import HAR filesShowSlowwebapp can be used to archive and view HAR files
- 6. Which Metrics?Overall page load timeDOM loading/interactive/complete, browser 1st render, …Per-item timingsDNS, SSL connect, time to first byte, total time to download bytes, …Headers, status codes, and contentFor a more detailed picture of the reasons for the page performanceCan help with debugging performance, e.g. were items compressed, or not being loaded dynamically, or certain items not being prioritized for above-the-fold rendering
- 7. Methods for gathering metricsSetting your own timings in test codeUsing the new ‘Navigation.Timings’ or Web Timings standard built into browsers Using browser plugins that report back the timings to you, e.g. WebPageTestPer HTTP request logs via Selenium’s built-in proxy Routing the browser traffic through a local proxy and gathering the statistics from there.Network traffic capture
- 8. Method 1: Own TimingsWebDriver driver = newChromeDriver();Date start = new Date();driver.get("http://www.webmetrics.com");Date end = new Date();
- 9. Method 2: Web TimingsCurrently Chrome and IE9 supported, coming soon for Firefoxhttp://w3c-test.org/webperf/specs/NavigationTiming/WebDriver driver = newChromeDriver();// A "base url", used by selenium to resolve relative URLs StringbaseUrl = "http://www.webmetrics.com"; driver.get(baseUrl); JavascriptExecutorjs = (JavascriptExecutor)driver; LongloadEventEnd= (Long) js.executeScript("returnwindow.performance.timing.loadEventEnd");LongnavigationStart= (Long) js.executeScript("returnwindow.performance.timing.navigationStart");Long answerToAllProblems =loadEventEnd - navigationStart;
- 10. Method 2: Web TimingsCould also modify this by using your own JavaScript loaded into the page to set timingsAlso would work in combination with the already set Web TimingsE.g. Set a Date().getTime() variable when some dynamically loaded content is loaded into the DOMLongperceivedLoadTime = (Long) js.executeScript("returnelementInsertTime – window.performance.timing.navigationStart");
- 11. Unfortunately it doesn't give timings per item downloaded, e.g. images, css, js, ....
- 12. Method 3: Browser PluginsTypically can give compute time metricsJavascript execution timeCPU usageRender timesIE8 - WebPageTestCPU usageVideo captureFirefox - Firebug Net Panel + NetExporthttps://github.com/lightbody/browsermob-page-perfhttps://github.com/AutomatedTester/AutomatedTester.PagePerf.gitDynaTrace browser plugin (FireFox/IE on Windows)Good breakdown of stats (Javascript execution times, CSS times, render, 'first impression' time).http://ajax.dynatrace.com/ajax/en/Chrome – some export from the Developer Tools Network Panel?
- 13. FirefoxProfilep=newFirefoxProfile();try{p.addExtension(newFile("c:/firebug-1.6.0.xpi"));p.addExtension(newFile("c:/netExport-0.8b9.xpi"));p.addExtension(newFile("c:/firestarter-0.1.a5.xpi"));}catch(IOExceptione){thrownewRuntimeException(“Failed to load extensions:",e);}p.setPreference("extensions.firebug.netexport.autoExportActive",true);//p.setPreference("extensions.firebug.netexport.defaultLogDir", "c:/");p.setPreference("extensions.firebug.onByDefault",true);p.setPreference("extensions.firebug.defaultPanelName","net");p.setPreference("extensions.firebug.net.enableSites",true);p.setPreference("extensions.firebug.previousPlacement",1);driver=newFirefoxDriver(p);Method 3: Example - Firebug + NetExport
- 14. Method 4: Example - Firebug + NetExport
- 15. Method 4: Use a ProxyUse built in Selenium 1 proxyUse a 3rd party proxy library Many available, few capture metrics in a convenient wayTwo good ones:BrowserMob ProxyFiddler
- 16. Method 4: Selenium built-in ProxyUse API Selenium.captureNetworkTraffic()Selenium selenium = newDefaultSelenium("localhost", 4444,"*firefox", "http://www.webmetrics.com"); selenium.start("captureNetworkTraffic=true");selenium.open("http://www.webmetrics.com"); Stringjson = selenium.captureNetworkTraffic("json"); selenium.stop();
- 17. Method 4: Selenium built-in ProxyNot in HTTP Archive (HAR) format, but does report:HTTP status code, URL and HTTP methodRequest & response headersOverall request->response timingBytes downloadedWorks with Chrome and FirefoxDoesn’t work for WebDriver tests.Still work with Selenium 2 RC, but only if send jobs via Selenese API.Not much control over the proxy
- 18. Method 4: Advantages of using a ProxyTesting benefits beyond performance monitoringBlacklisting/whitelisting URLsComparing page load times of site with/without external contentNot hitting analytics/advertising contentSimulating external content being offlineURL rewritesRedirect production URLs back to staging/test environmentPretend to be production as far as the browser is concerned (e.g. for cookies)Make sure ad/metrics requests are being made
- 19. Method 4: Advantages of using a ProxyHeader changesSet the user agent manually to test different browser behaviorAuto authenticationWait until all content is downloadedHTTP idle for X secondsLimit bandwidth
- 20. Method 4: BrowserMob ProxyOpen source cross platform proxy written in Java.HTTP Archive supportFriendly APISource code available:https://github.com/lightbody/browsermob-proxy
- 21. Method 4: BrowserMob ProxyExport HAR Sampleimport org.browsermob.proxy.ProxyServer;ProxyServer proxy = new ProxyServer(9090);proxy.start();Proxy mobProxy = new Proxy().setHttpProxy("localhost:9090");DesiredCapabilities caps = new DesiredCapabilities();caps.setCapability("proxy", mobProxy);FirefoxDriver driver = new FirefoxDriver(caps);
- 22. Method 4: BrowserMob ProxyRun test, denoting pages in HARproxy.newHar("Yahoo");driver.get("http://yahoo.com");proxy.endPage();proxy.newPage("CNN");driver.get("http://cnn.com");proxy.endPage();
- 23. Method 4: BrowserMob ProxyWrite out HTTP Archive fileproxy.getHar().writeTo(newFile("test.har"));
- 24. Method 4: BrowserMob ProxyBlacklist/Whitelist URLsproxy.blacklistRequests(regex, responseCode)proxy.whitelistRequests(regex, responseCode)Redirecting URLsproxy.rewriteUrl(regex, replace)Limit Bandwidthproxy.setDownstreamKbps(kbps)proxy.setUpstreamKbps(kbps)
- 25. Method 4: BrowserMob ProxyWhen to useCross platformJavaCan set the browser proxy
- 26. Method 4: Proxy - FiddlerCoreFiddler is an application for viewing HTTP traffic on Windows.Works with Chrome, FireFox, Internet Explorer.Fiddler Application is built on FiddlerCore.NET libraryAllows extensive programmatic control over the proxyCaptures HTTP timingsAllows request/responses to be intercepted and modifiedConfigures itself as default Windows proxy (supports proxy chaining)Can decrypt SSL traffic
- 27. To start the proxy and register it as the default system wide proxy.Each HTTP transaction can then be recorded as follows:Method 4: Proxy - FiddlerCore// Initialize the proxyFiddler.FiddlerApplication.Startup( 8877, FiddlerCoreStartupFlags.Default);var items = new List<Fiddler.Session>(); Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.SessionoS){items.Add(oS);};
- 28. Method 4: Proxy - FiddlerCoreRun Selenium test as normalAs each item is downloaded it will be added to the ’items’ var in previous slide.string baseUrl = "http://www.webmetrics.com";varwebDriver =newOpenQA.Selenium.Firefox.FirefoxDriver();var selenium = newSelenium.WebDriverBackedSelenium (webDriver, baseUrl);selenium.Start();selenium.Open(baseUrl);selenium.WaitForPageToLoad("30000");selenium.Stop();
- 29. Method 4: Proxy (Fiddler -> HAR)Using the HAR Exporter to convert the sessions to HARAdd FiddlerCore-BasicFormats.dll to the project references and load the assembly:https://www.fiddler2.com/dl/FiddlerCore-BasicFormats.zipString exePath = Assembly.GetExecutingAssembly().Location;String path = Path.Combine(Path.GetDirectoryName(exePath), @"FiddlerCore-BasicFormats.dll");FiddlerApplication.oTranscoders.ImportTranscoders(path);
- 30. Method 4: Proxy (Fiddler -> HAR)Finally, export the HAR to a file:varoExportOptions = new Dictionary<string, object>();string filename = @"output.har”;oExportOptions.Add("Filename", filename);Fiddler.FiddlerApplication.DoExport("HTTPArchive v1.2", sessions, oExportOptions, null);
- 31. Method 4: Proxy Bonus FeaturesModifying HTTP requestsFiddler.FiddlerApplication.BeforeRequest +=delegate(Fiddler.SessionoS){// Ignore requests made to the main site.RegexmatchUrl = newRegex("webmetrics\\.com”);if (matchUrl.IsMatch(oS.fullUrl)) {oS.utilCreateResponseAndBypassServer();oS.responseCode = 200; }};In the same way responses from the server can be modifiedFiddler.FiddlerApplication.BeforeResponse
- 32. Method 4: Proxy – FiddlerCoreWhen to useWindows OnlyCross browser.NET
- 33. Method 5: Wire HTTP captureCaptured network traffic can be converted to HAROn Unix/OSX use tcpdumptcpdump -i en0 -n -s 0 -wtraffic.pcapOn Windows use WinPCapwinpcap -i 0 -n -s 0 -wtraffic.pcapThen use pcap2har to do the conversion:main.pytraffic.pcap http-requests.harPcap2harhttps://github.com/andrewf/pcap2har
- 34. Method 5: Wire HTTP captureCaptures allHTTP trafficTimings based on actual network trafficNo interference from proxyNo extra network delays/context switches talking to the proxyBrowsers sometimes behave differently when talking to a proxy
- 35. SummaryDiscussed 5 methods for recording performance metricsUse timers around selenium commandsUse WebTimings JavaScript APIFirebug & NetExportProxyUse the built-in Selenium proxyUse a 3rd party proxySniff network traffic
- 36. LinksBrowserMob proxyhttps://github.com/lightbody/browsermob-proxyFiddler Cookbookhttp://www.fiddler2.com/fiddler/dev/ScriptSamples.aspExamples from this talkhttps://github.com/watsonmw/selenium-pageloadmetrics
Editor's Notes
- #2: Welcome to our talk about getting performance metrics in a day
- #3: Outline the talk, what is the takeawayRun your standard unit tests (Junit, minitest, ….) but put in performance testing as well
- #4: Becoming the standard mechanism to store/dump performance data
- #7: Lots of text on this slide, we’ll keep it more entetaining laterOverall page load time= The obvious starting point and usually the easiest to acquire
- #8: Setting your own timings – obvious, but is not always very accurate if you are in a sequence of steps and capture page transitions and other delays Very coarse, incorporates latency/setup/teardown delays in the times reported, e.g. it sometimes takes a second or 2 for an IE window to become responsiveWeb Timings == Very simple at the moment, but gives very accurate overall stats, like initial connection time, DOM ready, and total request->response times.Browser plugins == Accurate picture of what the browser is doing, but not so straightforward to incorporate into your code.Proxy == Gives you very good timings for http traffic but can’t give you timings like render time or DOM ready events.Network traffic capture tools like WinPCAP, or wireshark
- #14: Doesn't seem to work currently in FF4Will have to parse the HAR format if you just want the basic timing out of it
- #16: Going to go over:- How to grab metrics using a proxy How using a proxy can be beneficial for testing in general
- #28: (Fiddler refers to thehttp transaction as 'sessions')
- #33: Documentation spread out over blogs, fiddler google groups, and fiddler extensions pageAPI is little difficult