Skip to content

Instantly share code, notes, and snippets.

@connorjclark

connorjclark/README.md

Last active May 15, 2020 18:02
Show Gist options
  • Save connorjclark/8afe673d4e7c6e17204834a256e7caf1 to your computer and use it in GitHub Desktop.
Save connorjclark/8afe673d4e7c6e17204834a256e7caf1 to your computer and use it in GitHub Desktop.
Download ZIP
Lighthouse Scoring Changes 5.6.0 -> 6.0
Raw
README.md

Results

name url mean mean Δ
performance https://www.partypoker.com/ 33.8 -> 48.6 14.8
performance https://www.vevo.com/ 17.6 -> 30.8 13.2
performance https://www.ebs.in/IPS/ 12.4 -> 25.4 13
performance https://www.ifeng.com/ 37.2 -> 49.4 12.2
performance https://www.thefreedictionary.com/ 29.2 -> 38.2 9
performance https://www.tianya.cn/m/ 29.8 -> 38.4 8.6
performance https://depositfiles.com/ 38.8 -> 47 8.2
performance https://www.imageshack.us/login 69.8 -> 78 8.2
performance https://www.att.com/ 36 -> 43.2 7.2
performance https://www.cnet.com/ 25 -> 32.2 7.2
performance https://www.ebay.com/ 44.2 -> 51 6.8
performance https://www.56.com/ 35.4 -> 41.8 6.4
performance https://noclip.website/ 24 -> 30.2 6.2
performance https://www.mlb.com/ 18.8 -> 23.4 4.6
performance https://www.foxnews.com/ 18 -> 22.2 4.2
performance https://en.softonic.com/ 30.8 -> 34.8 4
performance https://www.rakuten.co.jp/ 17 -> 20.6 3.6
performance https://www.codewars.com/ 76.4 -> 78.6 2.2
performance https://www.verizonwireless.com/ 32 -> 33.8 1.8
performance https://sfbay.craigslist.org/ 96.6 -> 98 1.4
performance https://www.bing.com/ 98.6 -> 99 0.4
performance https://m.huffpost.com/ 18.2 -> 18.2 0
performance https://www.linkedin.com/ 96.2 -> 96.2 0
performance https://www.twitpic.com/ 100 -> 100 0
performance https://mail.ru/ 15.8 -> 15 -0.8
performance https://www.4shared.com/ 56 -> 55.2 -0.8
performance https://www.wikipedia.org/ 100 -> 99.2 -0.8
performance https://www.blogger.com/about/ 99 -> 97.8 -1.2
performance https://www.dawn.com/ 28 -> 26.4 -1.6
performance https://www8.hp.com/us/en/home.html 14.4 -> 12.6 -1.8
performance https://m.facebook.com/ 85.8 -> 83.8 -2
performance https://weather.com/ 26.2 -> 24.2 -2
performance https://m.youdao.com/ 99.2 -> 97 -2.2
performance https://www.espn.com/ 17.4 -> 15 -2.4
performance https://www.thestar.com.my/ 9.6 -> 6.8 -2.8
performance https://www.netflix.com/ 52 -> 48.6 -3.4
performance https://m.hexun.com/ 44 -> 40.2 -3.8
performance https://www.hulu.com/welcome 39.4 -> 35.2 -4.2
performance https://www.msn.com/ 57.2 -> 52.6 -4.6
performance http://www.zol.com.cn/ 65 -> 59.4 -5.6
performance https://www.so-net.ne.jp/m/ 22.2 -> 16 -6.2
performance https://www.addthis.com/ 62.6 -> 56.2 -6.4
performance https://www.orange.fr/portail 24 -> 17.6 -6.4
performance https://www.java.com/en/ 96 -> 89.2 -6.8
performance https://gm.58.com/glsanfrancisco-sl/ 83.4 -> 76.4 -7
performance https://www.onet.pl/ 46.6 -> 39 -7.6
performance https://birdsarentreal.com/ 27.4 -> 19.2 -8.2
performance https://m.sogou.com/ 62.4 -> 54.2 -8.2
performance https://www.deviantart.com/ 37.2 -> 29 -8.2
performance https://www.hatena.ne.jp/ 38.6 -> 30.4 -8.2
performance https://m.mop.com/ 68.2 -> 59.2 -9
performance https://wap.sogou.com/ 62.8 -> 53.8 -9
performance https://www.nih.gov/ 61.6 -> 52.6 -9
performance https://www.metacafe.com/ 52 -> 42.2 -9.8
performance https://mobile.twitter.com/ 38.2 -> 28 -10.2
performance https://www.gmx.net/ 50 -> 39.2 -10.8
performance https://www.flipkart.com/ 36.2 -> 25.2 -11
performance https://www.ocn.ne.jp/ 41 -> 29.8 -11.2
performance https://www.nokia.com/ 56.2 -> 44.2 -12
performance https://www.reddit.com/ 37.2 -> 25 -12.2
performance https://www.scribd.com/ 56.4 -> 43 -13.4
performance https://stripe.com/docs 54.4 -> 40.6 -13.8
performance https://www.domaintools.com/ 37 -> 23 -14
performance https://www.alexa.com/ 52 -> 37.4 -14.6
performance https://www.tumblr.com/ 62.4 -> 47.4 -15
performance https://www.mozilla.org/en-US/ 92.6 -> 76.6 -16
performance https://www.amazon.co.jp/ 51 -> 34 -17
performance https://www.skype.com/en/ 77.8 -> 60.8 -17
performance https://www.ning.com/ 52.2 -> 32.6 -19.6
performance https://www.typepad.com/ 80.6 -> 60.2 -20.4
performance https://www.mgid.com/ru 56.8 -> 35.6 -21.2
performance https://www.shopping.com/ 69.4 -> 46.4 -23
performance https://en-maktoob.yahoo.com/?p=xa 70.6 -> 46 -24.6
performance https://www.irs.gov/ 68.6 -> 44 -24.6
performance https://www.instagram.com/ 80 -> 51.4 -28.6

Steps to Reproduce

We run Lighthouse 5 times on a set of URLs and compare the median runs to each other. Lighthouse 6.0 is used to collect artifacts, and those artifacts are re-used with 5.6.0, so there is no page variance. Only scoring changes are captured.

  1. Run Lighthouse on GCP (it takes too long locally and my machine will fall asleep without my attention) [1]
  2. Copy results to timings-data/urls-6.0
  3. Copy the same results to timings-data/urls-5.6.0, but delete all the LHR json files. [2]
  4. Run the "collect" command for the 5.6.0 dataset. The artifacts are present, but since we deleted the LHRs it will re-run the auditting step of Lighthouse (fast) [3]
  5. run the comparison command

[1]

# Run Lighthouse on GCP
# I tweaked an existing script we have to do a bulk run: https://github.com/GoogleChrome/lighthouse/blob/collect-gcp/lighthouse-core/scripts/lantern/collect/gcp-create-and-run.sh
git checkout collect-gcp # branched off dc5bb65
sh lighthouse-core/scripts/lantern/collect/gcp-create-and-run.sh

[2]

mkdir timings-data; cd timings-data
unzip ~/lh-6-data.zip # See Resources to download the same data
cp -r urls-6.0 urls-5.6.0
find urls-5.6.0 -name '*lhr*' -delete
cd .. # return to LH root

[3]

git checkout v5.6.0
git checkout origin/collect-gcp -- lighthouse-core/scripts/compare-runs.js lighthouse-core/scripts/lantern/collect/common.js lighthouse-core/scripts/lantern/collect/urls.js
yarn
urls=(`node -e 'console.log(require("./lighthouse-core/scripts/lantern/collect/urls.js").join("\n"))'`)
# Go comment out `throw new Error('Cannot change settings between gathering and auditing');` in runner.js
# Go comment out `lhr.i18n.icuMessagePaths =` in runner.js
node --max-old-space-size=4096 lighthouse-core/scripts/compare-runs.js \
  --name urls-5.6.0 \
  --collect \
  -n 5 \
  --lh-flags='--only-categories=performance' \
  --urls "${urls[@]}"

[4]

# Compare data.
node lighthouse-core/scripts/compare-runs.js \
  --name urls-5.6.0 \
  --name urls-6.0 \
  --compare \
  --filter='performance$' \
  --report-exclude 'key|min|max|stdev|cv|^n$' \
  --delta-property-sort mean

Resources

Data (must request access but will be given freely)

Similar analysis for 11ty sites

@paulirish
Copy link

Scatterplot:

image

@patrickhulce
Copy link

small correction to the script but other than that worked beautifully 👍

step 3

node --max-old-space-size=4096 lighthouse-core/scripts/compare-runs.js \
  --name urls-5.6.0 \
  --collect \ # <-- this should be --audit
  -n 5 \
  --lh-flags='--only-categories=performance' \
  --urls "${urls[@]}"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment