Metrics-Driven Engineering

Metrics-Driven Engineering

Mike Brittain @ mikebrittain
Director of engineering, Infrastructure

October 13, 2011

How many new visits?
How many listings created?
How many registrations?
How do people use Etsy?
How many convos sent?
How many purchases?
How many new shops?

Search indexing?
How fast are pages generating?
Async tasks currently in queue?
What is the application doing?
Developer API auth and rate limiting?
Images resized and stored?
Error and warning rates?

Replication slave lag?
Memcache hits/misses?
Available connections?
Are the servers in good shape ?
Database queries per second?
Total outgoing bandwidth?
CPU, Memory, I/O?

$314 Million GMS 2010


credit: pentarux (ﬂickr)

25 Million Unique Visitors
1 Billion page views per month

credit: pentarux (ﬂickr)

Engineering team grew 500%
over 18 months

credit: martin_heigan (ﬂickr)

Always Be Shipping

credit: ibailemon (ﬂickr)

Always Be Shipping
(even if it’s your ﬁrst day)

credit: ibailemon (ﬂickr)

90+ Engineers
40+ Deploys / day

credit: misswired (ﬂickr)

$cfg = array(
'checkout' => array('enabled' => 'on'),
'homepage' => array('enabled' => 'on'),
'profiles' => array('enabled' => 'on'),
'new_search' => array('enabled' => 'off'),
);

Conﬁg Flags
Enable and disable features quickly

$cfg = array(
'checkout' => array('enabled' => 'on'),
'homepage' => array('enabled' => 'on'),
'profiles' => array('enabled' => 'on'),
'new_search' => array('enabled' => 'off'),
);

Conﬁg Flags
Enable and disable features quickly
Plus “admin-only,” percentage ramp-up, A/B testing,
whitelists, blacklists, etc...

inevitable!
Failure is not an option

inevitable!
a learning opportunity!

inevitable!
a learning opportunity!
DETECTABLE!

A: Well, the Ops team manages the network, racks
the servers, installed the monitoring tools, wears
the pagers, blah, blah, blah...

Engineers build the application

Logging
Graphing
OPS ENG
Trending
Alerting

“Engineers are too busy writing
features to build metrics.”

Metrics are part of every feature
...and so are conﬁg ﬂags

Cacti (network, SNMP)
Ganglia (machines)
Graphite (application)
Splunk (log analysis, nightly reports)
Nagios (alerting)
Logging
Logster
StatsD

Ganglia
Cluster-oriented
Huge community contributed recipes
Custom metrics (gmetad)

Graphite
Single-instance
Create new metrics on-the-ﬂy
Customize via URLs and display functions

It’s 2:48 PM.
Do you know where your
logs are?

Logger::log_error("User login failed.
Reason: $msg for $username", “login”);

web0054 [Fri Mar 04 16:27:48 2011]
[error] [login] [mk04gw1p71] User login
failed. Reason: wrong password for ...

LogFormat "%h %l %u %t "%r" %>s %b"
common

LogFormat %{True-Client-IP}i %l %t "%r
" %>s %b "%{Referer}i"
"%{User-Agent}i"
%{etsy_shop_id}n %{etsy_uaid}n %V
%{etsy_ab_selections}n
%{etsy_request_uuid}n
%{etsy_api_consumer_key}n
%{etsy_api_method_name}n
%{php_memory_usage_bytes}n
%{php_time_microsec}n %D" combined

grep "/listing/" access.log |
awk '{sum=sum+$(NF-2)} END {print sum/NR}'

web0001 [04:28:54 2011] [error] [client 10.101.x.x] Help me, Rhonda.
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Oh noooooo!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Heeeeeeellllllllllllllppppp!
web0001 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!
web0201 [04:28:54 2011] [warning] [client 10.101.x.x] Gaaaaahhh!
web0034 [04:28:54 2011] [warning] [client 10.101.x.x] Oh nooooooooooo
web0001 [04:28:54 2011] [error] [client 10.101.x.x] Gaaaaahhh!!!
web0201 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.
web0055 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!
web0002 [04:28:54 2011] [warning] [client 10.101.x.x] Sky is falling.
web0020 [04:28:54 2011] [error] [client 10.101.x.x] Sky is falling.
web0087 [04:28:54 2011] [fatal] [client 10.101.x.x] Sky is falling.
web0003 [04:28:54 2011] [error] [client 10.101.x.x] You've been eaten by a grue.
web0066 [04:28:54 2011] [fatal] [client 10.101.x.x] Gaaaaahhh!!!

Logster
Fatals Errors Warnings

Logster
Run by cron
Keeps a cursor on your log ﬁle
Aggregate lines anyway you want
Output to Ganglia or Graphite
Simple parsers
github.com/etsy

if (fields['log_level'] == “fatal”):
self.fatals += 1

elif (fields['log_level'] == “error”):
self.errors += 1

elif (fields['log_level'] == “warning”):
self.warnings += 1

...

MetricObject("fatals",
(self.fatals / self.duration), "per sec")

MetricObject("errors",
(self.errors / self.duration), "per sec")

MetricObject("warning",
(self.warnings / self.duration), "per sec")

StatsD
Network daemon (node.js)
Accepts data over UDP
Flushes to Graphite every 10 sec
One-line of code
github.com/etsy

StatsD::increment("logins.success");

StatsD::increment("logins.success");

logins

StatsD::timing("gearman.time", $msec);

StatsD::timing("gearman.time", $msec);

90th pct

average

lower

echo "events.deploy.site 1 `date +%s`"
| nc graphite.etsycorp.com 2003

Vertical Line Technology!
target=drawAsInfinite(events.deploy.site)

We could stare at graphs all day...

http://graphite/render?
from=-1hours&width=600&height=200
&target=webs.errorLog.warning&rawData=1

http://graphite/render?
from=-1hours&width=600&height=200
&target=webs.errorLog.warning&rawData=1

webs.errorLog.warning,1318444930,1318448530,60|
5.0,1.0,3.0,1.0,0.0,9.0,0.0,1.0,3.0,2.0,1.0,6.0,2.0,6.0,3.0,6.0,4.0,4.0,2.0,
1.0,1.0,8.0,2.0,3.0,6.0,3.0,5.0,3.0,0.0,4.0,6.0,2.0,0.0,2.0,0.0,4.0,0.0,3.0,
1.0,3.0,4.0,2.0,10.0,3.0,0.0,6.0,0.0,4.0,2.0,5.0,18.0,1.0,1.0,2.0,1.0,8.0,5.
0,1.0,1.0,None

Holt-Winters Conﬁdence Bands

upper

lower

Business metrics
+ Conﬁdence bands
_____________
Alertable metrics

40,000+ metrics at Etsy
Systems, Applications, Business

Kind of Hard :-/
<a href="http://graphite.etsycorp.com/render?from=-1hours&width=800&height=600&title=File+or
+Script+Not+Found&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite
%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production
%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite
%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,
%23ff0000,%23006633,%23cc6600">
<img src="http://graphite.etsycorp.com/render?
from=-1hours&width=280&height=220&title=File+or+Script+Not
+Found&hideLegend=1&yMin=0&target=webs.errorLog.notExist&target=drawAsInfinite
%28deploys.config.production%29&target=drawAsInfinite%28deploys.web.production
%29&target=drawAsInfinite%28deploys.search.production%29&target=drawAsInfinite
%28deploys.imagestorage.other%29&colorList=%2300cc00,%230000ff,
%23ff0000,%23006633,%23cc6600">
</a>

Super Easy!
$g = new Graphite($time);
$g->setTitle('File Not Found');
$g->addMetric('webs.errorLog.notExist', '#00cc00');
echo $g->getDashboardHTML(280, 220);

Metrics!
Metrics + Events
Metrics + Alerts

Metrics!
Metrics + Events
Metrics + Alerts
Metrics + Metrics

High-level, real-time visibility

Homework
codeascraft.etsy.com
github.com/etsy Get in touch
mike @ etsy . com
We’re always looking for people @ mikebrittain
who are interested in this kind of
stuﬀ...

Thank You
etsy.com/careers

Metrics-Driven Engineering

More Related Content

Viewers also liked (20)

Similar to Metrics-Driven Engineering (20)

Recently uploaded (20)

Metrics-Driven Engineering