SlideShare a Scribd company logo
L E S S O N S L E A R N E D AT T W I T T E R
H A D O O P P E R F O R M A N C E
O P T I M I Z AT I O N AT S C A L E
A L E X L E V E N S O N |
I A N O ' C O N N E L L |
@ T H I S W I L LW O R K
@ 0 X 1 3 8
DATA PLATFORM @TWITTER
Develop, maintain, and support the core data processing
libraries used at Twitter
In a good position to make system-wide performance
improvements
Core Data Libraries Team
DATA PLATFORM @TWITTER
Idiomatic functional Scala library for writing Hadoop map reduce
Functional programming is a natural fit for map reduce
Compile time type checked
Core Data Libraries Team
github.com/twitter/scalding
DATA PLATFORM @TWITTER
Columnar storage format for the Hadoop ecosystem
Uses the Google Dremel column shredding and assembly
algorithm
Core Data Libraries Team
APACHE PARQUET
github.com/apache/parquet-mr
DATA PLATFORM @TWITTER
Streaming map reduce for hybrid realtime / batch topologies
Write once, execute in parallel on Storm / Heron (online) and
Scalding (offline)
Core Data Libraries Team
SUMMINGBIRD
github.com/twitter/summingbird
Hadoop at Twitter Scale
H A D O O P AT T W I T T E R
300+PETABYTES OF
DATA
100k MAP REDUCE
JOBS DAILY
MULTIPLES OF
1000+MACHINE
HADOOP
CLUSTERS
MULTIPLE
LARGEST
HADOOP
CLUSTERS IN
THE WORLD
AMONG THE
At this scale, even small system-wide
improvements can save significant
amounts of compute resources
C O S T AT S C A L E
What does your Hadoop cluster
spend most of its time doing?
W H AT T O I M P R O V E ?
Profile your cluster, you might be
surprised by what you find
M E A S U R E - D O N ' T G U E S S
ENABLE JVM PROFILING WITH -XPROF
Built into the JVM (HotSpot), so there's nothing to install
Xprof: a low overhead profiler built into the jvm
mapreduce.task.profile='true'
mapreduce.task.profile.maps='0-'
mapreduce.task.profile.reduces='0-'
mapreduce.task.profile.params='-Xprof'
ENABLE JVM PROFILING WITH -XPROF
Low overhead (uses stack sampling)
Surfaces the most expensive methods
Prints directly to task logs (stdout)
Xprof: a low overhead profiler built into the jvm
Flat profile of 412.48 secs (38743 total ticks): SpillThread
Interpreted + native Method
12.5% 0 + 32215 org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressBytesDirect
4.6% 0 + 822 java.io.FileOutputStream.writeBytes
...
19.4% 352 + 3082 Total interpreted (including elided)
Compiled + native Method
50.0% 8549 + 299 java.lang.StringCoding.decode
16.9% 2823 + 158 cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement
4.1% 734 + 0 sun.nio.cs.UTF_8$Decoder.decode
2.3% 401 + 0 org.apache.hadoop.mapred.IFileOutputStream.write
2.0% 352 + 0 cascading.tuple.hadoop.util.TupleComparator.compare
1.7% 296 + 0 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare
...
79.0% 13514 + 467 Total compiled
Thread-local ticks:
54.3% 21053 Blocked (of total)
HADOOP CONFIGURATION OBJECT
Looks and behaves a lot like a HashMap
Surprisingly expensive
Configuration conf = new Configuration()
conf.set("myKey", "myValue")
String value = conf.get("myKey")
HADOOP CONFIGURATION OBJECT
Constructor reads + unzips + parses an XML file from disk
Surprisingly expensive
public class KryoSerialization {
public KryoSerialization() {
this(new Configuration())
}
}
HADOOP CONFIGURATION OBJECT
get() method involves regular expressions, variable substitution
Surprisingly expensive
String value = conf.get("myKey")
HADOOP CONFIGURATION OBJECT
Calling these methods in a loop, or once per record, is
expensive
Some (non trivial) jobs were spending 30% of their time in
Configuration methods
Surprisingly expensive
It's hard to predict what needs to
be optimized without a profiler
L E S S O N L E A R N E D
If you don't profile, you could be
missing easy wins
L E S S O N L E A R N E D
Measure whether IO or CPU is your
biggest cost
L E S S O N L E A R N E D
INTERMEDIATE COMPRESSION
Xprof surfaced that compression + decompression in the spill
thread was taking a lot of time
Intermediate outputs are temporary
We now use lz4 instead of lzo level 3, which produces 30%
larger intermediate data that's faster to read
Made some large jobs 1.5X faster
Find the right balance
Record Serialization + Deserialization
can be the most expensive part of
your job
L E S S O N L E A R N E D
Record Serialization is CPU intensive,
and may overshadow IO
L E S S O N L E A R N E D
How to reduce costs due to record
serialization?
L E S S O N L E A R N E D
USE HADOOP'S RAW COMPARATOR API
Hadoop MR deserializes the map output keys in order to sort
them between the map and reduce phases
Don't make sorting more expensive than it already is
deserialize(keyBytes1).compare(deserialize(keyBytes2))
USE HADOOP'S RAW COMPARATOR API
This can cost a lot, especially for complex non-primitive keys,
which is fairly common
Don't make sorting more expensive than it already is
requests.groupBy { req => (req.country, req.client) }
USE HADOOP'S RAW COMPARATOR API
This can cost a lot, especially for complex non-primitive keys,
which is fairly common
Don't make sorting more expensive than it already is
Complex object
that requires sorting
requests.groupBy { req => (req.country, req.client) }
Flat profile of 412.48 secs (38743 total ticks): SpillThread
Interpreted + native Method
12.5% 0 + 32215 org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressBytesDirect
4.6% 0 + 822 java.io.FileOutputStream.writeBytes
...
19.4% 352 + 3082 Total interpreted (including elided)
Compiled + native Method
50.0% 8549 + 299 java.lang.StringCoding.decode
16.9% 2823 + 158 cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement
4.1% 734 + 0 sun.nio.cs.UTF_8$Decoder.decode
2.3% 401 + 0 org.apache.hadoop.mapred.IFileOutputStream.write
2.0% 352 + 0 cascading.tuple.hadoop.util.TupleComparator.compare
1.7% 296 + 0 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare
...
79.0% 13514 + 467 Total compiled
Thread-local ticks:
54.3% 21053 Blocked (of total)
USE HADOOP'S RAW COMPARATOR API
Hadoop comes with a RawComparator API for comparing
records in their serialized (raw) form
Don't make sorting more expensive than it already is
deserialize(keyBytes1).compare(deserialize(keyBytes2))
compare(keyBytes1, keyBytes2)
USE HADOOP'S RAW COMPARATOR API
Hadoop comes with a RawComparator API for comparing
records in their serialized (raw) form
Don't make sorting more expensive than it already is
public interface RawComparator<T> {
public int compare(byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2);
}
USE HADOOP'S RAW COMPARATOR API
Unfortunately, this requires you to write a custom comparator
by hand
And assumes that your data is actually easy to compare in its
serialized form
Don't make sorting more expensive than it already is
public interface RawComparator<T> {
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2);
}
SCALA MACROS FOR RAW COMPARATORS
Macros to the rescue!
A slightly more hipster API for Raw Comparators in Scala
And a handful of macros to generate implementations of this
API for tuples, case classes, thrift objects, primitives, Strings,
etc.
SCALA MACROS FOR RAW COMPARATORS
1 3 f o o 0 1 17 1 88 ...
Macros to the rescue!
First, creates a custom dense serialization format that's easy to
compare
1 3 f o o 0 1 22 0 ... ...
non-null String
null value
non-null int non-null int
null value
SCALA MACROS FOR RAW COMPARATORS
1 3 f o o 0 1 17 1 88 ...
Macros to the rescue!
Then, creates a compare method that takes advantage of this
format
1 3 f 0 o 0 1 22 0 ... ...
SCALA MACROS FOR RAW COMPARATORS
Macros to the rescue!
TotalComputeTime
Default Raw Comparators
1.5X
FASTER
How to reduce costs due to record
serialization?
L E S S O N L E A R N E D
COLUMN PROJECTION
Don't read or deserialize data that you don't need
struct User {
1: i64 id
2: Address address
3: string name
4 list<Interest> interests
}
COLUMN PROJECTION
Columnar file formats like Apache Parquet support this directly
Specialized record deserializers can skip over unwanted fields
in row oriented storage
Don't read or deserialize data that you don't need
APACHE PARQUET
Columnar storage for the people
In traditional row-oriented storage layout, an entire record is
stored sequentially
R1.A R1.B R1.C R2.A R2.B R2.C R3.A R3.B R3.C
APACHE PARQUET
Columnar storage for the people
In traditional row-oriented storage layout, an entire record is
stored sequentially
9903489083
"123 elm street"
"alice"
"columnar file formats"
9903489084
"333 oak street"
"bob"
"Hadoop"
Compressed with lzo / gzip / snappy
APACHE PARQUET
Columnar storage for the people
In columnar storage layout, an entire column is stored
sequentially
R1.A R2.A R3.A R1.B R2.B R3.B R1.C R2.C R3.C
APACHE PARQUET
Columnar storage for the people
All user ids stored together
In columnar storage layout, an entire column is stored
sequentially
9903489083
9903489084
9903489085
9903489075
9903489088
9903489087
"123 elm street"
"333 oak street"
"827 maple drive"
APACHE PARQUET
Columnar storage for the people
Schema aware storage can use specialized encodings
9903489083
9903489084
9903489085
9903489075
9903489088
9903489087
9903489083
+1
+1
-10
+3
-1
delta
"twitter.com/foo/bar"
"blog.twitter.com"
"twitter.com/foo/bar"
"twitter.com/foo/bar"
"blog.twitter.com"
"blog.twitter.com"
"blog.twitter.com"
"blog.twitter.com/123"
"twitter.com/foo/bar": 0
"blog.twitter.com": 1
"blog.twitter.com/123": 2
0
1
0
0
1
1
1
2
dictionary
FILE SIZE COMPARISON
SizeinGB
B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet
2X
SMALLER
B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet
APACHE PARQUET
Columnar storage for the people
Collocating entire columns allows for efficient
column projection
Read off disk only the columns you need
Possibly more importantly: deserialize only
the columns you need
TotalComputeTime
1 column 10 columns 40 columns
Parquet Lzo Thrift
COLUMN PROJECTION WITH PARQUET
3X
FASTER
1.5X
FASTER
1.15X
FASTER
TotalComputeTime
1 column 10 columns 40 columns
Parquet Lzo Thrift
COLUMN PROJECTION WITH PARQUET
3X
FASTER
1.5X
FASTER
1.15X
FASTER
APACHE PARQUET
Columnar storage for the people
Parquet is often slower to read all columns than row oriented
storage
Parquet is a dense format, read performance scales with the
number of columns in the schema -- nulls take time read
Sparse, row oriented formats (thrift) scale with the number of
columns present in the data -- nulls take no time read
COLUMN PROJECTION FOR ROW ORIENTED DATA
Row oriented is a very common way to store Thrift, Avro,
Protocol Buffers, etc.
Specialized record deserializers can skip over unwanted fields
in these row oriented storage formats
Prototype implemented as a Scala macro that creates a custom
deserializer at compile time
Don't deserialize data that you don't need
COLUMN PROJECTION FOR ROW ORIENTED DATA
Don't deserialize data that you don't need
198 111 121 054 e l m _ s t r ... a l i c e ...
Decode User Id to Long
Skip over unwanted address field
Decode Name to String
COLUMN PROJECTION FOR ROW ORIENTED DATA
No IO savings
But only decodes the fields you care about into objects
CPU time spent decoding Strings can be huge compared to
time it takes to load + ignore the encoded bytes
Don't deserialize data that you don't need
TotalComputeTime
Number of Columns Selected
1 7 10 13 48
Parquet Thrift
Parquet Pig
Lzo Thrift + Projection
COLUMN PROJECTION: THRIFT VS. PARQUET
Parquet Thrift has a lot
of room for
improvement
Parquet faster than row
oriented until 13 columns
This schema is relatively
flat, and most columns
populated
APACHE PARQUET
Columnar storage for the people
Predicate push-down also allows parquet to
skip over records that don't match your filter
criteria
Parquet stores statistics about chunks of
records, so in some cases entire chunks of
data can be skipped after examining these
statistics
APACHE PARQUET
Columnar storage for the people
Combining both column projection and predicate push
down is a powerful combination
TotalComputeTime
Lzo Thrift Parquet + Filter Parquet + Filter + Project
FILTER PUSH DOWN WITH PARQUET
4.3X
FASTER
APACHE PARQUET
Columnar storage for the people
Predicate push-down performance depends on the nature
of the filter
Searching for rare records is the best case, entire chunks of
records are likely to not contain the records you are looking
for
Key take aways
I N S U M M A R Y
IN SUMMARY
Key takeaways
Profile!
Serialization is expensive, and Hadoop does a lot of it
Choose a storage format that fits your access patterns
Use column projection
Sorting is expensive -- use Raw Comparators
IO may not be your bottleneck -- more IO for less CPU may be
a good tradeoff
ACKNOWLEDGEMENTS
Thanks to everyone involved!
Dmitriy Ryaboy @squarecog
Gera Shegalov @gerashegalov
Julien Le Dem @J_
Katya Gonina @katyagonina
Mansur Ashraf @mansur_ashraf
Oscar Boykin @posco
Sriram Krishnan @krishnansriram
Tianshuo Deng @tsdeng
Zak Taylor @zakattacktaylor
And many more!
GET INVOLVED
Contributions always welcome!
github.com/twitter/scalding
github.com/twitter/algebird
github.com/twitter/chill
github.com/apache/parquet-mr
JOIN THE FLOCK
We're Hiring!
Work on data processing challenges at scale
Strong commitment to open source
jobs.twitter.com
Data Platform: (https://about.twitter.com/careers/positions?jvi=oipMYfwb,Job)
Q U E S T I O N S ?
A L E X L E V E N S O N |
I A N O ' C O N N E L L |
@ T H I S W I L LW O R K
@ 0 X 1 3 8

More Related Content

What's hot (20)

PDF
Intro to HBase
alexbaranau
 
PPTX
Hive + Tez: A Performance Deep Dive
DataWorks Summit
 
PDF
Inside Parquet Format
Yue Chen
 
PDF
Spark SQL Deep Dive @ Melbourne Spark Meetup
Databricks
 
PPTX
HBase Low Latency
DataWorks Summit
 
PDF
BlueStore: a new, faster storage backend for Ceph
Sage Weil
 
PPT
Chicago Data Summit: Apache HBase: An Introduction
Cloudera, Inc.
 
PPTX
HBase and HDFS: Understanding FileSystem Usage in HBase
enissoz
 
PPTX
ORC File - Optimizing Your Big Data
DataWorks Summit
 
PDF
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Glen Hawkins
 
PPTX
Hive+Tez: A performance deep dive
t3rmin4t0r
 
PDF
Parquet Hadoop Summit 2013
Julien Le Dem
 
PPTX
Supporting Apache HBase : Troubleshooting and Supportability Improvements
DataWorks Summit
 
PDF
How Impala Works
Yue Chen
 
PDF
Intro to Delta Lake
Databricks
 
PDF
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...
Spark Summit
 
PDF
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers
Cloudera, Inc.
 
PDF
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
SrirakshaSrinivasan2
 
PDF
Spark Summit EU talk by Mike Percy
Spark Summit
 
PDF
Databricks Delta Lake and Its Benefits
Databricks
 
Intro to HBase
alexbaranau
 
Hive + Tez: A Performance Deep Dive
DataWorks Summit
 
Inside Parquet Format
Yue Chen
 
Spark SQL Deep Dive @ Melbourne Spark Meetup
Databricks
 
HBase Low Latency
DataWorks Summit
 
BlueStore: a new, faster storage backend for Ceph
Sage Weil
 
Chicago Data Summit: Apache HBase: An Introduction
Cloudera, Inc.
 
HBase and HDFS: Understanding FileSystem Usage in HBase
enissoz
 
ORC File - Optimizing Your Big Data
DataWorks Summit
 
Oracle Active Data Guard: Best Practices and New Features Deep Dive
Glen Hawkins
 
Hive+Tez: A performance deep dive
t3rmin4t0r
 
Parquet Hadoop Summit 2013
Julien Le Dem
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
DataWorks Summit
 
How Impala Works
Yue Chen
 
Intro to Delta Lake
Databricks
 
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...
Spark Summit
 
HBase HUG Presentation: Avoiding Full GCs with MemStore-Local Allocation Buffers
Cloudera, Inc.
 
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
SrirakshaSrinivasan2
 
Spark Summit EU talk by Mike Percy
Spark Summit
 
Databricks Delta Lake and Its Benefits
Databricks
 

Viewers also liked (15)

PPTX
Pysparkで始めるデータ分析
Tanaka Yuichi
 
PDF
Parquet Strata/Hadoop World, New York 2013
Julien Le Dem
 
PPTX
Pythonで入門するApache Spark at PyCon2016
Tatsuya Atsumi
 
PPTX
Redshift Spectrumを使ってみた話
Yoshiki Kouno
 
PDF
Efficient Data Storage for Analytics with Apache Parquet 2.0
Cloudera, Inc.
 
PPTX
Alibaba Cloud
Serverworks Co.,Ltd.
 
PDF
AWS Black Belt Online Seminar 2017 Amazon Connect
Amazon Web Services Japan
 
PDF
AWS Black Belt - AWS Glue
Amazon Web Services Japan
 
PDF
AWS Black Belt Online Seminar 2017 Amazon GameLift
Amazon Web Services Japan
 
PDF
Docker入門: コンテナ型仮想化技術の仕組みと使い方
Yuichi Ito
 
PDF
AWS BlackBelt AWS上でのDDoS対策
Amazon Web Services Japan
 
PDF
AWS Black Belt Online Seminar 2017 Amazon EMR
Amazon Web Services Japan
 
PDF
AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計
Amazon Web Services Japan
 
PDF
AWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハック
Amazon Web Services Japan
 
PDF
Amazon Athena 初心者向けハンズオン
Amazon Web Services Japan
 
Pysparkで始めるデータ分析
Tanaka Yuichi
 
Parquet Strata/Hadoop World, New York 2013
Julien Le Dem
 
Pythonで入門するApache Spark at PyCon2016
Tatsuya Atsumi
 
Redshift Spectrumを使ってみた話
Yoshiki Kouno
 
Efficient Data Storage for Analytics with Apache Parquet 2.0
Cloudera, Inc.
 
Alibaba Cloud
Serverworks Co.,Ltd.
 
AWS Black Belt Online Seminar 2017 Amazon Connect
Amazon Web Services Japan
 
AWS Black Belt - AWS Glue
Amazon Web Services Japan
 
AWS Black Belt Online Seminar 2017 Amazon GameLift
Amazon Web Services Japan
 
Docker入門: コンテナ型仮想化技術の仕組みと使い方
Yuichi Ito
 
AWS BlackBelt AWS上でのDDoS対策
Amazon Web Services Japan
 
AWS Black Belt Online Seminar 2017 Amazon EMR
Amazon Web Services Japan
 
AWS Black Belt Online Seminar 2017 AWSへのネットワーク接続とAWS上のネットワーク内部設計
Amazon Web Services Japan
 
AWS Black Belt Online Seminar 2017 Amazon Pinpoint で始めるモバイルアプリのグロースハック
Amazon Web Services Japan
 
Amazon Athena 初心者向けハンズオン
Amazon Web Services Japan
 
Ad

Similar to Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twitter (Alex Levenson) (20)

PPTX
Real time hadoop + mapreduce intro
Geoff Hendrey
 
PPTX
Large Scale Data With Hadoop
guest27e6764
 
PDF
Lecture 4: Data-Intensive Computing for Text Analysis (Fall 2011)
Matthew Lease
 
PPT
Hadoop
Girish Khanzode
 
DOCX
Hadoop Seminar Report
Atul Kushwaha
 
PPTX
Bigdata workshop february 2015
clairvoyantllc
 
PDF
getFamiliarWithHadoop
AmirReza Mohammadi
 
PDF
Scaling Storage and Computation with Hadoop
yaevents
 
PPT
Finding the needles in the haystack. An Overview of Analyzing Big Data with H...
Chris Baglieri
 
PDF
Hadoop Tutorial with @techmilind
EMC
 
PDF
Hadoop paper
ATWIINE Simon Alex
 
DOC
Hadoop
Himanshu Soni
 
PDF
Hadoop, Taming Elephants
Ovidiu Dimulescu
 
PPTX
Hadoop introduction , Why and What is Hadoop ?
sudhakara st
 
PDF
Getting Started with Hadoop
Josh Devins
 
PDF
Big data and hadoop overvew
Kunal Khanna
 
PPT
Hadoop - Introduction to HDFS
Vibrant Technologies & Computers
 
PPT
Hadoop distributed file system (HDFS), HDFS concept
kuthubussaman1
 
PDF
Hadoop Primer
Steve Staso
 
PPTX
Introduction to Hadoop and Big Data
Joe Alex
 
Real time hadoop + mapreduce intro
Geoff Hendrey
 
Large Scale Data With Hadoop
guest27e6764
 
Lecture 4: Data-Intensive Computing for Text Analysis (Fall 2011)
Matthew Lease
 
Hadoop Seminar Report
Atul Kushwaha
 
Bigdata workshop february 2015
clairvoyantllc
 
getFamiliarWithHadoop
AmirReza Mohammadi
 
Scaling Storage and Computation with Hadoop
yaevents
 
Finding the needles in the haystack. An Overview of Analyzing Big Data with H...
Chris Baglieri
 
Hadoop Tutorial with @techmilind
EMC
 
Hadoop paper
ATWIINE Simon Alex
 
Hadoop, Taming Elephants
Ovidiu Dimulescu
 
Hadoop introduction , Why and What is Hadoop ?
sudhakara st
 
Getting Started with Hadoop
Josh Devins
 
Big data and hadoop overvew
Kunal Khanna
 
Hadoop - Introduction to HDFS
Vibrant Technologies & Computers
 
Hadoop distributed file system (HDFS), HDFS concept
kuthubussaman1
 
Hadoop Primer
Steve Staso
 
Introduction to Hadoop and Big Data
Joe Alex
 
Ad

Recently uploaded (20)

PPTX
Empowering Asian Contributions: The Rise of Regional User Groups in Open Sour...
Shane Coughlan
 
PDF
MiniTool Partition Wizard 12.8 Crack License Key LATEST
hashhshs786
 
PDF
MiniTool Partition Wizard Free Crack + Full Free Download 2025
bashirkhan333g
 
PDF
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
PDF
The 5 Reasons for IT Maintenance - Arna Softech
Arna Softech
 
PDF
Automate Cybersecurity Tasks with Python
VICTOR MAESTRE RAMIREZ
 
PDF
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
PPTX
Homogeneity of Variance Test Options IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PDF
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
PDF
SAP Firmaya İade ABAB Kodları - ABAB ile yazılmıl hazır kod örneği
Salih Küçük
 
PDF
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
PPTX
Tally_Basic_Operations_Presentation.pptx
AditiBansal54083
 
PPTX
Foundations of Marketo Engage - Powering Campaigns with Marketo Personalization
bbedford2
 
PPTX
Finding Your License Details in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PDF
Top Agile Project Management Tools for Teams in 2025
Orangescrum
 
PPTX
Agentic Automation Journey Series Day 2 – Prompt Engineering for UiPath Agents
klpathrudu
 
PPTX
AEM User Group: India Chapter Kickoff Meeting
jennaf3
 
PPTX
Agentic Automation: Build & Deploy Your First UiPath Agent
klpathrudu
 
PDF
Empower Your Tech Vision- Why Businesses Prefer to Hire Remote Developers fro...
logixshapers59
 
PDF
IDM Crack with Internet Download Manager 6.42 Build 43 with Patch Latest 2025
bashirkhan333g
 
Empowering Asian Contributions: The Rise of Regional User Groups in Open Sour...
Shane Coughlan
 
MiniTool Partition Wizard 12.8 Crack License Key LATEST
hashhshs786
 
MiniTool Partition Wizard Free Crack + Full Free Download 2025
bashirkhan333g
 
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
The 5 Reasons for IT Maintenance - Arna Softech
Arna Softech
 
Automate Cybersecurity Tasks with Python
VICTOR MAESTRE RAMIREZ
 
iTop VPN With Crack Lifetime Activation Key-CODE
utfefguu
 
Homogeneity of Variance Test Options IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
SAP Firmaya İade ABAB Kodları - ABAB ile yazılmıl hazır kod örneği
Salih Küçük
 
Odoo CRM vs Zoho CRM: Honest Comparison 2025
Odiware Technologies Private Limited
 
Tally_Basic_Operations_Presentation.pptx
AditiBansal54083
 
Foundations of Marketo Engage - Powering Campaigns with Marketo Personalization
bbedford2
 
Finding Your License Details in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
Top Agile Project Management Tools for Teams in 2025
Orangescrum
 
Agentic Automation Journey Series Day 2 – Prompt Engineering for UiPath Agents
klpathrudu
 
AEM User Group: India Chapter Kickoff Meeting
jennaf3
 
Agentic Automation: Build & Deploy Your First UiPath Agent
klpathrudu
 
Empower Your Tech Vision- Why Businesses Prefer to Hire Remote Developers fro...
logixshapers59
 
IDM Crack with Internet Download Manager 6.42 Build 43 with Patch Latest 2025
bashirkhan333g
 

Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twitter (Alex Levenson)

  • 1. L E S S O N S L E A R N E D AT T W I T T E R H A D O O P P E R F O R M A N C E O P T I M I Z AT I O N AT S C A L E A L E X L E V E N S O N | I A N O ' C O N N E L L | @ T H I S W I L LW O R K @ 0 X 1 3 8
  • 2. DATA PLATFORM @TWITTER Develop, maintain, and support the core data processing libraries used at Twitter In a good position to make system-wide performance improvements Core Data Libraries Team
  • 3. DATA PLATFORM @TWITTER Idiomatic functional Scala library for writing Hadoop map reduce Functional programming is a natural fit for map reduce Compile time type checked Core Data Libraries Team github.com/twitter/scalding
  • 4. DATA PLATFORM @TWITTER Columnar storage format for the Hadoop ecosystem Uses the Google Dremel column shredding and assembly algorithm Core Data Libraries Team APACHE PARQUET github.com/apache/parquet-mr
  • 5. DATA PLATFORM @TWITTER Streaming map reduce for hybrid realtime / batch topologies Write once, execute in parallel on Storm / Heron (online) and Scalding (offline) Core Data Libraries Team SUMMINGBIRD github.com/twitter/summingbird
  • 6. Hadoop at Twitter Scale H A D O O P AT T W I T T E R
  • 8. 100k MAP REDUCE JOBS DAILY MULTIPLES OF
  • 11. At this scale, even small system-wide improvements can save significant amounts of compute resources C O S T AT S C A L E
  • 12. What does your Hadoop cluster spend most of its time doing? W H AT T O I M P R O V E ?
  • 13. Profile your cluster, you might be surprised by what you find M E A S U R E - D O N ' T G U E S S
  • 14. ENABLE JVM PROFILING WITH -XPROF Built into the JVM (HotSpot), so there's nothing to install Xprof: a low overhead profiler built into the jvm mapreduce.task.profile='true' mapreduce.task.profile.maps='0-' mapreduce.task.profile.reduces='0-' mapreduce.task.profile.params='-Xprof'
  • 15. ENABLE JVM PROFILING WITH -XPROF Low overhead (uses stack sampling) Surfaces the most expensive methods Prints directly to task logs (stdout) Xprof: a low overhead profiler built into the jvm
  • 16. Flat profile of 412.48 secs (38743 total ticks): SpillThread Interpreted + native Method 12.5% 0 + 32215 org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressBytesDirect 4.6% 0 + 822 java.io.FileOutputStream.writeBytes ... 19.4% 352 + 3082 Total interpreted (including elided) Compiled + native Method 50.0% 8549 + 299 java.lang.StringCoding.decode 16.9% 2823 + 158 cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement 4.1% 734 + 0 sun.nio.cs.UTF_8$Decoder.decode 2.3% 401 + 0 org.apache.hadoop.mapred.IFileOutputStream.write 2.0% 352 + 0 cascading.tuple.hadoop.util.TupleComparator.compare 1.7% 296 + 0 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare ... 79.0% 13514 + 467 Total compiled Thread-local ticks: 54.3% 21053 Blocked (of total)
  • 17. HADOOP CONFIGURATION OBJECT Looks and behaves a lot like a HashMap Surprisingly expensive Configuration conf = new Configuration() conf.set("myKey", "myValue") String value = conf.get("myKey")
  • 18. HADOOP CONFIGURATION OBJECT Constructor reads + unzips + parses an XML file from disk Surprisingly expensive public class KryoSerialization { public KryoSerialization() { this(new Configuration()) } }
  • 19. HADOOP CONFIGURATION OBJECT get() method involves regular expressions, variable substitution Surprisingly expensive String value = conf.get("myKey")
  • 20. HADOOP CONFIGURATION OBJECT Calling these methods in a loop, or once per record, is expensive Some (non trivial) jobs were spending 30% of their time in Configuration methods Surprisingly expensive
  • 21. It's hard to predict what needs to be optimized without a profiler L E S S O N L E A R N E D
  • 22. If you don't profile, you could be missing easy wins L E S S O N L E A R N E D
  • 23. Measure whether IO or CPU is your biggest cost L E S S O N L E A R N E D
  • 24. INTERMEDIATE COMPRESSION Xprof surfaced that compression + decompression in the spill thread was taking a lot of time Intermediate outputs are temporary We now use lz4 instead of lzo level 3, which produces 30% larger intermediate data that's faster to read Made some large jobs 1.5X faster Find the right balance
  • 25. Record Serialization + Deserialization can be the most expensive part of your job L E S S O N L E A R N E D
  • 26. Record Serialization is CPU intensive, and may overshadow IO L E S S O N L E A R N E D
  • 27. How to reduce costs due to record serialization? L E S S O N L E A R N E D
  • 28. USE HADOOP'S RAW COMPARATOR API Hadoop MR deserializes the map output keys in order to sort them between the map and reduce phases Don't make sorting more expensive than it already is deserialize(keyBytes1).compare(deserialize(keyBytes2))
  • 29. USE HADOOP'S RAW COMPARATOR API This can cost a lot, especially for complex non-primitive keys, which is fairly common Don't make sorting more expensive than it already is requests.groupBy { req => (req.country, req.client) }
  • 30. USE HADOOP'S RAW COMPARATOR API This can cost a lot, especially for complex non-primitive keys, which is fairly common Don't make sorting more expensive than it already is Complex object that requires sorting requests.groupBy { req => (req.country, req.client) }
  • 31. Flat profile of 412.48 secs (38743 total ticks): SpillThread Interpreted + native Method 12.5% 0 + 32215 org.apache.hadoop.io.compress.lz4.Lz4Compressor.compressBytesDirect 4.6% 0 + 822 java.io.FileOutputStream.writeBytes ... 19.4% 352 + 3082 Total interpreted (including elided) Compiled + native Method 50.0% 8549 + 299 java.lang.StringCoding.decode 16.9% 2823 + 158 cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement 4.1% 734 + 0 sun.nio.cs.UTF_8$Decoder.decode 2.3% 401 + 0 org.apache.hadoop.mapred.IFileOutputStream.write 2.0% 352 + 0 cascading.tuple.hadoop.util.TupleComparator.compare 1.7% 296 + 0 org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare ... 79.0% 13514 + 467 Total compiled Thread-local ticks: 54.3% 21053 Blocked (of total)
  • 32. USE HADOOP'S RAW COMPARATOR API Hadoop comes with a RawComparator API for comparing records in their serialized (raw) form Don't make sorting more expensive than it already is deserialize(keyBytes1).compare(deserialize(keyBytes2)) compare(keyBytes1, keyBytes2)
  • 33. USE HADOOP'S RAW COMPARATOR API Hadoop comes with a RawComparator API for comparing records in their serialized (raw) form Don't make sorting more expensive than it already is public interface RawComparator<T> { public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2); }
  • 34. USE HADOOP'S RAW COMPARATOR API Unfortunately, this requires you to write a custom comparator by hand And assumes that your data is actually easy to compare in its serialized form Don't make sorting more expensive than it already is public interface RawComparator<T> { public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2); }
  • 35. SCALA MACROS FOR RAW COMPARATORS Macros to the rescue! A slightly more hipster API for Raw Comparators in Scala And a handful of macros to generate implementations of this API for tuples, case classes, thrift objects, primitives, Strings, etc.
  • 36. SCALA MACROS FOR RAW COMPARATORS 1 3 f o o 0 1 17 1 88 ... Macros to the rescue! First, creates a custom dense serialization format that's easy to compare 1 3 f o o 0 1 22 0 ... ... non-null String null value non-null int non-null int null value
  • 37. SCALA MACROS FOR RAW COMPARATORS 1 3 f o o 0 1 17 1 88 ... Macros to the rescue! Then, creates a compare method that takes advantage of this format 1 3 f 0 o 0 1 22 0 ... ...
  • 38. SCALA MACROS FOR RAW COMPARATORS Macros to the rescue! TotalComputeTime Default Raw Comparators 1.5X FASTER
  • 39. How to reduce costs due to record serialization? L E S S O N L E A R N E D
  • 40. COLUMN PROJECTION Don't read or deserialize data that you don't need struct User { 1: i64 id 2: Address address 3: string name 4 list<Interest> interests }
  • 41. COLUMN PROJECTION Columnar file formats like Apache Parquet support this directly Specialized record deserializers can skip over unwanted fields in row oriented storage Don't read or deserialize data that you don't need
  • 42. APACHE PARQUET Columnar storage for the people In traditional row-oriented storage layout, an entire record is stored sequentially R1.A R1.B R1.C R2.A R2.B R2.C R3.A R3.B R3.C
  • 43. APACHE PARQUET Columnar storage for the people In traditional row-oriented storage layout, an entire record is stored sequentially 9903489083 "123 elm street" "alice" "columnar file formats" 9903489084 "333 oak street" "bob" "Hadoop" Compressed with lzo / gzip / snappy
  • 44. APACHE PARQUET Columnar storage for the people In columnar storage layout, an entire column is stored sequentially R1.A R2.A R3.A R1.B R2.B R3.B R1.C R2.C R3.C
  • 45. APACHE PARQUET Columnar storage for the people All user ids stored together In columnar storage layout, an entire column is stored sequentially 9903489083 9903489084 9903489085 9903489075 9903489088 9903489087 "123 elm street" "333 oak street" "827 maple drive"
  • 46. APACHE PARQUET Columnar storage for the people Schema aware storage can use specialized encodings 9903489083 9903489084 9903489085 9903489075 9903489088 9903489087 9903489083 +1 +1 -10 +3 -1 delta "twitter.com/foo/bar" "blog.twitter.com" "twitter.com/foo/bar" "twitter.com/foo/bar" "blog.twitter.com" "blog.twitter.com" "blog.twitter.com" "blog.twitter.com/123" "twitter.com/foo/bar": 0 "blog.twitter.com": 1 "blog.twitter.com/123": 2 0 1 0 0 1 1 1 2 dictionary
  • 47. FILE SIZE COMPARISON SizeinGB B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet 2X SMALLER B64 Lzo Thrift Block Lzo Thrift Gzipped Json Lzo Parquet
  • 48. APACHE PARQUET Columnar storage for the people Collocating entire columns allows for efficient column projection Read off disk only the columns you need Possibly more importantly: deserialize only the columns you need
  • 49. TotalComputeTime 1 column 10 columns 40 columns Parquet Lzo Thrift COLUMN PROJECTION WITH PARQUET 3X FASTER 1.5X FASTER 1.15X FASTER
  • 50. TotalComputeTime 1 column 10 columns 40 columns Parquet Lzo Thrift COLUMN PROJECTION WITH PARQUET 3X FASTER 1.5X FASTER 1.15X FASTER
  • 51. APACHE PARQUET Columnar storage for the people Parquet is often slower to read all columns than row oriented storage Parquet is a dense format, read performance scales with the number of columns in the schema -- nulls take time read Sparse, row oriented formats (thrift) scale with the number of columns present in the data -- nulls take no time read
  • 52. COLUMN PROJECTION FOR ROW ORIENTED DATA Row oriented is a very common way to store Thrift, Avro, Protocol Buffers, etc. Specialized record deserializers can skip over unwanted fields in these row oriented storage formats Prototype implemented as a Scala macro that creates a custom deserializer at compile time Don't deserialize data that you don't need
  • 53. COLUMN PROJECTION FOR ROW ORIENTED DATA Don't deserialize data that you don't need 198 111 121 054 e l m _ s t r ... a l i c e ... Decode User Id to Long Skip over unwanted address field Decode Name to String
  • 54. COLUMN PROJECTION FOR ROW ORIENTED DATA No IO savings But only decodes the fields you care about into objects CPU time spent decoding Strings can be huge compared to time it takes to load + ignore the encoded bytes Don't deserialize data that you don't need
  • 55. TotalComputeTime Number of Columns Selected 1 7 10 13 48 Parquet Thrift Parquet Pig Lzo Thrift + Projection COLUMN PROJECTION: THRIFT VS. PARQUET Parquet Thrift has a lot of room for improvement Parquet faster than row oriented until 13 columns This schema is relatively flat, and most columns populated
  • 56. APACHE PARQUET Columnar storage for the people Predicate push-down also allows parquet to skip over records that don't match your filter criteria Parquet stores statistics about chunks of records, so in some cases entire chunks of data can be skipped after examining these statistics
  • 57. APACHE PARQUET Columnar storage for the people Combining both column projection and predicate push down is a powerful combination
  • 58. TotalComputeTime Lzo Thrift Parquet + Filter Parquet + Filter + Project FILTER PUSH DOWN WITH PARQUET 4.3X FASTER
  • 59. APACHE PARQUET Columnar storage for the people Predicate push-down performance depends on the nature of the filter Searching for rare records is the best case, entire chunks of records are likely to not contain the records you are looking for
  • 60. Key take aways I N S U M M A R Y
  • 61. IN SUMMARY Key takeaways Profile! Serialization is expensive, and Hadoop does a lot of it Choose a storage format that fits your access patterns Use column projection Sorting is expensive -- use Raw Comparators IO may not be your bottleneck -- more IO for less CPU may be a good tradeoff
  • 62. ACKNOWLEDGEMENTS Thanks to everyone involved! Dmitriy Ryaboy @squarecog Gera Shegalov @gerashegalov Julien Le Dem @J_ Katya Gonina @katyagonina Mansur Ashraf @mansur_ashraf Oscar Boykin @posco Sriram Krishnan @krishnansriram Tianshuo Deng @tsdeng Zak Taylor @zakattacktaylor And many more!
  • 63. GET INVOLVED Contributions always welcome! github.com/twitter/scalding github.com/twitter/algebird github.com/twitter/chill github.com/apache/parquet-mr
  • 64. JOIN THE FLOCK We're Hiring! Work on data processing challenges at scale Strong commitment to open source jobs.twitter.com Data Platform: (https://about.twitter.com/careers/positions?jvi=oipMYfwb,Job)
  • 65. Q U E S T I O N S ? A L E X L E V E N S O N | I A N O ' C O N N E L L | @ T H I S W I L LW O R K @ 0 X 1 3 8