The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Move to Spark-(Yandu Oppacher, Shopify)
4 likes•2,515 views
The document discusses the challenges faced by Shopify in using its existing data warehouse and ETL processes due to increasing data volume and complexity. It describes Shopify's attempts to use Pig and Luigi as well as Platfora to address these issues, but notes they did not meet Shopify's needs. Shopify then moved to using Spark due to its fast performance, nice development model using Python, and ability to better handle their data and query complexity. The summary provides an overview of why Shopify changed its data warehousing approach and the key technology it adopted.
Related topics:1 of 13
Downloaded 118 times
More Related Content
What's hot (20)
PPTX
Using Hadoop to build a Data Quality Service for both real-time and batch dataDataWorks Summit/Hadoop SummitÂ
Using Hadoop to build a Data Quality Service for both real-time and batch dataDataWorks Summit/Hadoop Summit
Â
Ad
Viewers also liked (20)
Ad
Similar to The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Move to Spark-(Yandu Oppacher, Shopify) (20)
More from Spark Summit (20)
Recently uploaded (20)
The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Move to Spark-(Yandu Oppacher, Shopify)
- 1. The Little Warehouse That Couldn’t Or: How We Learned to Stop Worrying and Move to Spark 1 Yandu Oppacher (@yandu) Data Infrastructure
- 3. ETL Warehouse Reporting August 2013 TilllerRuby Vertica 3
- 4. Why we had to move • Data volume • Data/Query complexity • Performance issues 4
- 5. Couple of false starts 5 Pig + Luigi Pig + Oozie Platfora
- 6. –platfora.com “Without coding or ETL, data warehousing, BI tools, or breaking a sweat.” 6
- 7. Enter Spark • Fast • Nice development model • Python 7
- 9. GMV A Case Study 9 165,000+ ACTIVE SHOPIFY MERCHANTS $8 BILLION+ CUMULATIVE GMV
- 10. Growing pains • Joins • Groupings • General data skew • Getting to know python’s performance quirks 10
- 11. Starscream 11 • specialized joins • resolvers • range • cassandra • overby • contracts • incrementalized fact builds
- 13. Thank you 13 Yandu Oppacher (@yandu) Data Infrastructure