Impala use case @ Zoosk
8 likes2,098 views
Zoosk is an online dating platform utilizing behavioral data for matchmaking across 80 countries and 25 languages. The document discusses big data architecture, focusing on data pipeline design, machine learning, and performance comparisons between Impala and Hive for data processing. It highlights the strategic use of Impala for immediate queries and Hive for large-scale processing tasks.
Impala use case @ Zoosk
- 1. Zoosk ! Big Data Architecture MARCH 1, 2016 Marilson Campos Sr. Data Architect – Data Engineering
- 2. 2Zoosk. Inc. | About Zoosk • Online dating site & mobile apps • Uses behavioral data to match couples. • 80 Countries. • 25 languages.
- 3. 3Zoosk. Inc. | About Me ! Email: [email protected] - Led “Large data” projects since 1998. - Hadoop since 2009 - Focus on: • Complex data pipeline design. • Scaling machine learning processing. - Data pipeline projects • #1 search engine Latin America Acquired by Yahoo. • Indexing all blogs in US Buzzlogic. • Machine learning pipelines at scale Rocketfuel. • Machine learning & Enterprise Data bus Zoosk.
- 4. 4Zoosk. Inc. | Agenda • Big data @ Zoosk. • Data pipelines. • Leveraging Impala. • Impala and Hive trade offs. • Q & A.
- 5. 5Zoosk. Inc. | 1. Big Data @ Zoosk
- 6. 6Zoosk. Inc. | 2. Data as a product
- 7. 7Zoosk. Inc. | 3. Usable data sets x Temporary data
- 8. 8Zoosk. Inc. | 4. Hive Streaming
- 9. 9Zoosk. Inc. | 5. Two cluster configuration
- 10. 10Zoosk. Inc. | 6. Impala performance Impala has the largest impact. Parquet adds: +30% performance on Hive. 4x performance on impala. Some parquet tables will be a lot smaller than standard format. In some cases 1/6th of the size.
- 11. 11Zoosk. Inc. | 7. Impala vs. Hive (again!) Hive Impala System type Batch processing. Interactive. Resource Utilization Conservative. Aggressive. Predictable time completion. Yes. Almost linear. Works well with complex queries. Not always. Generally very fast, but can degenerate in queries with lots of joins. Allow code injection Yes. Injection of M/R functions with Hive streaming. No. Does not rely on Map/ Reduce. User defined functions Yes. Yes.
- 12. 12Zoosk. Inc. | 8. Where to use Impala or Hive ? Impala: - We consider it a premium service. - Use for queries performing aggregations where immediate response brings value. (*) - In some cases, it makes sense for apps to call it. Hive: - Standard service. - Use for processing the data pipeline. - Use integrated with Java, Python and R. - Use for complex queries on very large sets. (*) Everybody wants their queries to run faster. This is not a valid reason. J
- 13. 13Zoosk. Inc. | Q & A