Performance evaluation of apache tajo
6 likes•2,052 views
This document evaluates the performance of Apache Tajo compared to other systems like Hive and Spark SQL. It runs benchmarks using the TPC-DS benchmark on Google Cloud Platform. The evaluation shows that Tajo is generally the fastest system and scales well to larger datasets and clusters. Some queries that involve complex joins and aggregations see more room for improvement in Tajo. The document encourages contributors to get involved in optimizing and enhancing Tajo's performance.
Performance evaluation of apache tajo
- 1. Performance Evaluation of Apache Tajo Jihoon Son / Gruter Inc.
- 2. Goals ● Performance comparison with other systems ● Scalability test of Tajo 2
- 3. Evaluation on Cloud Environment ● Google Cloud Platform ○ Instance type: n1-standard-8 ■ 8 core, 30GB RAM 3
- 4. TPC-DS ● Data ○ 24 tables ■ Plain text format ■ Stored on Google Cloud Storage ● Query ○ Which can be executed on every system without modifications ■ For Hive, 0.12 doesn't support implicit join, so every query had to be changed 4
- 5. Performance Comparison with Other Systems 5
- 6. Target Systems ● Tajo (0.11.0) ○ Default configuration provided by GCP ■ Use the whole cpu and memory ● Hive (0.12) ○ Baseline performance ○ Default configuration provided by GCP ■ Use the whole cpu and memory 6
- 7. Target Systems ● Spark-SQL (1.5.0) ○ Default configuration provided by GCP ■ Use the whole cpu and memory ■ Tungsten enabled by default ○ spark.sql.shuffle.partitions is adjusted for better performance 7
- 8. SF 1000, 50 instances 8
- 9. SF 1000, 50 instances 9 Whole data can be loaded into memory
- 10. SF 1000, 50 instances 10 Tajo is always the fastest one except q50
- 11. SF 1000, 50 instances 11 Cannot be run on 1TB
- 12. SF 10000, 50 instances 12
- 13. SF 10000, 50 instances 13 Complex query Join of 2 tables and 1 derived table (5 tables Join) + Aggregation + Sort
- 14. SF 10000, 50 instances 14 Already fixed in 0.11.1 (TAJO-1271, TAJO-1950, TAJO-1983, TAJO-2000)
- 16. SF 10000 16
- 18. SF 10000 18 q55 is a very simple query, so Tajo does not need huge resource to execute it.
- 19. Get Involved! ● We are recruiting contributors! ● General ○ http://tajo.apache.org/ ● Getting Started ○ http://tajo.apache.org/docs/current/getting_started.html ● Downloads ○ http://tajo.apache.org/downloads.html ● Issue tracker ○ http://issues.apache.org/jira/browse/TAJO ● Join the mailing list ○ [email protected] ○ [email protected] 19
- 20. Q & A 20