Abstract image of a woman sitting on books and studying on a laptop

Apache Drill talk ApacheCon 2018

A
Aman Sinha

Apache Drill is a distributed SQL query engine that enables fast analytics over NoSQL databases and distributed file systems. It has a plugin-based architecture that allows it to access different data sources. For NoSQL databases, Drill leverages secondary indexes to generate index-based query plans for predicates on non-key columns. For distributed file systems like HDFS, Drill performs partition pruning based on directory metadata and filter pushdown based on Parquet row group statistics to speed up queries. Drill's extensible framework allows data sources to provide metadata like indexes, statistics, and partitioning functions to optimize query execution.

Software
1 of 40
Downloaded 10 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
How Apache Drill enables fast analytics over NoSQL databases and distributed file systems Aman Sinha
About me • Apache Drill PMC and Apache Calcite PMC. Past PMC chair of Drill. • Employment: – Currently at MapR, Santa Clara. Industry’s leading platform for AI and Analytics. – Past: ParAccel (columnar DB whose technology powers Amazon Redshift), IBM Silicon Valley Lab • Main areas of interest: SQL query processing for RDBMS, NoSQL, Hadoop • Contact: amansinha@apache.org, Github: https://github.com/amansinha100
Talk Outline • Motivation • Brief overview of Drill Architecture • Improving query performance on NoSQL databases • Improving query performance on distributed file systems • Best practices
Motivation NoSQL DB tables DFS data (Parquet, CSV, JSON files) Data ingest ETL Additional data ingest ● Enterprises often have heterogeneous data sources in the same cluster ○ Typically, NoSQL distributed databases and flat files in a distributed file system
Need a Unified Query Capability NoSQL DB tables DFS data (Parquet, CSV, JSON files) Data ingest ETL Additional data ingest Query Engine Provide SQL operations within and across data sources
Key Challenge NoSQL DB tables DFS data (Parquet, CSV, JSON files) Data ingest ETL Additional data ingest Query Engine How to make these couplings highly performant in a cluster deployment ?
Brief Overview of Drill Architecture
Apache Drill Introduction • Distributed SQL query engine – Originally inspired in part by Google Dremel paper – Became TLP in December 2014. Excerpt from the ASF announcement:: “World's first schema-free SQL query engine brings self-service data exploration to Apache Hadoop™” – Supports wide range of data sources in the form of Plugins • Data sources – Distributed file systems: HDFS, MapR-FS – NoSQL databases: HBase, MapR-DB (binary and JSON), MongoDB – Streaming: Kafka – JDBC – Hive (tables are managed by Hive but querying is done through Drill for interactive performance) – Others: OpenTSDB, Kudu, Amazon S3 • File data formats: Parquet, CSV, TSV, JSON • Extensible
- Sub-directory - HBase namespace - Hive database SELECT * FROM dfs.yelp.`business.json` Workspace - Pathnames - HBase table - Hive table Table - File system - HBase - Hive Storage Plugin Components of a FROM Clause in Drill
HDFS HBase MapR- DB HDFS HBase MapR- DB DRILLBIT (any drillbit can be Foreman) HDFS HBase MapR- DB DRILLBIT DRILLBIT JDBC/ODBC Client Web Console Data Sources (exposed to Drill as Storage/Format Plugins) Zookeeper (could be co-resident with Drillbit nodes) Cluster Architecture with Drill BI Tools
Architecture Summary • Schema-on-read • No centralized metastore • Fully Java based • In-memory columnar processing • Mostly off-heap memory management (negligible GC overhead) • Code generation for run-time operators • Optimistic, pipelined execution model • Spill to disk for blocking operations under memory pressure • Integrated with YARN for resource management • Provides a strong framework for UDFs
Operator Execution Model Receiver Sender Fragment boundary An Exchange operator Scan Op Root Op ● Iterator based model: PULL model within a major fragment, PUSH model across major fragments ○ Several minor fragments (threads) constitute a major fragment ● Parent operator calls next() on its child ● Data is processed in ‘Record Batches’ (upper bounded to 64K records per batch) ● Data flow is pipelined until a blocking operator is encountered (Sort, Hash) ● Operators do run-time code generation for each new Schema
Drill Web UI with Operator Profiles
Columnar In-memory Format RecordBatch BatchSchema (List of MaterializedField) ● Predecessor to Apache Arrow ● MaterializedField has a name and a data type ● Value Vectors are facades to byte buffers provided by underlying Netty ● Fixed-width, Variable-width value vectors ● Various data types ● 3 sub types of Value Vectors ○ Optional (Nullable) ○ Required (Non-Nullable) ○ Repeated ● Code generation: Each operator (except Scan) generates Java code at run-time based on RecordBatch schema ValueVectors
Data Locality and Parallelization ● Drill tries to ensure that scan threads are co-located with the data file ● Encapsulated as ‘affinity’ List<EndpointAffinity> getOperatorAffinity(); Core interface methods for parallelization at the scan level int getMinParallelizationWidth(); int getMaxParallelizationWidth();
Back to the Performance Question … • Exploiting locality helps reduce network data transfers • Parallelization improves CPU utilization • What about disk I/O ? – Rest of the talk will discuss this in the context of 2 types of data sources: • NoSQL distributed databases • Distributed file systems (focus on Parquet format)
Improving query performance on NoSQL databases
Secondary Index: Background • HBase and MapR-DB tables have primary key (row key) column – This column values are sorted – Efficient range pruning is done for rowkey predicates: WHERE rowkey BETWEEN 150 AND 350 • Secondary columns (e.g ‘State’) values are not sorted – Predicate WHERE state = ‘CA’ need full table scan ! • Solution ? Create secondary index ‘tables’ – PK of index table is a concatenation: state + rowkey : • (AZ_500), … (CA_250), (CA_300), ... 100 200 300 400 1000 Sequential scan only this range regions rowkey
Secondary Index • NoSQL DBs supporting secondary index – MongoDB – MapR-DB JSON – HBase + Phoenix – Couchbase – Cassandra • What’s missing ? – Other than Hbase + Phoenix, others don’t have an ANSI SQL interface – There’s a need for a generalized cost-based index planning and execution framework – A key requirement: • Framework must be able to support ‘global’ non-covering indexes, not just covering index
Leveraging Secondary Index via Drill • Storage/Format plugin whose backend supports secondary indexing – Reference implementation is with MapR-DB JSON • Index metadata is exposed to Drill planner through well defined interfaces • Statistics (if available) are also exposed • New run-time operators added for executing index plans • Drill Planner extends Apache Calcite’s planner with additional rules for index planning. – Generates index-access plans and compare them cost-wise (using Apache Calcite’s Volcano planner) to full table scan and with each other • Feature will be available in upcoming Drill release (please follow JIRA: DRILL-6381, PR: https://github.com/apache/drill/pull/1466)
Types of Queries Eligible for Index Planning • WHERE clause with local filters – <, >, =, BETWEEN – IN, LIKE – Eligible ANDed conditions – Eligible ORed conditions – Certain types of functions, e.g CAST(zipcode as BIGINT) = 12345 (only if data source supports functional indexes) • ORDER BY • GROUP BY (using StreamingAggregate) • JOIN (using MergeJoin)
Leading Prefix Columns and Statistics • Query predicate: WHERE a > 10 AND b < 30 a b c Composite index key a c b Comments Leading prefix columns ‘a’ and ‘b’ hence full predicate is eligible for index range pruning Leading prefix columns only ‘a’ hence only a > 10 eligible for index range pruning • Statistics – Index planning relies on statistics exposed by underlying DB (in future Drill will collect these) – Each individual conjunct may have an estimated row count • a > 10 : 2M rows • b < 30: 5M rows • Total row count of table: 100M rows Selectivity = 0.02 * 0.05 IndexSelector uses these for cost-based analysis
Covering vs. Non-Covering Indexes • Covering: All columns referenced in the query are available in the index – Easier to handle by the planner. Generate an index-only plan. • Non-Covering: Only a subset of the columns are available in the index – Needs more supporting infrastructure from planner and executor • RowKey join • Restricted (‘skip’ scan) • Range partitioning with a plugin-specific partitioning function
Join-back to Primary Table (for non-covering indexes) • SELECT * FROM T WHERE a > 10 AND b < 30 • Composite key index on {a, b} • How to produce the remaining (‘star’) columns ? Index Scan a > 10 AND b < 30 RowKey Join Range Partition Exchange Restricted (a.k.a ‘skip’ ) scan Return rows Row keys Supply row keys Do ‘bucketing’ of row keys belonging to the same region/tablet (this needs knowledge of the tablet map)
Index Intersection • SELECT * FROM T WHERE a > 10 AND b < 30 • Suppose single key index exists on ‘a’ and ‘b’ Index Scan a > 10 Index Scan b < 30 Intersect HashJoin Broadcast Exchange RowKey Join Range Partition Exchange Restricted Scan
Example: GROUP BY queries SELECT a, b, SUM(c) FROM T WHERE .. GROUP BY a, b • Suppose composite key index exists on {a, b} • Planner will create 2 types of plans – HashAggregate plan which does hashing on {a, b} – StreamingAggregate plan which relies on sorted input on {a, b} • The sorted input is provided by the index • This plan is typically cheaper than the HashAggregate plan
Sample interfaces to be implemented by plugin ● DbGroupScan ○ IndexCollection getSecondaryIndexCollection(RelNode scan) ○ DbGroupScan getRestrictedScan(List<SchemaPath> columns); ○ PartitionFunction getRangePartitionFunction(List<FieldReference> refList) ○ PluginCost getPluginCostModel() ● PluginCost ○ int getSequentialBlockReadCost(GroupScan scan) ○ int getRandomBlockReadCost(GroupScan scan) ● IndexDiscover ○ IndexCollection getTableIndex(String tableName) ● IndexDefinition ○ List<LogicalExpression> getRowKeyColumns() ○ List<LogicalExpression> getIndexColumns() ○ List<LogicalExpression> getNonIndexColumns() ○ Map<LogicalExpression, RelFieldCollation> getCollationMap()
Improving query performance on Distributed File Systems
Directory based partition pruning Orders / 2015 / Jan / 1.parquet Orders / 2015 / Jan / 2.parquet dir0 dir1 Orders / 2018 / Aug / 30.parquet Orders / 2018 / Aug / 31.parquet Implicit metadata columns CREATE VIEW V1 AS SELECT `dir0` as `year`, `dir1` as `month`, o_custkey, o_totalprice, o_orderstatus FROM dfs.`Orders`; SELECT o_custkey, SUM(o_totalprice) FROM V1 WHERE `year` = 2018 AND `month` = ‘July’ GROUP BY o_custkey; ● 2 ways to query the implicit directory columns in filter conditions ○ WHERE dir0 = 2018 ○ Via a view with aliasing as below
File based partition pruning (Parquet) • CREATE TABLE T1 (a, b, c) PARTITION BY a, b AS SELECT col1 as a, col2 as b, col3 as c FROM … • CTAS with Partition-By creates separate files, each file with 1 partition value • Multiple files may be created for the same partition value • Can prune entire file (row group) based on filter on partitioning column
Handling Complex Predicates Boolean operator AND, OR Comparison operators <, >, = etc. ● The leaf nodes are either partitioning columns or non-partitioning columns or constants. E.g `year` = 2015 ● Given arbitrary expression tree, Drill will determine what predicates can be pushed down safely for partition pruning. ○ OR can only be pushed if both sides are eligible
Parquet Row Group Metadata "path" : "/Users/asinha/data/table3_meta/0_0_3.parquet", "length" : 245, "rowGroups" : [ { "start" : 4, "length" : 91, "rowCount" : 1, "hostAffinity" : { "localhost" : 1.0 }, "columns" : [ { "name" : "`c1`", "primitiveType" : "INT32", "originalType" : null, "nulls" : 0, "max" : 5, "min" : 5 }, { "name" : "`c3`", "primitiveType" : "BINARY", "originalType" : "UTF8", "nulls" : 0, "max" : { "bytes" : "IHl5" }, "min" : { "bytes" : "IHl5" } } ] ● Drill has to read RG metadata during planning time ○ Get host affinity, stats etc. ● Doesn’t scale for hundreds of thousands of files ● One solution: Cache the metadata on disk by running explicit ‘REFRESH TABLE METADATA <table>’ command ● Can improve planning time by 10x
Parquet Filter Pushdown • Applied during planning process – Based on MIN/MAX statistics of the column in Parquet Row Group – Eliminates row groups after intersecting ANDed filters • Pushdown is slightly different from partition pruning – Pushdown is applicable for all columns that have min/max statistics, not just partitioning columns • Pushdown applicable for scalar columns and complex type columns – NOTE: Filter on VARCHAR and DECIMAL types are not currently pushed down but support is being added for upcoming Drill release
Filter Pushdown and Pruning • Complex Types Support SELECT * FROM table WHERE col_name.nested_col_int = 23 SELECT * FROM table WHERE col_name.nested_col_bln is not null SELECT * FROM table WHERE col_name.nested_col_arr[0] > 10 • Limitations: – Logical optimizations can be applied only for the columns with available statistics SELECT * FROM table WHERE col_name.nested_col_arr[2] is null – Other limitations are the same as for the Filter Pushdown for scalar data types
Filter Pushdown and Pruning ● Transitive Closure SELECT * FROM dfs.`/tmp/first` t1 JOIN dfs.`/tmp/SECOND` t2 ON t1.`MONTH` = t2.`MONTH` WHERE t2.`MONTH` = 4 Predicate pushed down on both sides of Join More examples: SELECT * FROM t1 JOIN t2 T2 ON t1.a = t2.a WHERE t2.a NOT IN (4, 6) SELECT * FROM t1 JOIN (SELECT a, b FROM table WHERE a = 1987 AND b = 5) t2 ON t1.a = t2.a AND t1.b = t2.b
Run-Time Filter Pushdown for HashJoin SELECT a1 FROM fact WHERE b1 IN (SELECT b2 FROM dim WHERE dim_col = ‘CA’) Hash Join (b1 = b2) Scan ‘Fact’ table Scan ‘Dim’ table Filter dim_col = ‘CA’ Hash Exchange Hash Exchange Build sideProbe side Qualified b2 values Bloom filter on b1 values Goal: Substantially reduce network transfer by this exchange
Best Practices for Parquet Data Layout • Large number of small files or fewer large files ? – Recommended to have fewer large files subject to constraints such as Parquet block size and desired parallelism • Too many small files (order of few KBs) may cause data skew and affect performance • Too few large files reduces parallelism … need to determine sweet spot • Block size considerations – Best to have Parquet block size equal or less than HDFS/MapR-FS block size • Larger block size will spread 1 row group over 2 or more nodes incurring remote reads. • Preferred compression type – Snappy is preferred for performance, gzip for space but higher cost to decompress – Default for Drill - snappy
Summary • Apache Drill’s extensible architecture makes it easy to support multiple data sources – While exploiting data locality and parallelism suited to the data source • Fast analytics on NoSQL databases using a new generalized framework to do index based planning and execution • Fast analytics on distributed file system tables by doing intelligent partitioning and filter pushdown
Useful links • Drill resources – http://drill.apache.org – Twitter: @ApacheDrill – Mailing lists: • user@drill.apache.org • dev@drill.apache.org • Get involved with the Drill community !
Q & A

More Related Content

PDF
Accelerating SQL queries in NoSQL Databases using Apache Drill and Secondary ...
Aman Sinha
 
PDF
Automobile Route Matching with Dynamic Time Warping Using PySpark with Cather...
Databricks
 
PDF
Large-Scaled Telematics Analytics in Apache Spark with Wayne Zhang and Neil P...
Databricks
 
PDF
ONE FOR ALL! Using Apache Calcite to make SQL smart
Evans Ye
 
PDF
Cost-based Query Optimization
DataWorks Summit/Hadoop Summit
 
PDF
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
Julian Hyde
 
PDF
Apache Spark Core—Deep Dive—Proper Optimization
Databricks
 
PDF
Apache Carbondata: An Indexed Columnar File Format for Interactive Query with...
Spark Summit
 
Accelerating SQL queries in NoSQL Databases using Apache Drill and Secondary ...
Aman Sinha
 
Automobile Route Matching with Dynamic Time Warping Using PySpark with Cather...
Databricks
 
Large-Scaled Telematics Analytics in Apache Spark with Wayne Zhang and Neil P...
Databricks
 
ONE FOR ALL! Using Apache Calcite to make SQL smart
Evans Ye
 
Cost-based Query Optimization
DataWorks Summit/Hadoop Summit
 
Cost-based Query Optimization in Apache Phoenix using Apache Calcite
Julian Hyde
 
Apache Spark Core—Deep Dive—Proper Optimization
Databricks
 
Apache Carbondata: An Indexed Columnar File Format for Interactive Query with...
Spark Summit
 

What's hot (20)

PDF
A smarter Pig: Building a SQL interface to Apache Pig using Apache Calcite
Julian Hyde
 
PDF
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Spark Summit
 
PDF
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Christian Tzolov
 
PPTX
Producing Spark on YARN for ETL
DataWorks Summit/Hadoop Summit
 
PDF
SparkSQL: A Compiler from Queries to RDDs
Databricks
 
PPTX
Optimizing Apache Spark SQL Joins
Databricks
 
PDF
Physical Plans in Spark SQL
Databricks
 
PDF
Tactical data engineering
Julian Hyde
 
PDF
Apache CarbonData:New high performance data format for faster data analysis
liang chen
 
PDF
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Julian Hyde
 
PPTX
Spark meetup v2.0.5
Yan Zhou
 
PDF
Apache Calcite: One Frontend to Rule Them All
Michael Mior
 
PDF
Don't optimize my queries, organize my data!
Julian Hyde
 
PDF
Cost-Based Optimizer in Apache Spark 2.2 Ron Hu, Sameer Agarwal, Wenchen Fan ...
Databricks
 
PDF
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Julian Hyde
 
PPTX
Spark etl
Imran Rashid
 
PDF
Improving Spark SQL at LinkedIn
Databricks
 
PDF
Jump Start into Apache Spark (Seattle Spark Meetup)
Denny Lee
 
PPTX
Apache Phoenix: Transforming HBase into a SQL Database
DataWorks Summit
 
PDF
Recent Developments In SparkR For Advanced Analytics
Databricks
 
A smarter Pig: Building a SQL interface to Apache Pig using Apache Calcite
Julian Hyde
 
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
Spark Summit
 
Using Apache Calcite for Enabling SQL and JDBC Access to Apache Geode and Oth...
Christian Tzolov
 
Producing Spark on YARN for ETL
DataWorks Summit/Hadoop Summit
 
SparkSQL: A Compiler from Queries to RDDs
Databricks
 
Optimizing Apache Spark SQL Joins
Databricks
 
Physical Plans in Spark SQL
Databricks
 
Tactical data engineering
Julian Hyde
 
Apache CarbonData:New high performance data format for faster data analysis
liang chen
 
Apache Calcite: A Foundational Framework for Optimized Query Processing Over ...
Julian Hyde
 
Spark meetup v2.0.5
Yan Zhou
 
Apache Calcite: One Frontend to Rule Them All
Michael Mior
 
Don't optimize my queries, organize my data!
Julian Hyde
 
Cost-Based Optimizer in Apache Spark 2.2 Ron Hu, Sameer Agarwal, Wenchen Fan ...
Databricks
 
Data all over the place! How SQL and Apache Calcite bring sanity to streaming...
Julian Hyde
 
Spark etl
Imran Rashid
 
Improving Spark SQL at LinkedIn
Databricks
 
Jump Start into Apache Spark (Seattle Spark Meetup)
Denny Lee
 
Apache Phoenix: Transforming HBase into a SQL Database
DataWorks Summit
 
Recent Developments In SparkR For Advanced Analytics
Databricks
 
Ad

Similar to Apache Drill talk ApacheCon 2018 (20)

PDF
10 Reasons to Start Your Analytics Project with PostgreSQL
Satoshi Nagayasu
 
PPT
Hive Evolution: ApacheCon NA 2010
John Sichi
 
PDF
Apache Hadoop 1.1
Sperasoft
 
PPTX
Strata NY 2018: The deconstructed database
Julien Le Dem
 
PPTX
Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
Fwdays
 
PDF
Spark and Couchbase: Augmenting the Operational Database with Spark
Spark Summit
 
PPTX
SQL Server 2014 Memory Optimised Tables - Advanced
Tony Rogerson
 
PDF
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Xu Jiang
 
PDF
From flat files to deconstructed database
Julien Le Dem
 
PPT
Hadoop hbase introduction
Jakub Stransky
 
PPT
No sql Database
mymail2ashok
 
PPTX
Emerging technologies /frameworks in Big Data
Rahul Jain
 
PPTX
SQL Server 2014 In-Memory OLTP
Tony Rogerson
 
PPTX
HBaseConAsia2018 Track2-4: HTAP DB-System: AsparaDB HBase, Phoenix, and Spark
Michael Stack
 
PPTX
Best Practices and Performance Tuning of U-SQL in Azure Data Lake (SQL Konfer...
Michael Rys
 
PDF
SQL on Hadoop
nvvrajesh
 
PDF
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
Data Con LA
 
PPT
A Scalable Data Transformation Framework using the Hadoop Ecosystem
Serendio Inc.
 
PDF
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
PPTX
Challenges of Implementing an Advanced SQL Engine on Hadoop
DataWorks Summit
 
10 Reasons to Start Your Analytics Project with PostgreSQL
Satoshi Nagayasu
 
Hive Evolution: ApacheCon NA 2010
John Sichi
 
Apache Hadoop 1.1
Sperasoft
 
Strata NY 2018: The deconstructed database
Julien Le Dem
 
Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...
Fwdays
 
Spark and Couchbase: Augmenting the Operational Database with Spark
Spark Summit
 
SQL Server 2014 Memory Optimised Tables - Advanced
Tony Rogerson
 
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Xu Jiang
 
From flat files to deconstructed database
Julien Le Dem
 
Hadoop hbase introduction
Jakub Stransky
 
No sql Database
mymail2ashok
 
Emerging technologies /frameworks in Big Data
Rahul Jain
 
SQL Server 2014 In-Memory OLTP
Tony Rogerson
 
HBaseConAsia2018 Track2-4: HTAP DB-System: AsparaDB HBase, Phoenix, and Spark
Michael Stack
 
Best Practices and Performance Tuning of U-SQL in Azure Data Lake (SQL Konfer...
Michael Rys
 
SQL on Hadoop
nvvrajesh
 
Big Data Day LA 2016/ Big Data Track - How To Use Impala and Kudu To Optimize...
Data Con LA
 
A Scalable Data Transformation Framework using the Hadoop Ecosystem
Serendio Inc.
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Databricks
 
Challenges of Implementing an Advanced SQL Engine on Hadoop
DataWorks Summit
 
Ad

Recently uploaded (20)

PDF
Top 10 Software Development Trends to Watch in 2025 🚀.pdf
KodekX
 
PDF
Designing Intelligence for the Shop Floor.pdf
nikglvk
 
PPTX
Introduction to Windows Operating System
ssuser1028f8
 
PPTX
Computer Software and OS of computer science of grade 11.pptx
GauravDahal9
 
PDF
EaseUS PDF Editor Pro 6.2.0.2 Crack with License Key 2025
halimaanam8
 
PPTX
WiFi Honeypot Detecscfddssdffsedfseztor.pptx
singlesrihari
 
PPTX
Monitoring Stack: Grafana, Loki & Promtail
GhanshyamSwami3
 
PDF
Multiverse AI Review 2025: Access All TOP AI Model-Versions!
Moshiur Rahman Jisan
 
PDF
Visual explanation of Dijkstra's Algorithm using Python
Universidad Nacional de Ingenieria
 
PPTX
Oracle Fusion HCM Cloud Demo for Beginners
DharanOracleerp
 
PDF
How Tridens DevSecOps Ensures Compliance, Security, and Agility
Tridens
 
PPTX
Tech Workshop Escape Room Tech Workshop
muthuiyad
 
PPTX
Cybersecurity: Protecting the Digital World
SURULIAPPANPREMKMAR
 
PPTX
AMADEUS TRAVEL AGENT SOFTWARE | AMADEUS TICKETING SYSTEM
philipnathen82
 
PPTX
Log360_SIEM_Solutions Overview PPT_Feb 2020.pptx
nonameist3434
 
DOCX
How to Use SharePoint as an ISO-Compliant Document Management System
Sharepoint Designs
 
PDF
AI-Powered Threat Modeling: The Future of Cybersecurity by Arun Kumar Elengov...
Arun Kumar Elengovan
 
PDF
Salesforce Agentforce AI Implementation.pdf
Robert Mark
 
PPTX
CNN LeNet5 Architecture: Neural Networks
DrRPonnusamyCIT
 
PDF
How to Make Money in the Metaverse_ Top Strategies for Beginners.pdf
Herond Labs
 
Top 10 Software Development Trends to Watch in 2025 🚀.pdf
KodekX
 
Designing Intelligence for the Shop Floor.pdf
nikglvk
 
Introduction to Windows Operating System
ssuser1028f8
 
Computer Software and OS of computer science of grade 11.pptx
GauravDahal9
 
EaseUS PDF Editor Pro 6.2.0.2 Crack with License Key 2025
halimaanam8
 
WiFi Honeypot Detecscfddssdffsedfseztor.pptx
singlesrihari
 
Monitoring Stack: Grafana, Loki & Promtail
GhanshyamSwami3
 
Multiverse AI Review 2025: Access All TOP AI Model-Versions!
Moshiur Rahman Jisan
 
Visual explanation of Dijkstra's Algorithm using Python
Universidad Nacional de Ingenieria
 
Oracle Fusion HCM Cloud Demo for Beginners
DharanOracleerp
 
How Tridens DevSecOps Ensures Compliance, Security, and Agility
Tridens
 
Tech Workshop Escape Room Tech Workshop
muthuiyad
 
Cybersecurity: Protecting the Digital World
SURULIAPPANPREMKMAR
 
AMADEUS TRAVEL AGENT SOFTWARE | AMADEUS TICKETING SYSTEM
philipnathen82
 
Log360_SIEM_Solutions Overview PPT_Feb 2020.pptx
nonameist3434
 
How to Use SharePoint as an ISO-Compliant Document Management System
Sharepoint Designs
 
AI-Powered Threat Modeling: The Future of Cybersecurity by Arun Kumar Elengov...
Arun Kumar Elengovan
 
Salesforce Agentforce AI Implementation.pdf
Robert Mark
 
CNN LeNet5 Architecture: Neural Networks
DrRPonnusamyCIT
 
How to Make Money in the Metaverse_ Top Strategies for Beginners.pdf
Herond Labs
 

Apache Drill talk ApacheCon 2018

  • 1. How Apache Drill enables fast analytics over NoSQL databases and distributed file systems Aman Sinha
  • 2. About me • Apache Drill PMC and Apache Calcite PMC. Past PMC chair of Drill. • Employment: – Currently at MapR, Santa Clara. Industry’s leading platform for AI and Analytics. – Past: ParAccel (columnar DB whose technology powers Amazon Redshift), IBM Silicon Valley Lab • Main areas of interest: SQL query processing for RDBMS, NoSQL, Hadoop • Contact: [email protected], Github: https://github.com/amansinha100
  • 3. Talk Outline • Motivation • Brief overview of Drill Architecture • Improving query performance on NoSQL databases • Improving query performance on distributed file systems • Best practices
  • 4. Motivation NoSQL DB tables DFS data (Parquet, CSV, JSON files) Data ingest ETL Additional data ingest ● Enterprises often have heterogeneous data sources in the same cluster ○ Typically, NoSQL distributed databases and flat files in a distributed file system
  • 5. Need a Unified Query Capability NoSQL DB tables DFS data (Parquet, CSV, JSON files) Data ingest ETL Additional data ingest Query Engine Provide SQL operations within and across data sources
  • 6. Key Challenge NoSQL DB tables DFS data (Parquet, CSV, JSON files) Data ingest ETL Additional data ingest Query Engine How to make these couplings highly performant in a cluster deployment ?
  • 7. Brief Overview of Drill Architecture
  • 8. Apache Drill Introduction • Distributed SQL query engine – Originally inspired in part by Google Dremel paper – Became TLP in December 2014. Excerpt from the ASF announcement:: “World's first schema-free SQL query engine brings self-service data exploration to Apache Hadoop™” – Supports wide range of data sources in the form of Plugins • Data sources – Distributed file systems: HDFS, MapR-FS – NoSQL databases: HBase, MapR-DB (binary and JSON), MongoDB – Streaming: Kafka – JDBC – Hive (tables are managed by Hive but querying is done through Drill for interactive performance) – Others: OpenTSDB, Kudu, Amazon S3 • File data formats: Parquet, CSV, TSV, JSON • Extensible
  • 9. - Sub-directory - HBase namespace - Hive database SELECT * FROM dfs.yelp.`business.json` Workspace - Pathnames - HBase table - Hive table Table - File system - HBase - Hive Storage Plugin Components of a FROM Clause in Drill
  • 10. HDFS HBase MapR- DB HDFS HBase MapR- DB DRILLBIT (any drillbit can be Foreman) HDFS HBase MapR- DB DRILLBIT DRILLBIT JDBC/ODBC Client Web Console Data Sources (exposed to Drill as Storage/Format Plugins) Zookeeper (could be co-resident with Drillbit nodes) Cluster Architecture with Drill BI Tools
  • 11. Architecture Summary • Schema-on-read • No centralized metastore • Fully Java based • In-memory columnar processing • Mostly off-heap memory management (negligible GC overhead) • Code generation for run-time operators • Optimistic, pipelined execution model • Spill to disk for blocking operations under memory pressure • Integrated with YARN for resource management • Provides a strong framework for UDFs
  • 12. Operator Execution Model Receiver Sender Fragment boundary An Exchange operator Scan Op Root Op ● Iterator based model: PULL model within a major fragment, PUSH model across major fragments ○ Several minor fragments (threads) constitute a major fragment ● Parent operator calls next() on its child ● Data is processed in ‘Record Batches’ (upper bounded to 64K records per batch) ● Data flow is pipelined until a blocking operator is encountered (Sort, Hash) ● Operators do run-time code generation for each new Schema
  • 13. Drill Web UI with Operator Profiles
  • 14. Columnar In-memory Format RecordBatch BatchSchema (List of MaterializedField) ● Predecessor to Apache Arrow ● MaterializedField has a name and a data type ● Value Vectors are facades to byte buffers provided by underlying Netty ● Fixed-width, Variable-width value vectors ● Various data types ● 3 sub types of Value Vectors ○ Optional (Nullable) ○ Required (Non-Nullable) ○ Repeated ● Code generation: Each operator (except Scan) generates Java code at run-time based on RecordBatch schema ValueVectors
  • 15. Data Locality and Parallelization ● Drill tries to ensure that scan threads are co-located with the data file ● Encapsulated as ‘affinity’ List<EndpointAffinity> getOperatorAffinity(); Core interface methods for parallelization at the scan level int getMinParallelizationWidth(); int getMaxParallelizationWidth();
  • 16. Back to the Performance Question … • Exploiting locality helps reduce network data transfers • Parallelization improves CPU utilization • What about disk I/O ? – Rest of the talk will discuss this in the context of 2 types of data sources: • NoSQL distributed databases • Distributed file systems (focus on Parquet format)
  • 17. Improving query performance on NoSQL databases
  • 18. Secondary Index: Background • HBase and MapR-DB tables have primary key (row key) column – This column values are sorted – Efficient range pruning is done for rowkey predicates: WHERE rowkey BETWEEN 150 AND 350 • Secondary columns (e.g ‘State’) values are not sorted – Predicate WHERE state = ‘CA’ need full table scan ! • Solution ? Create secondary index ‘tables’ – PK of index table is a concatenation: state + rowkey : • (AZ_500), … (CA_250), (CA_300), ... 100 200 300 400 1000 Sequential scan only this range regions rowkey
  • 19. Secondary Index • NoSQL DBs supporting secondary index – MongoDB – MapR-DB JSON – HBase + Phoenix – Couchbase – Cassandra • What’s missing ? – Other than Hbase + Phoenix, others don’t have an ANSI SQL interface – There’s a need for a generalized cost-based index planning and execution framework – A key requirement: • Framework must be able to support ‘global’ non-covering indexes, not just covering index
  • 20. Leveraging Secondary Index via Drill • Storage/Format plugin whose backend supports secondary indexing – Reference implementation is with MapR-DB JSON • Index metadata is exposed to Drill planner through well defined interfaces • Statistics (if available) are also exposed • New run-time operators added for executing index plans • Drill Planner extends Apache Calcite’s planner with additional rules for index planning. – Generates index-access plans and compare them cost-wise (using Apache Calcite’s Volcano planner) to full table scan and with each other • Feature will be available in upcoming Drill release (please follow JIRA: DRILL-6381, PR: https://github.com/apache/drill/pull/1466)
  • 21. Types of Queries Eligible for Index Planning • WHERE clause with local filters – <, >, =, BETWEEN – IN, LIKE – Eligible ANDed conditions – Eligible ORed conditions – Certain types of functions, e.g CAST(zipcode as BIGINT) = 12345 (only if data source supports functional indexes) • ORDER BY • GROUP BY (using StreamingAggregate) • JOIN (using MergeJoin)
  • 22. Leading Prefix Columns and Statistics • Query predicate: WHERE a > 10 AND b < 30 a b c Composite index key a c b Comments Leading prefix columns ‘a’ and ‘b’ hence full predicate is eligible for index range pruning Leading prefix columns only ‘a’ hence only a > 10 eligible for index range pruning • Statistics – Index planning relies on statistics exposed by underlying DB (in future Drill will collect these) – Each individual conjunct may have an estimated row count • a > 10 : 2M rows • b < 30: 5M rows • Total row count of table: 100M rows Selectivity = 0.02 * 0.05 IndexSelector uses these for cost-based analysis
  • 23. Covering vs. Non-Covering Indexes • Covering: All columns referenced in the query are available in the index – Easier to handle by the planner. Generate an index-only plan. • Non-Covering: Only a subset of the columns are available in the index – Needs more supporting infrastructure from planner and executor • RowKey join • Restricted (‘skip’ scan) • Range partitioning with a plugin-specific partitioning function
  • 24. Join-back to Primary Table (for non-covering indexes) • SELECT * FROM T WHERE a > 10 AND b < 30 • Composite key index on {a, b} • How to produce the remaining (‘star’) columns ? Index Scan a > 10 AND b < 30 RowKey Join Range Partition Exchange Restricted (a.k.a ‘skip’ ) scan Return rows Row keys Supply row keys Do ‘bucketing’ of row keys belonging to the same region/tablet (this needs knowledge of the tablet map)
  • 25. Index Intersection • SELECT * FROM T WHERE a > 10 AND b < 30 • Suppose single key index exists on ‘a’ and ‘b’ Index Scan a > 10 Index Scan b < 30 Intersect HashJoin Broadcast Exchange RowKey Join Range Partition Exchange Restricted Scan
  • 26. Example: GROUP BY queries SELECT a, b, SUM(c) FROM T WHERE .. GROUP BY a, b • Suppose composite key index exists on {a, b} • Planner will create 2 types of plans – HashAggregate plan which does hashing on {a, b} – StreamingAggregate plan which relies on sorted input on {a, b} • The sorted input is provided by the index • This plan is typically cheaper than the HashAggregate plan
  • 27. Sample interfaces to be implemented by plugin ● DbGroupScan ○ IndexCollection getSecondaryIndexCollection(RelNode scan) ○ DbGroupScan getRestrictedScan(List<SchemaPath> columns); ○ PartitionFunction getRangePartitionFunction(List<FieldReference> refList) ○ PluginCost getPluginCostModel() ● PluginCost ○ int getSequentialBlockReadCost(GroupScan scan) ○ int getRandomBlockReadCost(GroupScan scan) ● IndexDiscover ○ IndexCollection getTableIndex(String tableName) ● IndexDefinition ○ List<LogicalExpression> getRowKeyColumns() ○ List<LogicalExpression> getIndexColumns() ○ List<LogicalExpression> getNonIndexColumns() ○ Map<LogicalExpression, RelFieldCollation> getCollationMap()
  • 28. Improving query performance on Distributed File Systems
  • 29. Directory based partition pruning Orders / 2015 / Jan / 1.parquet Orders / 2015 / Jan / 2.parquet dir0 dir1 Orders / 2018 / Aug / 30.parquet Orders / 2018 / Aug / 31.parquet Implicit metadata columns CREATE VIEW V1 AS SELECT `dir0` as `year`, `dir1` as `month`, o_custkey, o_totalprice, o_orderstatus FROM dfs.`Orders`; SELECT o_custkey, SUM(o_totalprice) FROM V1 WHERE `year` = 2018 AND `month` = ‘July’ GROUP BY o_custkey; ● 2 ways to query the implicit directory columns in filter conditions ○ WHERE dir0 = 2018 ○ Via a view with aliasing as below
  • 30. File based partition pruning (Parquet) • CREATE TABLE T1 (a, b, c) PARTITION BY a, b AS SELECT col1 as a, col2 as b, col3 as c FROM … • CTAS with Partition-By creates separate files, each file with 1 partition value • Multiple files may be created for the same partition value • Can prune entire file (row group) based on filter on partitioning column
  • 31. Handling Complex Predicates Boolean operator AND, OR Comparison operators <, >, = etc. ● The leaf nodes are either partitioning columns or non-partitioning columns or constants. E.g `year` = 2015 ● Given arbitrary expression tree, Drill will determine what predicates can be pushed down safely for partition pruning. ○ OR can only be pushed if both sides are eligible
  • 32. Parquet Row Group Metadata "path" : "/Users/asinha/data/table3_meta/0_0_3.parquet", "length" : 245, "rowGroups" : [ { "start" : 4, "length" : 91, "rowCount" : 1, "hostAffinity" : { "localhost" : 1.0 }, "columns" : [ { "name" : "`c1`", "primitiveType" : "INT32", "originalType" : null, "nulls" : 0, "max" : 5, "min" : 5 }, { "name" : "`c3`", "primitiveType" : "BINARY", "originalType" : "UTF8", "nulls" : 0, "max" : { "bytes" : "IHl5" }, "min" : { "bytes" : "IHl5" } } ] ● Drill has to read RG metadata during planning time ○ Get host affinity, stats etc. ● Doesn’t scale for hundreds of thousands of files ● One solution: Cache the metadata on disk by running explicit ‘REFRESH TABLE METADATA <table>’ command ● Can improve planning time by 10x
  • 33. Parquet Filter Pushdown • Applied during planning process – Based on MIN/MAX statistics of the column in Parquet Row Group – Eliminates row groups after intersecting ANDed filters • Pushdown is slightly different from partition pruning – Pushdown is applicable for all columns that have min/max statistics, not just partitioning columns • Pushdown applicable for scalar columns and complex type columns – NOTE: Filter on VARCHAR and DECIMAL types are not currently pushed down but support is being added for upcoming Drill release
  • 34. Filter Pushdown and Pruning • Complex Types Support SELECT * FROM table WHERE col_name.nested_col_int = 23 SELECT * FROM table WHERE col_name.nested_col_bln is not null SELECT * FROM table WHERE col_name.nested_col_arr[0] > 10 • Limitations: – Logical optimizations can be applied only for the columns with available statistics SELECT * FROM table WHERE col_name.nested_col_arr[2] is null – Other limitations are the same as for the Filter Pushdown for scalar data types
  • 35. Filter Pushdown and Pruning ● Transitive Closure SELECT * FROM dfs.`/tmp/first` t1 JOIN dfs.`/tmp/SECOND` t2 ON t1.`MONTH` = t2.`MONTH` WHERE t2.`MONTH` = 4 Predicate pushed down on both sides of Join More examples: SELECT * FROM t1 JOIN t2 T2 ON t1.a = t2.a WHERE t2.a NOT IN (4, 6) SELECT * FROM t1 JOIN (SELECT a, b FROM table WHERE a = 1987 AND b = 5) t2 ON t1.a = t2.a AND t1.b = t2.b
  • 36. Run-Time Filter Pushdown for HashJoin SELECT a1 FROM fact WHERE b1 IN (SELECT b2 FROM dim WHERE dim_col = ‘CA’) Hash Join (b1 = b2) Scan ‘Fact’ table Scan ‘Dim’ table Filter dim_col = ‘CA’ Hash Exchange Hash Exchange Build sideProbe side Qualified b2 values Bloom filter on b1 values Goal: Substantially reduce network transfer by this exchange
  • 37. Best Practices for Parquet Data Layout • Large number of small files or fewer large files ? – Recommended to have fewer large files subject to constraints such as Parquet block size and desired parallelism • Too many small files (order of few KBs) may cause data skew and affect performance • Too few large files reduces parallelism … need to determine sweet spot • Block size considerations – Best to have Parquet block size equal or less than HDFS/MapR-FS block size • Larger block size will spread 1 row group over 2 or more nodes incurring remote reads. • Preferred compression type – Snappy is preferred for performance, gzip for space but higher cost to decompress – Default for Drill - snappy
  • 38. Summary • Apache Drill’s extensible architecture makes it easy to support multiple data sources – While exploiting data locality and parallelism suited to the data source • Fast analytics on NoSQL databases using a new generalized framework to do index based planning and execution • Fast analytics on distributed file system tables by doing intelligent partitioning and filter pushdown
  • 39. Useful links • Drill resources – http://drill.apache.org – Twitter: @ApacheDrill – Mailing lists: • [email protected] • [email protected] • Get involved with the Drill community !
  • 40. Q & A