SlideShare a Scribd company logo
Introduction to Apache Drill
Michael Hausenblas, Chief Data Engineer EMEA, MapR
    6th Swiss Big Data User Group Meeting, Zurich, 2013-03-25

                               1
Kudos to http://cmx.io/




                              2
                          2
Workloads
• Batch processing (MapReduce)

• Light-weight OLTP (HBase, Cassandra, etc.)

• Stream processing (Storm, S4)

• Search (Solr, Elasticsearch)

• Interactive, ad-hoc query and analysis (?)



                                 3
Interactive Query at Scale



                       Impala




         low-latency
              4
Use Case I
• Jane, a marketing analyst
• Determine target segments
• Data from different sources




                       5
Use Case II
• Logistics – supplier status
• Queries
      – How many shipments from supplier X?
      – How many shipments in region Y?
                                                 {
                                                  "shipment": 100123,
SUPPLIER_ID   NAME                  REGION        "supplier": "ACM",
                                                  “timestamp": "2013-02-01",
ACM           ACME Corp             US
                                                  "description": ”first delivery today”
GAL           GotALot Inc           US           },
                                                 {
BAP           Bits and Pieces Ltd   Europe        "shipment": 100124,
                                                  "supplier": "BAP",
ZUP           Zu Pli                Asia          "timestamp": "2013-02-02",
                                                  "description": "hope you enjoy it”
                                                 }
                                             6   …
Today’s Solutions
• RDBMS-focused
   – ETL data from MongoDB and Hadoop
   – Query data using SQL

• MapReduce-focused
  – ETL from RDBMS and MongoDB
  – Use Hive, etc.




                           7
Requirements
•   Support for different data sources
•   Support for different query interfaces
•   Low-latency/real-time
•   Ad-hoc queries
•   Scalable, reliable




                          8
Google’s Dremel




http://research.google.com/pubs/pub36632.html


                                                9
Apache Drill Overview
•   Inspired by Google’s Dremel
•   Standard SQL 2003 support
•   Other QL possible
•   Plug-able data sources
•   Support for nested data
•   Schema is optional
•   Community driven, open, 100’s involved

                        10
Apache Drill Overview




          11
High-level Architecture




           12
High-level Architecture
•   Each node: Drillbit - maximize data locality
•   Co-ordination, query planning, execution, etc, are distributed
•   By default Drillbits hold all roles
•   Any node can act as endpoint for a query


      Drillbit       Drillbit         Drillbit    Drillbit



      Storage        Storage          Storage     Storage
      Process        Process          Process     Process

       node           node             node        node

                                 13
High-level Architecture
• Zookeeper for ephemeral cluster membership info
• Distributed cache (Hazelcast) for metadata, locality
  information, etc.
                                                                                     Zookeeper



       Drillbit            Drillbit              Drillbit           Drillbit
    Distributed Cache   Distributed Cache    Distributed Cache   Distributed Cache


      Storage             Storage                Storage           Storage
      Process             Process                Process           Process

        node                node                  node               node

                                            14
High-level Architecture
• Originating Drillbit acts as foreman, manages query execution,
  scheduling, locality information, etc.
• Streaming data communication avoiding SerDe
                                                                                     Zookeeper



       Drillbit            Drillbit              Drillbit           Drillbit
    Distributed Cache   Distributed Cache    Distributed Cache   Distributed Cache


      Storage             Storage                Storage           Storage
      Process             Process                Process           Process

        node                node                  node               node

                                            15
Principled Query Execution


Source                    Logical                              Physical
Query        Parser        Plan                 Optimizer       Plan       Execution




SQL 2003   parser API   query: [
                         {
                                                    topology              scanner API
DrQL                       @id: "log",
                           op: "sequence",
MongoQL                    do: [
                            {
DSL                           op: "scan",
                              source: “logs”
                            },
                            {
                              op:
                                "filter",
                              condition:
                                "x > 3”
                            },
                                               16
Drillbit Modules
                          RPC Endpoint



 SQL
                                                              Scheduler




                                                                          Storage Engine Interface
                                                                                                      DFS Engine




                                         Physical Plan
         Logical Plan




HiveQL
                        Optimizer                             Foreman

 Pig                                                                                                 HBase Engine

                                                              Operators
Mongo



Parser

                         Distributed Cache

                                                         17
Key Features
•   Full SQL 2003
•   Nested data
•   Optional schema
•   Extensibility points




                           18
Full SQL – ANSI SQL 2003
• SQL-like is often not enough
• Integration with existing tools
   – Datameer, Tableau, Excel, SAP Crystal Reports
   – Use standard ODBC/JDBC driver




                               19
Nested Data
• Nested data becoming prevalent
   – JSON/BSON, XML, ProtoBuf, Avro
   – Some data sources support it natively
     (MongoDB, etc.)
• Flattening nested data is error-prone
• Extension to ANSI SQL 2003




                                20
Optional Schema
• Many data sources don’t have rigid schemas
   – Schema changes rapidly
   – Different schema per record (e.g. HBase)
• Supports queries against unknown schema
• User can define schema or via discovery




                              21
Extensibility Points
 •   Source query – parser API
 •   Custom operators, UDF – logical plan
 •   Optimizer
 •   Data sources and formats – scanner API




Source                 Logical                Physical
Query      Parser       Plan      Optimizer    Plan      Execution




                                 22
… and Hadoop?
• HDFS can be a data source

• Complementary use cases …

• … use Apache Drill
   – Find record with specified condition
   – Aggregation under dynamic conditions

• … use MapReduce
   – Data mining with multiple iterations
   – ETL
https://cloud.google.com/files/BigQueryTechnicalWP.pdf
                                                         23
                                                    23
Example
{
 "id": "0001",
 "type": "donut",
 ”ppu": 0.55,
 "batters":
 {
                                                                             {
   "batter”:
                                                                                  "sales" : 700.0,
   [
                                                                                  "typeCount" : 1,
       { "id": "1001", "type": "Regular" },
                                                                                  "quantity" : 700,
       { "id": "1002", "type": "Chocolate" },
                                                                                  "ppu" : 1.0
…
                                                                             }
                                                                              {
                                                                                  "sales" : 109.71,
data source: donuts.json                                                          "typeCount" : 2,
                                                                                  "quantity" : 159,
 query:[ {                                                                        "ppu" : 0.69
      op:"sequence",                                                         }
      do:[                                                                    {
           {                                                                      "sales" : 184.25,
              op: "scan",                                                         "typeCount" : 2,
              ref: "donuts",                                                      "quantity" : 335,
              source: "local-logs",                                               "ppu" : 0.55
              selection: {data: "activity"}                                  }
           },
           {                                                                result: out.json
              op: "filter",
              expr: "donuts.ppu < 2.00"
           },
…

logical plan: simple_plan.json                  https://cwiki.apache.org/confluence/display/DRILL/Demo+HowTo

                                                      24
Status
• Heavy development by multiple organizations

• Available
  – Logical plan (ADSP)
  – Reference interpreter
  – Basic SQL parser
  – Basic demo
  – Basic HBase back-end

                            25
Status
March/April

•   Larger SQL syntax
•   Physical plan
•   In-memory compressed data interfaces
•   Distributed execution focused on large cluster
    high performance sort, aggregation and join


                          26
Contributing
• Dremel-inspired columnar format: Twitter’s Parquet and
  Hive’s ORC file

• Integration with Hive metastore (?)

• DRILL-13 Storage Engine: Define Java Interface

• DRILL-15 Build HBase storage engine implementation




                               27
Contributing
• DRILL-48 RPC interface for query submission and physical plan
  execution

• DRILL-53 Setup cluster configuration and membership mgmt
  system
   – ZK for coordination
   – Helix for partition and resource assignment (?)

• Further schedule
   – Alpha Q2
   – Beta Q3
                               28
Kudos to …
•   Julian Hyde, Pentaho
•   Timothy Chen, Microsoft
•   Chris Merrick, RJMetrics
•   David Alves, UT Austin
•   Sree Vaadi, SSS/NGData
•   Jacques Nadeau, MapR
•   Ted Dunning, MapR

                         29
Engage!
• Follow @ApacheDrill on Twitter

• Sign up at mailing lists (user | dev)
  http://incubator.apache.org/drill/mailing-lists.html


• Learn where and how to contribute
  https://cwiki.apache.org/confluence/display/DRILL/Contributing


• Keep an eye on http://drill-user.org/




                                     30

More Related Content

What's hot (20)

PDF
Apache Drill @ PJUG, Jan 15, 2013
Gera Shegalov
 
PDF
Hadoop User Group - Status Apache Drill
MapR Technologies
 
PPTX
Apache drill
Jakub Pieprzyk
 
PPTX
Free Code Friday: Drill 101 - Basics of Apache Drill
MapR Technologies
 
PPTX
Drilling into Data with Apache Drill
MapR Technologies
 
KEY
Building a Business on Hadoop, HBase, and Open Source Distributed Computing
Bradford Stephens
 
PPTX
Spark SQL versus Apache Drill: Different Tools with Different Rules
DataWorks Summit/Hadoop Summit
 
PPTX
Rethinking SQL for Big Data with Apache Drill
MapR Technologies
 
PPTX
Hadoop And Their Ecosystem
sunera pathan
 
PDF
Apache Drill - Why, What, How
mcsrivas
 
PDF
May 2013 HUG: HCatalog/Hive Data Out
Yahoo Developer Network
 
PDF
Apache Drill: Building Highly Flexible, High Performance Query Engines by M.C...
The Hive
 
PPTX
SQL-on-Hadoop with Apache Drill
MapR Technologies
 
PDF
Scaling HDFS to Manage Billions of Files with Key-Value Stores
DataWorks Summit
 
PDF
Apache Hadoop and HBase
Cloudera, Inc.
 
PDF
Hadoop Summit San Jose 2014: Costing Your Big Data Operations
Sumeet Singh
 
PPTX
Hadoop configuration & performance tuning
Vitthal Gogate
 
PDF
Hadoop Overview & Architecture
EMC
 
PDF
Integration of HIve and HBase
Hortonworks
 
PPTX
HBaseCon 2015: Analyzing HBase Data with Apache Hive
HBaseCon
 
Apache Drill @ PJUG, Jan 15, 2013
Gera Shegalov
 
Hadoop User Group - Status Apache Drill
MapR Technologies
 
Apache drill
Jakub Pieprzyk
 
Free Code Friday: Drill 101 - Basics of Apache Drill
MapR Technologies
 
Drilling into Data with Apache Drill
MapR Technologies
 
Building a Business on Hadoop, HBase, and Open Source Distributed Computing
Bradford Stephens
 
Spark SQL versus Apache Drill: Different Tools with Different Rules
DataWorks Summit/Hadoop Summit
 
Rethinking SQL for Big Data with Apache Drill
MapR Technologies
 
Hadoop And Their Ecosystem
sunera pathan
 
Apache Drill - Why, What, How
mcsrivas
 
May 2013 HUG: HCatalog/Hive Data Out
Yahoo Developer Network
 
Apache Drill: Building Highly Flexible, High Performance Query Engines by M.C...
The Hive
 
SQL-on-Hadoop with Apache Drill
MapR Technologies
 
Scaling HDFS to Manage Billions of Files with Key-Value Stores
DataWorks Summit
 
Apache Hadoop and HBase
Cloudera, Inc.
 
Hadoop Summit San Jose 2014: Costing Your Big Data Operations
Sumeet Singh
 
Hadoop configuration & performance tuning
Vitthal Gogate
 
Hadoop Overview & Architecture
EMC
 
Integration of HIve and HBase
Hortonworks
 
HBaseCon 2015: Analyzing HBase Data with Apache Hive
HBaseCon
 

Viewers also liked (11)

PDF
Presto @ Facebook: Past, Present and Future
DataWorks Summit
 
PPTX
Presto overview
Shixiong Zhu
 
PDF
Understanding Presto - Presto meetup @ Tokyo #1
Sadayuki Furuhashi
 
PDF
Presto Meetup @ Facebook (3/22/2016)
Martin Traverso
 
PDF
Presto as a Service - Tips for operation and monitoring
Taro L. Saito
 
PDF
Presto anatomy
Dongmin Yu
 
PDF
Presto in my_use_case
wyukawa
 
PPTX
Presto: Distributed sql query engine
kiran palaka
 
PDF
Presto at Hadoop Summit 2016
kbajda
 
PDF
Zeromq anatomy & jeromq
Dongmin Yu
 
PDF
Giraph+Gora in ApacheCon14
Renato Javier Marroquín Mogrovejo
 
Presto @ Facebook: Past, Present and Future
DataWorks Summit
 
Presto overview
Shixiong Zhu
 
Understanding Presto - Presto meetup @ Tokyo #1
Sadayuki Furuhashi
 
Presto Meetup @ Facebook (3/22/2016)
Martin Traverso
 
Presto as a Service - Tips for operation and monitoring
Taro L. Saito
 
Presto anatomy
Dongmin Yu
 
Presto in my_use_case
wyukawa
 
Presto: Distributed sql query engine
kiran palaka
 
Presto at Hadoop Summit 2016
kbajda
 
Zeromq anatomy & jeromq
Dongmin Yu
 
Giraph+Gora in ApacheCon14
Renato Javier Marroquín Mogrovejo
 
Ad

Similar to Introduction to Apache Drill (20)

PPTX
Hadoop Summit - Hausenblas 20 March
MapR Technologies
 
PDF
Swiss Big Data User Group - Introduction to Apache Drill
MapR Technologies
 
PPTX
Drill njhug -19 feb2013
MapR Technologies
 
PPT
Wmware NoSQL
Murat Çakal
 
PDF
Using Spring with NoSQL databases (SpringOne China 2012)
Chris Richardson
 
PDF
Hadoop, Taming Elephants
Ovidiu Dimulescu
 
PDF
Introduction to Apache Geode (Cork, Ireland)
Anthony Baker
 
PPTX
PhillyDB Talk - Beyond Batch
boorad
 
PPTX
Large scale computing with mapreduce
hansen3032
 
PPTX
Hadoop ppt on the basics and architecture
saipriyacoool
 
PDF
Slides for the Apache Geode Hands-on Meetup and Hackathon Announcement
VMware Tanzu
 
PDF
Apache Geode Meetup, London
Apache Geode
 
PPTX
001 hbase introduction
Scott Miao
 
PPTX
Hadoop World 2011: Hadoop and RDBMS with Sqoop and Other Tools - Guy Harrison...
Cloudera, Inc.
 
ODP
Kerry osborne hadoop meets exadata
Enkitec
 
PPTX
Drill dchug-29 nov2012
MapR Technologies
 
KEY
DevNation Atlanta
boorad
 
PDF
DConf2015 - Using D for Development of Large Scale Primary Storage
Liran Zvibel
 
PPTX
HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
Cloudera, Inc.
 
PDF
Apache Geode Meetup, Cork, Ireland at CIT
Apache Geode
 
Hadoop Summit - Hausenblas 20 March
MapR Technologies
 
Swiss Big Data User Group - Introduction to Apache Drill
MapR Technologies
 
Drill njhug -19 feb2013
MapR Technologies
 
Wmware NoSQL
Murat Çakal
 
Using Spring with NoSQL databases (SpringOne China 2012)
Chris Richardson
 
Hadoop, Taming Elephants
Ovidiu Dimulescu
 
Introduction to Apache Geode (Cork, Ireland)
Anthony Baker
 
PhillyDB Talk - Beyond Batch
boorad
 
Large scale computing with mapreduce
hansen3032
 
Hadoop ppt on the basics and architecture
saipriyacoool
 
Slides for the Apache Geode Hands-on Meetup and Hackathon Announcement
VMware Tanzu
 
Apache Geode Meetup, London
Apache Geode
 
001 hbase introduction
Scott Miao
 
Hadoop World 2011: Hadoop and RDBMS with Sqoop and Other Tools - Guy Harrison...
Cloudera, Inc.
 
Kerry osborne hadoop meets exadata
Enkitec
 
Drill dchug-29 nov2012
MapR Technologies
 
DevNation Atlanta
boorad
 
DConf2015 - Using D for Development of Large Scale Primary Storage
Liran Zvibel
 
HBaseCon 2012 | Building a Large Search Platform on a Shoestring Budget
Cloudera, Inc.
 
Apache Geode Meetup, Cork, Ireland at CIT
Apache Geode
 
Ad

More from Swiss Big Data User Group (20)

PDF
Making Hadoop based analytics simple for everyone to use
Swiss Big Data User Group
 
PDF
A real life project using Cassandra at a large Swiss Telco operator
Swiss Big Data User Group
 
PDF
Data Analytics – B2B vs. B2C
Swiss Big Data User Group
 
PDF
SQL on Hadoop
Swiss Big Data User Group
 
PDF
Building a Hadoop Data Warehouse with Impala
Swiss Big Data User Group
 
PDF
Closing The Loop for Evaluating Big Data Analysis
Swiss Big Data User Group
 
PDF
Big Data and Data Science for traditional Swiss companies
Swiss Big Data User Group
 
PPTX
Design Patterns for Large-Scale Real-Time Learning
Swiss Big Data User Group
 
PDF
Educating Data Scientists of the Future
Swiss Big Data User Group
 
PDF
Unleash the power of Big Data in your existing Data Warehouse
Swiss Big Data User Group
 
PDF
Big data for Telco: opportunity or threat?
Swiss Big Data User Group
 
PDF
Project "Babelfish" - A data warehouse to attack complexity
Swiss Big Data User Group
 
PDF
Brainserve Datacenter: the High-Density Choice
Swiss Big Data User Group
 
PDF
Urturn on AWS: scaling infra, cost and time to maket
Swiss Big Data User Group
 
PDF
The World Wide Distributed Computing Architecture of the LHC Datagrid
Swiss Big Data User Group
 
PPTX
New opportunities for connected data : Neo4j the graph database
Swiss Big Data User Group
 
PDF
Technology Outlook - The new Era of computing
Swiss Big Data User Group
 
PDF
In-Store Analysis with Hadoop
Swiss Big Data User Group
 
PDF
Big Data Visualization With ParaView
Swiss Big Data User Group
 
PPTX
Oracle's BigData solutions
Swiss Big Data User Group
 
Making Hadoop based analytics simple for everyone to use
Swiss Big Data User Group
 
A real life project using Cassandra at a large Swiss Telco operator
Swiss Big Data User Group
 
Data Analytics – B2B vs. B2C
Swiss Big Data User Group
 
Building a Hadoop Data Warehouse with Impala
Swiss Big Data User Group
 
Closing The Loop for Evaluating Big Data Analysis
Swiss Big Data User Group
 
Big Data and Data Science for traditional Swiss companies
Swiss Big Data User Group
 
Design Patterns for Large-Scale Real-Time Learning
Swiss Big Data User Group
 
Educating Data Scientists of the Future
Swiss Big Data User Group
 
Unleash the power of Big Data in your existing Data Warehouse
Swiss Big Data User Group
 
Big data for Telco: opportunity or threat?
Swiss Big Data User Group
 
Project "Babelfish" - A data warehouse to attack complexity
Swiss Big Data User Group
 
Brainserve Datacenter: the High-Density Choice
Swiss Big Data User Group
 
Urturn on AWS: scaling infra, cost and time to maket
Swiss Big Data User Group
 
The World Wide Distributed Computing Architecture of the LHC Datagrid
Swiss Big Data User Group
 
New opportunities for connected data : Neo4j the graph database
Swiss Big Data User Group
 
Technology Outlook - The new Era of computing
Swiss Big Data User Group
 
In-Store Analysis with Hadoop
Swiss Big Data User Group
 
Big Data Visualization With ParaView
Swiss Big Data User Group
 
Oracle's BigData solutions
Swiss Big Data User Group
 

Recently uploaded (20)

PDF
Jak MŚP w Europie Środkowo-Wschodniej odnajdują się w świecie AI
dominikamizerska1
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PPTX
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PDF
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
DOCX
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
PDF
IoT-Powered Industrial Transformation – Smart Manufacturing to Connected Heal...
Rejig Digital
 
PDF
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
PDF
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
PDF
Advancing WebDriver BiDi support in WebKit
Igalia
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PDF
What Makes Contify’s News API Stand Out: Key Features at a Glance
Contify
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
Jak MŚP w Europie Środkowo-Wschodniej odnajdują się w świecie AI
dominikamizerska1
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
IoT-Powered Industrial Transformation – Smart Manufacturing to Connected Heal...
Rejig Digital
 
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
Advancing WebDriver BiDi support in WebKit
Igalia
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
What Makes Contify’s News API Stand Out: Key Features at a Glance
Contify
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 

Introduction to Apache Drill

  • 1. Introduction to Apache Drill Michael Hausenblas, Chief Data Engineer EMEA, MapR 6th Swiss Big Data User Group Meeting, Zurich, 2013-03-25 1
  • 3. Workloads • Batch processing (MapReduce) • Light-weight OLTP (HBase, Cassandra, etc.) • Stream processing (Storm, S4) • Search (Solr, Elasticsearch) • Interactive, ad-hoc query and analysis (?) 3
  • 4. Interactive Query at Scale Impala low-latency 4
  • 5. Use Case I • Jane, a marketing analyst • Determine target segments • Data from different sources 5
  • 6. Use Case II • Logistics – supplier status • Queries – How many shipments from supplier X? – How many shipments in region Y? { "shipment": 100123, SUPPLIER_ID NAME REGION "supplier": "ACM", “timestamp": "2013-02-01", ACM ACME Corp US "description": ”first delivery today” GAL GotALot Inc US }, { BAP Bits and Pieces Ltd Europe "shipment": 100124, "supplier": "BAP", ZUP Zu Pli Asia "timestamp": "2013-02-02", "description": "hope you enjoy it” } 6 …
  • 7. Today’s Solutions • RDBMS-focused – ETL data from MongoDB and Hadoop – Query data using SQL • MapReduce-focused – ETL from RDBMS and MongoDB – Use Hive, etc. 7
  • 8. Requirements • Support for different data sources • Support for different query interfaces • Low-latency/real-time • Ad-hoc queries • Scalable, reliable 8
  • 10. Apache Drill Overview • Inspired by Google’s Dremel • Standard SQL 2003 support • Other QL possible • Plug-able data sources • Support for nested data • Schema is optional • Community driven, open, 100’s involved 10
  • 13. High-level Architecture • Each node: Drillbit - maximize data locality • Co-ordination, query planning, execution, etc, are distributed • By default Drillbits hold all roles • Any node can act as endpoint for a query Drillbit Drillbit Drillbit Drillbit Storage Storage Storage Storage Process Process Process Process node node node node 13
  • 14. High-level Architecture • Zookeeper for ephemeral cluster membership info • Distributed cache (Hazelcast) for metadata, locality information, etc. Zookeeper Drillbit Drillbit Drillbit Drillbit Distributed Cache Distributed Cache Distributed Cache Distributed Cache Storage Storage Storage Storage Process Process Process Process node node node node 14
  • 15. High-level Architecture • Originating Drillbit acts as foreman, manages query execution, scheduling, locality information, etc. • Streaming data communication avoiding SerDe Zookeeper Drillbit Drillbit Drillbit Drillbit Distributed Cache Distributed Cache Distributed Cache Distributed Cache Storage Storage Storage Storage Process Process Process Process node node node node 15
  • 16. Principled Query Execution Source Logical Physical Query Parser Plan Optimizer Plan Execution SQL 2003 parser API query: [ { topology scanner API DrQL @id: "log", op: "sequence", MongoQL do: [ { DSL op: "scan", source: “logs” }, { op: "filter", condition: "x > 3” }, 16
  • 17. Drillbit Modules RPC Endpoint SQL Scheduler Storage Engine Interface DFS Engine Physical Plan Logical Plan HiveQL Optimizer Foreman Pig HBase Engine Operators Mongo Parser Distributed Cache 17
  • 18. Key Features • Full SQL 2003 • Nested data • Optional schema • Extensibility points 18
  • 19. Full SQL – ANSI SQL 2003 • SQL-like is often not enough • Integration with existing tools – Datameer, Tableau, Excel, SAP Crystal Reports – Use standard ODBC/JDBC driver 19
  • 20. Nested Data • Nested data becoming prevalent – JSON/BSON, XML, ProtoBuf, Avro – Some data sources support it natively (MongoDB, etc.) • Flattening nested data is error-prone • Extension to ANSI SQL 2003 20
  • 21. Optional Schema • Many data sources don’t have rigid schemas – Schema changes rapidly – Different schema per record (e.g. HBase) • Supports queries against unknown schema • User can define schema or via discovery 21
  • 22. Extensibility Points • Source query – parser API • Custom operators, UDF – logical plan • Optimizer • Data sources and formats – scanner API Source Logical Physical Query Parser Plan Optimizer Plan Execution 22
  • 23. … and Hadoop? • HDFS can be a data source • Complementary use cases … • … use Apache Drill – Find record with specified condition – Aggregation under dynamic conditions • … use MapReduce – Data mining with multiple iterations – ETL https://cloud.google.com/files/BigQueryTechnicalWP.pdf 23 23
  • 24. Example { "id": "0001", "type": "donut", ”ppu": 0.55, "batters": { { "batter”: "sales" : 700.0, [ "typeCount" : 1, { "id": "1001", "type": "Regular" }, "quantity" : 700, { "id": "1002", "type": "Chocolate" }, "ppu" : 1.0 … } { "sales" : 109.71, data source: donuts.json "typeCount" : 2, "quantity" : 159, query:[ { "ppu" : 0.69 op:"sequence", } do:[ { { "sales" : 184.25, op: "scan", "typeCount" : 2, ref: "donuts", "quantity" : 335, source: "local-logs", "ppu" : 0.55 selection: {data: "activity"} } }, { result: out.json op: "filter", expr: "donuts.ppu < 2.00" }, … logical plan: simple_plan.json https://cwiki.apache.org/confluence/display/DRILL/Demo+HowTo 24
  • 25. Status • Heavy development by multiple organizations • Available – Logical plan (ADSP) – Reference interpreter – Basic SQL parser – Basic demo – Basic HBase back-end 25
  • 26. Status March/April • Larger SQL syntax • Physical plan • In-memory compressed data interfaces • Distributed execution focused on large cluster high performance sort, aggregation and join 26
  • 27. Contributing • Dremel-inspired columnar format: Twitter’s Parquet and Hive’s ORC file • Integration with Hive metastore (?) • DRILL-13 Storage Engine: Define Java Interface • DRILL-15 Build HBase storage engine implementation 27
  • 28. Contributing • DRILL-48 RPC interface for query submission and physical plan execution • DRILL-53 Setup cluster configuration and membership mgmt system – ZK for coordination – Helix for partition and resource assignment (?) • Further schedule – Alpha Q2 – Beta Q3 28
  • 29. Kudos to … • Julian Hyde, Pentaho • Timothy Chen, Microsoft • Chris Merrick, RJMetrics • David Alves, UT Austin • Sree Vaadi, SSS/NGData • Jacques Nadeau, MapR • Ted Dunning, MapR 29
  • 30. Engage! • Follow @ApacheDrill on Twitter • Sign up at mailing lists (user | dev) http://incubator.apache.org/drill/mailing-lists.html • Learn where and how to contribute https://cwiki.apache.org/confluence/display/DRILL/Contributing • Keep an eye on http://drill-user.org/ 30

Editor's Notes

  • #5: Hive: compile to MR, Aster: external tables in MPP, Oracle/MySQL: export MR results to RDBMSDrill, Impala, CitusDB: real-time
  • #6: Suppose a marketing analyst trying to experiment with ways to do targeting of user segments for next campaign. Needs access to web logs stored in Hadoop, and also needs to access user profiles stored in MongoDB as well as access to transaction data stored in a conventional database.
  • #9: Re ad-hoc:You might not know ahead of time what queries you will want to make. You may need to react to changing circumstances.
  • #10: Two innovations: handle nested-data column style (column-striped representation) and query push-down
  • #14: Drillbits per node, maximize data localityCo-ordination, query planning, optimization, scheduling, execution are distributedBy default, Drillbits hold all roles, modules can optionally be disabled.Any node/Drillbit can act as endpoint for particular query.
  • #15: Zookeeper maintains ephemeral cluster membership information onlySmall distributed cache utilizing embedded Hazelcast maintains information about individual queue depth, cached query plans, metadata, locality information, etc.
  • #16: Originating Drillbit acts as foreman, manages all execution for their particular query, scheduling based on priority, queue depth and locality information.Drillbit data communication is streaming and avoids any serialization/deserializationRed arrow: originating drillbit, is the root of the multi-level serving tree, per query
  • #17: Source query - Human (eg DSL) or tool written(eg SQL/ANSI compliant) query Source query is parsed and transformed to produce the logical planLogical plan: dataflow of what should logically be doneTypically, the logical plan lives in memory in the form of Java objects, but also has a textual formThe logical query is then transformed and optimized into the physical plan.Optimizer introduces of parallel computation, taking topology into accountOptimizer handles columnar data to improve processing speedThe physical plan represents the actual structure of computation as it is done by the systemHow physical and exchange operators should be appliedAssignment to particular nodes and cores + actual query execution per node
  • #24: Relation of Drill to HadoopHadoop = HDFS + MapReduceDrill for:Finding particular records with specified conditions. For example, to findrequest logs with specified account ID.Quick aggregation of statistics with dynamically-changing conditions. For example, getting a summary of request traffic volume from the previous night for a web application and draw a graph from it.Trial-and-error data analysis. For example, identifying the cause of trouble and aggregating values by various conditions, including by hour, day and etc...MapReduce: Executing a complex data mining on Big Data which requires multiple iterations and paths of data processing with programmed algorithms.Executing large join operations across huge datasets.Exporting large amount of data after processing.