An introduction to apache drill presentation
6 likes2,712 views
The document provides an introduction to Apache Drill, an open source SQL query engine for analysis of large-scale datasets across Hadoop, NoSQL and cloud storage systems. It discusses Tomer Shiran's role in Apache Drill, provides an agenda for the talk, describes the need for interactive analysis of big data and how existing solutions are limited. It then outlines Apache Drill's architecture, key features like full SQL support, optional schemas and support for nested data formats.
![Nested
Data
JSON
• Nested
data
is
becoming
prevalent
{
"name":
"Homer",
– JSON,
BSON,
XML,
Protocol
Buffers,
Avro,
…
"gender":
"Male",
"followers":
100
– The
data
source
may
or
may
not
be
aware
children:
[
• MongoDB
supports
nested
data
na)vely
{name:
"Bart"},
{name:
"Lisa”}
• A
single
HBase
value
could
be
a
JSON
document
]
(compound
nested
type)
}
– Google
Dremel’s
innova)on
was
efficient
columnar
storage
and
querying
of
nested
data
• FlaGening
nested
data
is
error-‐prone
and
ocen
Avro
enum
Gender
{
impossible
MALE,
FEMALE
– Think
about
repeated
and
op)onal
fields
at
every
}
level…
record
User
{
• Apache
Drill
supports
nested
data
string
name;
Gender
gender;
– Extensions
to
ANSI
SQL:2003
long
followers;
}](https://image.slidesharecdn.com/anintroductiontoapachedrillpresentation-130305111328-phpapp01/85/An-introduction-to-apache-drill-presentation-13-320.jpg)
An introduction to apache drill presentation
- 1. Introduc)on to Apache Drill February 27, 2013 Tomer Shiran
- 2. Who Am I? • Tomer Shiran • [email protected] • Founding Member and CommiGer, Apache Drill • Director of Product Management, MapR Technologies
- 3. Agenda • Apache Drill overview • Key features • Status and progress • How to get involved
- 4. Big Data Workloads • ETL • Data mining • Index and model genera)on • Clustering, anomaly detec)on and classifica)on • Blob store • Lightweight OLTP on large datasets • Web crawling • Stream processing • Interac)ve analysis
- 5. Interac)ve Queries and Hadoop Exis)ng Solu)ons Compile SQL (HiveQL) to Export MapReduce results to MapReduce RDBMS and query RDBMS Emerging Technologies Stinger/Tez Impala Real-‐)me PostgreSQL-‐based PostgreSQL-‐based Hive performance Real-‐)me interac)ve analysis Hadoop analy)cs Hadoop analy)cs improvements interac)ve analysis HAWQ Phoenix Cascading Lingual PostgreSQL-‐based Compile ANSI SQL to Hadoop analy)cs SQL layer for HBase SQL layer for HBase MapReduce
- 6. Example Problem • Jane works as an Transac.on analyst at an e-‐ informa.on commerce company • How does she figure User out good targe)ng profiles segments for the next marke)ng campaign? • She has some ideas Access and lots of data logs
- 7. Solving the Problem with Tradi)onal Systems • Use an RDBMS – ETL the data from MongoDB and Hadoop into the RDBMS • MongoDB data must be flaGened, schema)zed, filtered and aggregated • Hadoop data must be filtered and aggregated – Query the data using any SQL-‐based tool • Use MapReduce – ETL the data from Oracle and MongoDB into Hadoop – Work with the MapReduce team to generate the desired analyses • Use Hive – ETL the data from Oracle and MongoDB into Hadoop • MongoDB data must be flaGened and schema)zed – But HiveQL is limited, queries take too long and BI tool support is limited
- 8. WWGD Distributed Interac.ve Batch NoSQL File System analysis processing GFS BigTable Dremel MapReduce Hadoop HDFS HBase ??? MapReduce Build Apache Drill to provide a true open source solu)on to interac)ve analysis of Big Data
- 9. Apache Drill Overview • Interac)ve analysis of Big Data using standard SQL • Fast – Low latency – Columnar execu)on • Inspired by Google Dremel/BigQuery Interac)ve queries – Complement na)ve interfaces and MapReduce/ Apache Drill Data analyst Hive/Pig Repor)ng 100 ms-‐20 min • Open – Community driven open source project – Under Apache Socware Founda)on • Modern – Standard ANSI SQL:2003 (select/into) Data mining – Nested/hierarchical data support MapReduce Modeling Hive – Schema is op)onal Pig Large ETL 20 min-‐20 hr – Supports RDBMS, Hadoop and NoSQL
- 10. How Does It Work? • Drillbits run on each node, designed to maximize data locality • Processing is done outside MapReduce SELECT * FROM paradigm (YARN is supported) oracle.transac)ons, • Queries can be fed to any Drillbit mongo.users, hdfs.events • Coordina)on, query planning, op)miza)on, LIMIT 1 scheduling, and execu)on are distributed
- 11. Key Features • Full SQL (ANSI SQL:2003) • Nested data • Schema is op)onal • Flexible and extensible architecture
- 12. Full SQL (ANSI SQL:2003) • Drill supports standard ANSI SQL:2003 – Correlated subqueries, analy)c func)ons, … – SQL-‐like is not enough • Use any SQL-‐based tool with Apache Drill – Tableau, Microstrategy, Excel, SAP Crystal Reports, Toad, SQuirreL, … – Standard ODBC and JDBC drivers Client Tableau Drillbit MicroStrategy Drill%ODBC% SQL%Query% Query% Driver Drillbits Driver Parser Planner Drill%Worker Drill%Worker Excel SAP%Crystal% Reports
- 13. Nested Data JSON • Nested data is becoming prevalent { "name": "Homer", – JSON, BSON, XML, Protocol Buffers, Avro, … "gender": "Male", "followers": 100 – The data source may or may not be aware children: [ • MongoDB supports nested data na)vely {name: "Bart"}, {name: "Lisa”} • A single HBase value could be a JSON document ] (compound nested type) } – Google Dremel’s innova)on was efficient columnar storage and querying of nested data • FlaGening nested data is error-‐prone and ocen Avro enum Gender { impossible MALE, FEMALE – Think about repeated and op)onal fields at every } level… record User { • Apache Drill supports nested data string name; Gender gender; – Extensions to ANSI SQL:2003 long followers; }
- 14. Schema is Op)onal • Many data sources do not have rigid schemas – Schemas change rapidly – Each record may have a different schema • Sparse and wide rows in HBase and Cassandra, MongoDB • Apache Drill supports querying against unknown schemas – Query any HBase, Cassandra or MongoDB table • User can define the schema or let the system discover it automa)cally – System of record may already have schema informa)on • Why manage it in a separate system? – No need to manage schema evolu)on Row Key CF contents CF anchor "com.cnn.www" contents:html = "<html>…" anchor:my.look.ca = "CNN.com" anchor:cnnsi.com = "CNN" "com.foxnews.www" contents:html = "<html>…" anchor:en.wikipedia.org = "Fox News" … … …
- 15. Flexible and Extensible Architecture • Apache Drill is designed for extensibility – Well-‐documented APIs and interfaces • Data sources and file formats – Implement a custom scanner to support a new data source or file format • Query languages – SQL:2003 is the primary language – Implement a custom Parser to support a Domain Specific Language – UDFs and UDTFs • Op)mizers – Drill will have a cost-‐based op)mizer – Clear surrounding APIs support easy op)mizer explora)on • Operators – Custom operators can be implemented • Special operators for Mahout (k-‐means) being designed – Operator push-‐down to data source (RDBMS)
- 16. What About Other SQL-‐on-‐Hadoop Systems? • Strengths – Code is more mature than Apache Drill • Already in beta (~2 quarters ahead) – Faster than Hive on some queries • Weaknesses – Proprietary or semi-‐open source – Query results must fit in memory (no spooling) – Early row materializa)on (no columnar execu)on) – Some are SQL-‐like (not SQL) – No support for nested data – Rigid schema is required – Limited flexibility and extensibility – Only support Hadoop and HBase (and no other NoSQL or RDBMS)
- 17. Status: In Progress • Heavy ac)ve development – 6-‐7 companies are contribu)ng • Available – Logical plan syntax and interpreter – Reference execu)on engine • In progress – SQL interpreter – Storage engine implementa)ons for Accumulo, Cassandra, HBase and various file formats • Significant community momentum – Over 250 people on the Drill mailing list – Over 250 members of the Bay Area Drill User Group – Drill meetups across the US and Europe – OpenDremel team joined Apache Drill • An)cipated schedule: – Prototype: Q1 – Alpha: Q2 – Beta: Q3
- 18. Why Apache Drill Will Be Successful Resources Community Architecture • Contributors have strong • Development done in the • Full SQL backgrounds from open • New data support companies like Oracle, • Ac)ve contributors from • Extensible APIs IBM Netezza, Informa)ca, mul)ple companies • Full Columnar Execu)on Clustrix and Pentaho • Rapidly growing • Beyond Hadoop
- 19. Interested in Apache Drill? • Many op)ons to contribute – Become a full )me Drill engineer @ MapR • Email [email protected] – Join the Drill mailing list and start contribu)ng • JIRAs, code, unit tests, documenta)on, … – Shoot me an email and we can discuss • Email [email protected]
- 20. QUESTIONS?
- 21. Why Not Leverage MapReduce? • Scheduling Model – Coarse resource model reduces hardware u)liza)on – Acquisi)on of resources typically takes 100’s of millis to seconds • Barriers – Map comple)on required before shuffle/reduce commencement – All maps must complete before reduce can start – In chained jobs, one job must finish en)rely before the next one can start • Persistence and Recoverability – Data is persisted to disk between each barrier – Serializa)on and deserializa)on are required between execu)on phase