Introduction to Hive and HCatalog
- 1. 1 Introduction to Apache Hive (and HCatalog) Mark Grover github.com/markgrover/nyc-hug- hive
- 2. Me! • Contributor to Apache Hive • Sec3on Author of O’Reilly’s Programming Hive book • So?ware Developer at Cloudera • @mark_grover • [email protected] • hFps://github.com/markgrover/nyc-‐hug-‐hive
- 3. Agenda • What is Hive? • Why use Hive? • Hive features • Hive architecture • HCatalog • Demo!
- 4. Preamble • This is a remote talk • Feel free to ask ques3ons any 3me!
- 5. Agenda • What is Hive? • Why use Hive? • Hive features • Hive architecture • HCatalog • Demo!
- 6. Hive • Data warehouse system for Hadoop • Enables Extract/Transform/Load (ETL) • Associate structure with a variety of data formats • Logical Table -‐> Physical Loca3on • Logical Table -‐> Physical Data Format Handler (SerDe) • Integrates with HDFS, HBase, MongoDB, etc. • Query execu3on in MapReduce
- 7. Agenda • What is Hive? • Why use Hive? • Hive features • Hive architecture • HCatalog • Demo!
- 8. Why use Hive? • MapReduce is catered towards developers • Run SQL-‐like queries that get compiled and run as MapReduce jobs • Data in Hadoop even though generally unstructured has some vague structure associated with it • Benefits of MapReduce + HDFS (Hadoop) • Fault tolerant • Robust • Scalable
- 9. Agenda • What is Hive? • Why use Hive? • Hive features • Hive architecture • HCatalog • Demo!
- 10. Hive features • Create table, create view, create index -‐ DDL • Select, where clause, group by, order by, joins • Pluggable User Defined Func3ons -‐ UDFs (e.g from_unix3me) • Pluggable User Defined Aggregate Func3ons -‐ UDAFs (e.g. count, avg) • Pluggable User Defined Table Genera3ng Func3ons -‐ UDTFs (e.g. explode)
- 11. Hive features • Pluggable custom Input/Output format • Pluggable Serializa3on Deserializa3on libraries (SerDes) • Pluggable custom map and reduce scripts
- 12. What Hive does NOT support • OLTP workloads -‐ low latency • Correlated subqueries • Not super performant with small amounts of data • How much data do you need to call it “Big Data”?
- 13. Other Hive features • Par33oning • Sampling • Bucke3ng • Various join op3miza3ons • Integra3on with HBase and other storage handlers • Views – Unmaterialized • Complex data types – arrays, structs, maps
- 14. Connecting to Hive • Hive Shell • JDBC driver • ODBC driver • Thri? client
- 15. Agenda • What is Hive? • Why use Hive? • Hive features • Hive architecture • HCatalog • Demo!
- 16. Hive metastore • Backed by RDBMS • Derby, MySQL, PostgreSQL, etc. supported • Default Embedded Derby • Not recommend for anything but a quick Proof of Concept • 3 different modes of opera3on: • Embedded Derby (default) • Local • Remote
- 18. Hive Remote Mode
- 19. Hive server
- 20. Problems with Hive Server • No sessions/concurrency • Essen3ally need 1 server per client • Security • Auding/Logging
- 21. Hive server 2
- 22. Hive architecture • Compiler • Parser • Type checking • Seman3c Analyzer • Plan Genera3on • Task Genera3on
- 23. Hive architecture • Execu3on Engine • Plan • Operators • SerDes • UDFs/UDAFs/UDTFs • Metastore • Stores schema of data • HCatalog
- 24. Architecture Summary • Use remote metastore service for sharing the metastore with HCatalog and other tools • Use Hive Server2 for concurrent queries
- 25. Agenda • What is Hive? • Why use Hive? • Hive features • Hive architecture • HCatalog • Demo!
- 26. Hive Metastore Remote Mode
- 27. HCatalog • Sub-‐component of Hive • Table and storage management service • Public APIs and webservice wrappers for accessing metadata in Hive metastore • Metastore contains informa3on of interest to other tools (Pig, MapReduce jobs) • Expose that informa3on as REST interface • WebHCat: Web Server for engaging with the Hive metastore
- 29. Agenda • What is Hive? • Why use Hive? • Hive features • Hive architecture • HCatalog • Demo!
- 30. Applications of Hive • Web Analy3cs • Retail • Healthcare • Spam detec3on • Data Mining • Ad op3miza3on • ETL workloads
- 31. How to install Hive • Download Apache Hive tarball • Use Apache Bigtop packages • Use a Demo VM
- 32. Want to learn more about Hive?
- 33. Contact info @mark_grover github.com/markgrover linkedin.com/in/grovermark [email protected]