Things Every Oracle DBA Needs to Know about the Hadoop Ecosystem
9 likes4,993 views
The document discusses the challenges and solutions associated with big data, focusing on the Hadoop ecosystem. It details components like HDFS, MapReduce, and YARN, and highlights the roles of database administrators (DBAs) in transitioning to big data technologies. The importance of adaptation and gaining programming skills in languages like Java and Python for DBAs is emphasized as the landscape of data management evolves.
Things Every Oracle DBA Needs to Know about the Hadoop Ecosystem
- 1. Zohar Elkayam www.realdbamagic.com Twitter: @realmgic Things Every Oracle DBA Needs to Know about the Hadoop Ecosystem
- 2. Who am I? • Zohar Elkayam, CTO at Brillix • DBA, team leader, database trainer, public speaker, and a senior consultant for over 18 years • Oracle ACE Associate • Involved with Big Data projects since 2011 • Blogger – www.realdbamagic.com and www.ilDBA.co.il http://brillix.co.il2
- 3. About Brillix • Brillix is a leading company that specialized in Data Management • We provide professional services, training and consulting for Databases, Security, NoSQL, and Big Data solutions • Providing the Brillix Big Data Experience Center 3
- 4. Agenda • What is the Big Data challenge? • A Big Data Solution: Hadoop • HDFS • MapReduce and YARN • Hadoop Ecosystem: HBase, Sqoop, Hive, Pig and other tools • Where does the DBA fits in? http://brillix.co.il4
- 6. The Big Data Challenge http://brillix.co.il6
- 7. Volume • Big data come in one size: Big. • Size is measured in Terabyte(1012), Petabyte(1015), Exabyte(1018), Zettabyte (1021) • The storing and handling of the data becomes an issue • Producing value out of the data in a reasonable time is an issue http://brillix.co.il7
- 8. Variety • Big Data extends beyond structured data, including semi-structured and unstructured information: logs, text, audio and videos • Wide variety of rapidly evolving data types requires highly flexible stores and handling http://brillix.co.il8 Un-Structured Structured Objects Tables Flexible Columns and Rows Structure Unknown Predefined Structure Textual and Binary Mostly Textual
- 9. Velocity • The speed in which the data is being generated and collected • Streaming data and large volume data movement • High velocity of data capture – requires rapid ingestion • Might cause a backlog problem http://brillix.co.il9
- 10. Okay, So What Defines Big Data? • When the data is too big or moves too fast to handle in a sensible amount of time • When the data doesn’t fit any conventional database structure • When the solution to the business need becomes part of the problem • When we think that we can still produce value from that data and want to handle it http://brillix.co.il10
- 11. Value Big data is not about the size of the data, It’s about the value within the data http://brillix.co.il11
- 12. How to do Big Data http://brillix.co.il12
- 13. 13
- 14. Big Data in Practice • Big data is big: technological infrastructure solutions needed • Big data is complicated: • We need developers to manage handling of the data • We need devops to manage the clusters • We need data analysts and data scientists to produce value http://brillix.co.il14
- 15. Infrastructure Challenges • Infrastructure that is built for: • Large-scale • Distributed / scaled out • Data-intensive jobs that spread the problem across clusters of server nodes http://brillix.co.il15
- 16. Infrastructure Challenges (cont.) • Storage: Efficient and cost-effective enough to capture and store terabytes, if not petabytes, of data • Network infrastructure that can quickly import large data sets and then replicate it to various nodes for processing • Security capabilities that protect highly-distributed infrastructure and data 16 http://brillix.co.il
- 17. A Big Data Solution: Apache Hadoop http://brillix.co.il17
- 18. Apache Hadoop • Open source project run by Apache Foundation (2006) • Hadoop brings the ability to cheaply process large amounts of data, regardless of its structure • It Is has been the driving force behind the growth of the big data industry • Get the public release from: • http://hadoop.apache.org/core/ http://brillix.co.il18
- 19. Original Hadoop Components • HDFS: Hadoop Distributed File System – distributed file system that runs in a clustered environment • MapReduce – programming paradigm for running processes over a clustered environments. Main idea: bring the program to the data 19 http://brillix.co.il
- 20. Hadoop Benefits • Reliable solution based on unreliable hardware • Designed for large files • Load data first, structure later • Designed to maximize throughput of large scans • Designed to leverage parallelism • Designed for scale out • Flexible development platform • Solution Ecosystem 20
- 21. What Hadoop Is Not? • Hadoop is not a database – it does not a replacement for DW, or for other relational databases • Hadoop is not for OLTP/real-time systems • Very good for large amount, not so much for smaller sets • Designed for clusters – there is no Hadoop monster server (single server) 21 http://brillix.co.il
- 22. Hadoop Limitations • Hadoop is scalable but it’s not fast • Some assembly is required • Batteries are not included • DIY mindset • Open source limitations apply • Technology is changing very rapidly http://brillix.co.il22
- 23. Hadoop under the Hood http://brillix.co.il23
- 24. Original Hadoop 1.0 Components • HDFS: Hadoop Distributed File System – distributed file system that runs in a clustered environment • MapReduce – programming paradigm for running processes over a clustered environments 24 http://brillix.co.il
- 25. Hadoop 2.0 • Hadoop 2.0 changed the Hadoop conception and introduced better resource management and speed: • Hadoop Common • HDFS • YARN • Multiple data processing frameworks including MapReduce, Spark and others http://brillix.co.il25
- 26. HDFS is... • A distributed file system • Designed to reliably store data using commodity hardware • Designed to expect hardware failures and stay resilient • Intended for large files • Designed for batch inserts 26 http://brillix.co.il
- 27. Files and Blocks • Files are split into blocks (single unit of storage) • Managed by Namenode and stored on Datanodes • Transparent to users • Replicated across machines at load time • Same block is stored on multiple machines • Good for fault-tolerance and access • Default replication factor is 3 27
- 28. HDFS Node Types HDFS has three types of Nodes: • Datanodes • Responsible for actual file store • Serving data from files (data) to client • Namenode (MasterNode) • Distribute files in the cluster • Responsible for the replication between the datanodes and for file blocks location • BackupNode • Backup node for the NameNode 28
- 29. HDFS is Good for... • Storing large files • Terabytes, Petabytes, etc... • Millions rather than billions of files • 128MB or more per file • Streaming data • Write once and read-many times patterns • Optimized for streaming reads rather than random reads 29
- 30. HDFS is Not So Good For... • Low-latency reads / Real-time application • High-throughput rather than low latency for small chunks of data • HBase addresses this issue • Large amount of small files • Better for millions of large files instead of billions of small files • Multiple Writers • Single writer per file • Writes at the end of files, no-support for arbitrary offset 30
- 31. MapReduce is... • A programming model for expressing distributed computations at a massive scale • An execution framework for organizing and performing such computations • Bring the code to the data, not the data to the code http://brillix.co.il31
- 32. The MapReduce Paradigm • Imposes key-value input/output • We implement two functions: • MAP - Takes a large problem and divides into sub problems and performs the same function on all sub-problems Map(k1, v1) -> list(k2, v2) • REDUCE - Combine the output from all sub-problems (each key goes to the same reducer) Reduce(k2, list(v2)) -> list(v3) • Framework handles everything else (almost) 32
- 34. MapReduce is Good For... • Embarrassingly parallel algorithms • Summing, grouping, filtering, joining • Off-line batch jobs on massive data sets • Analyzing an entire large dataset http://brillix.co.il34
- 35. MapReduce is NOT Good For... • Jobs that needs shared state or coordination • Tasks are shared-nothing • Shared-state requires scalable state store • Low-latency jobs • Jobs on smaller datasets • Finding individual records 35 http://brillix.co.il
- 36. YARN • Takes care of distributed processing and coordination • Scheduling • Jobs are broken down into smaller chunks called tasks • These tasks are scheduled to run on data nodes • Task Localization with Data • Framework strives to place tasks on the nodes that host the segment of data to be processed by that specific task • Code is moved to where the data is 36
- 37. YARN • Error Handling • Failures are an expected behavior so tasks are automatically re- tried on other machines • Data Synchronization • Shuffle and Sort barrier re-arranges and moves data between machines • Input and output are coordinated by the framework 37
- 38. Extending Hadoop
- 39. Improving Hadoop • Core Hadoop is complicated so some tools and solution frameworks were added to make things easier • There are over 80 different Apache projects for big data solution which uses Hadoop • Hadoop Distributions collects some of these tools and release them as a complete package • Cloudera • HortonWorks • MapR • Amazon EMR 39
- 40. Common HADOOP 2.0 Technology Eco System 40 http://brillix.co.il
- 41. Databases and DB Connectivity • HBase: NoSQL Key/Value wide-column oriented datastore that is native to HDFS • Sqoop: a tool designed to import data from and export data to relational databases (HDFS, Hbase, or Hive) 41
- 42. HBase • HBase is the closest thing we had to database in the early Hadoop days • Distributed key/value with wide-column oriented database built on top of HDFS - providing Big Table-like capabilities • Does not have a query language: only get, put, and scan commands • Usually compared with Cassandra (non-Hadoop native Apache project) 42
- 43. When do we use HBase? • Huge volumes of randomly accessed data • HBase is at its best when it’s accessed in a distributed fashion by many clients (high consistency) • Consider HBase when we are loading data by key, searching data by key (or range), serving data by key, querying data by key or when storing data by row that doesn’t conform well to a schema. 43
- 44. When NOT to use HBase • HBase doesn’t use SQL, don’t have an optimizer, doesn’t support in transactions or joins • HBase doesn’t have data types • See project Apache Phoenix for better data structure and query language when using HBase 44
- 45. Sqoop2 • Sqoop is a command line tool for moving data from RDBMS to Hadoop • Uses MapReduce program or Hive to load the data • Can also export data from HBase to RDBMS • Comes with connectors to MySQL, PostgreSQL, Oracle, SQL Server and DB2. • Example: $bin/sqoop import --connect 'jdbc:sqlserver://10.80.181.127;username=dbuser;password=dbpasswd;database=tpch' --table lineitem --hive-import $bin/sqoop export --connect 'jdbc:sqlserver://10.80.181.127;username=dbuser;password=dbpasswd;database=tpch' --table lineitem -- export-dir /data/lineitemData 45
- 46. Improving MapReduce Programmability • Pig: Programming language that simplifies Hadoop actions: loading, transforming and sorting of data • Hive: enables Hadoop to operate as data warehouse using SQL-like syntax 46
- 47. Pig • Pig is an abstraction on top of Hadoop • Provides high level programming language designed for data processing • Scripts converted into MapReduce code, and executed on the Hadoop Clusters • Makes ETL processing and other simple MapReduce easier without writing MapReduce code • Often replaced by more up-to-date tools like Apache Spark 47
- 48. Hive • Data warehousing solution built on top of Hadoop • Provides SQL-like query language named HiveQL • Minimal learning curve for people with SQL expertise • Data analysts are target audience • Early Hive development work started at Facebook in 2007 48
- 49. Hive Provides • Ability to bring structure to various data formats • Simple interface for ad hoc querying, analyzing and summarizing large amounts of data • Access to files on various data stores such as HDFS and HBase • Also see: Apache Impala (mainly in Cloudera) 49
- 50. Improving Hadoop – More useful tools • For improving coordination: Zookeeper • For Improving log collection: Flume • For improving scheduling/orchestration: Oozie • For Improving UI: Hue/Ambari 50
- 51. ZooKeeper • ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services • It allows distributed processes to coordinate with each other through a shared hierarchal namespace which is organized similarly to a standard file system • ZooKeeper stamps each update with a number that reflects the order of all ZooKeeper transactions 51
- 52. Flume • Flume is a distributed system for collecting log data from many sources, aggregating it, and writing it to HDFS • Flume does for file what Sqoop did for RDBMS • Flume maintains a central list of ongoing data flows, stored redundantly in Zookeeper. 52
- 53. 53
- 54. Is Hadoop the Only Big Data Solution? • No – There are other solutions: • Apache Spark and Apache Mesos frameworks • NoSQL systems (Apache Cassandra, CouchBase, MongoDB and many others) • Stream analysis (Apache Kafka, Apache Flink) • Machine learning (Apache Mahout, Spark MLib) • Some can be integrated with Hadoop, but some are independent http://brillix.co.il54
- 55. Where Does the DBA Fits In? • Big Data solutions are not databases. Databases are probably not going to disappear, but we feel the change even today: DBA’s must be ready for the change • DBA’s are the perfect candidates to transition into Big Data Experts: • Have system (OS, disk, memory, hardware) experience • Can understand data easily • DBA’s are used to work with developers and other data users http://brillix.co.il55
- 56. What DBAs Needs Now? • DBA’s will need to know more programming: Java, Scala, Python, R or any other popular language in the Big Data world will do • DBA’s needs to understand the position shifts, and the introduction of DevOps, Data Scientists, CDO etc. • Big Data is changing daily: we need to learn, read, and be involved before we are left behind… http://brillix.co.il56
- 58. Summary • Big Data is here – it’s complicated and RDBMS does not fit anymore • Big Data solutions are evolving Hadoop is an example for such a solution • Spark is very popular Big Data solution • DBA’s need to be ready for the change: Big Data solutions are not databases and we make ourselves ready http://brillix.co.il58
- 59. Thank You Zohar Elkayam twitter: @realmgic [email protected] www.realdbamagic.com http://brillix.co.il59
Editor's Notes
- #26: YARN = Yet another resource negotiator
- #41: Source: Hadoop documentation
- #53: One very common use of Hadoop is taking web server or other logs from a large number of machines, and periodically processing them to pull out analytics information. The Flume project is designed to make the data gathering process easy and scalable, by running agents on the source machines that pass the data updates to collectors, which then aggregate them into large chunks that can be efficiently written as HDFS files. It’s usually set up using a command-line tool that supports common operations, like tailing a file or listening on a network socket, and has tunable reliability guarantees that let you trade off performance and the potential for data loss.