Hadoop in Validated Environment - Data Governance Initiative
2 likes1,184 views
The document discusses Merck's data governance initiative to implement Hadoop and establish data governance policies and processes. It outlines their 90 day and 6 month roadmaps to set up a data lake, implement data cataloging and metadata management, and establish access controls and data lineage tracking to enable self-service analytics while ensuring data governance and security. Key components include Hadoop, Hive, HDFS, Ranger, Falcon, and metadata stored in the Hive metastore.
![Process Logs
2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1702)) - Get metadata for subqueries
17865102 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Get metadata for destination tables
2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1726)) - Get metadata for destination
tables
17865345 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Completed getting MetaData in Semantic Analysis
2015-06-04 12:53:31,844 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(10004)) - Completed getting MetaData
in Semantic Analysis
17865347 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Not invoking CBO because the statement has too few joins
2015-06-04 12:53:31,846 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:canHandleAstForCbo(10258)) - Not invoking CBO
because the statement has too few joins
Heart beat
17866695 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException [Error 10044]: Line 2:18 Cannot insert into target
table because column number/types are different ''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns.
org.apache.hadoop.hive.ql.parse.SemanticException: Line 2:18 Cannot insert into target table because column number/types are different
''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns.
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genConversionSelectOperator(SemanticAnalyzer.java:6535)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6336)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8977)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8868)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9713)](https://image.slidesharecdn.com/june10435pmmerckmsditlasekryzl-150619223331-lva1-app6892/85/Hadoop-in-Validated-Environment-Data-Governance-Initiative-37-320.jpg)
Hadoop in Validated Environment - Data Governance Initiative
- 1. Hadoop in Validated Environment Data Governance Initiative
- 2. Martin Ryzl Director, Analytics Platform Ivo Lasek Architect, Analytics Platform
- 5. 90 Days
- 7. Who is the dataset owner? How can I get access? What does the data mean? How can I reproduce the results? Where is the data I need?
- 8. 6 Months
- 9. Where Is the Data I Need?
- 10. Data Lake
- 12. Data Lake Merge Clean Security and Data Governance
- 14. Who Is the Dataset Owner?
- 18. How Can I Get Access?
- 23. What Does the Data Mean?
- 24. Semantic meaning – Metastore id name ssn birth_n o phone id personal_number employee_number first_name division
- 31. How Can I Reproduce the Data?
- 32. Reporting Data Reproducibility delta1 delta2 delta3 Raw Data Aggregate v1.0 delta1..delta3 Reporting Data
- 33. Reproducibility delta1 delta2 delta3 Raw Data delta1..delta3 Reporting Data Aggregate v1.0 delta4 delta5 delta6 delta1..delta7 Reporting Data Aggregate v1.0 delta7
- 34. Traceability delta1 delta2 delta3 Raw Data delta1..delta3 Reporting Data Aggregate v1.0 delta1..delta3 Cleaned Data Clean v1.0 delta4 delta5 delta6 delta1..delta7 Reporting Data Aggregate v1.0 delta1..delta7 Cleaned Data Clean v1.1 delta7
- 35. Data Lineage
- 36. Access Logs
- 37. Process Logs 2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1702)) - Get metadata for subqueries 17865102 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Get metadata for destination tables 2015-06-04 12:53:31,601 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:getMetaData(1726)) - Get metadata for destination tables 17865345 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Completed getting MetaData in Semantic Analysis 2015-06-04 12:53:31,844 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:analyzeInternal(10004)) - Completed getting MetaData in Semantic Analysis 17865347 [main] INFO org.apache.hadoop.hive.ql.parse.SemanticAnalyzer - Not invoking CBO because the statement has too few joins 2015-06-04 12:53:31,846 INFO [main] parse.SemanticAnalyzer (SemanticAnalyzer.java:canHandleAstForCbo(10258)) - Not invoking CBO because the statement has too few joins Heart beat 17866695 [main] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: SemanticException [Error 10044]: Line 2:18 Cannot insert into target table because column number/types are different ''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns. org.apache.hadoop.hive.ql.parse.SemanticException: Line 2:18 Cannot insert into target table because column number/types are different ''2015-06-04-07-50'': Table insclause-0 has 165 columns, but query has 166 columns. at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genConversionSelectOperator(SemanticAnalyzer.java:6535) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6336) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8977) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8868) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9713)
- 38. Who is the dataset owner? How can I get access? What does the data mean? How can I reproduce the results? Where is the data I need? HDFS/Hive Metastore Ranger Metastore Falcon
- 39. Contacts • Martin Ryzl ([email protected]) • Ivo Lasek ([email protected]) • http://www.merck.com/ • http://www.msdit.cz/