Data Governance for Data Lakes
2 likes•1,267 views
The document discusses the importance of governance in data lakes, highlighting the risks of turning data lakes into data swamps due to the absence of proper oversight. It emphasizes the need for systematic data management, including tracking data lineage, ensuring security and compliance, and providing metadata management tools like Teradata Loom. The document outlines solutions for building governed data lakes using Hadoop infrastructure and emphasizes the role of governance in enabling productivity and meeting regulatory requirements.
Data Governance for Data Lakes
- 1. Anarchy Doesn’t Scale: Your Data Lake Needs Governance Kiran Kamreddy Sr. Product Marketing Manager Hadoop Portfolio, Teradata
- 2. One Data Lake: Many Definitions
- 3. One Data Lake: Many Definitions A centralized repository of raw data into which many data-‐producing streams flow and from which downstream facili7es may draw. Informa4on Sources Data Lake Downstream Facili4es Data Variety is the driving factor in building a Data Lake
- 4. Data Lake Maturity & Risks Data Lake ini4a4ves usually start small – Single business unit – Few data sources, applica7ons – Few data scien7sts – Exclusive domain Informal, Tribal ways of sharing – Handshakes – Over the desk conversa7ons – No processes, mechanisms – Everyone is free to use Anarchy works well
- 5. Data Lake Maturity & Risks Data Lake ini4a4ves grow popular – More Business Units – More Data Sources – More Applica7ons – More Users …. grow into more complex environments Without proper governance mechanisms Data lakes risk turning data swamps Consistency Quality Security
- 6. What can Governance do for my data lake? Fundamental capabili4es for tracking, organizing and understanding data - Where did my data come from ? How is it being transformed ? - Track usage, resolve anomalies, visualize, op7mize and clarify data lineage - Search and access data ( not only browse ) - Assess data quality and fitness for purpose Specialized capabilities to meets security& compliance requirements - Govern who can/cannot access the data and who cannot - Data life cycle management, archiving and retention policies - Auditing, compliance Data Governance first approach to prevent turning to Data swamps RetrofiIng data governance is not feasible
- 7. Governance and Productivity Governance should supports day-‐to-‐day use of data – Data workers need a strong understanding – Roles for data stewards, data owners, data analysts/scien7sts need to be assigned Operational Metadata is critical to understanding – Where did it come from? – What is the environment? – landing zone, OS, Line of Business – What processes touched my data?did you lose any data? – Checksums etc. – When did the data get ingested, transformed? – Did it get exported, when, where how will it be used (organizational)? Provision consistent ingest methods that track operational metadata
- 8. What about Security & Compliance? • Compliance and Regulatory – Capture, store and move data – Sarbanes-‐Oxley, HIPAA, Basel II • Security – Authoriza7on, Authen7ca7on – Handling sensi7ve data • Audi4ng – Recording every aUempt to access • Archive & Reten4on – Data life cycle policies
- 9. Governance Challenges on Hadoop Data Lakes Hadoop is different to DW • Scale: High volumes of data, mul7ple user access • Variety: Schema-‐on-‐read, mul7ple formats of data • Mul7ple storage layers (HDFS, Hive, HBase) • Many processing engines (MR, Hive, Pig, Impala, Drill…) • Many workflow engines/schedules (Cron, Oozie, Falcon…) • Holis7c view of data with required context is difficult Hadoop needs less stringent, more flexible mechanisms Balance agility and self service with processes, rules, regulations Maintain Governance without losing Hadoop's power
- 10. • Apache Hadoop has built-‐in support for these capabili4es • HCatalog, Hive, etc • Hadoop distribu4on vendors have all made improvements in each of these areas • Navigator, Ranger, etc • Vendors provide specialized capabili4es in each area that go beyond what a Hadoop distribu4on provides Multiple Approaches….. but incomplete
- 11. One approach for Data Governance on Hadoop Teradata Loom® – Integrated Data Management for Hadoop Metadata management, Lineage, Data Wrangling Automa7c data cataloging, data profiling and sta7s7cs genera7on Teradata Rainstor – Data Archiving Structured data archiving in Hadoop with robust security Compliance and auditing ThinkBig – Hadoop professional services Hadoop Data Lake – packaged service/product offering to build and deploy high-quality, governed data lakes
- 12. Teradata Loom Find and Understand Your Data • Ac7veScan – Data cataloging – Event triggers – Job detec7on and lineage crea7on – Data profiling (sta7s7cs) • Workbench and Metadata Registry – Data explora7on and discovery – Technical and business metadata – Data sampling and previews – Lineage rela7onships – Search over metadata – REST API – easily integrate third-‐party apps Prepare Your Data • Data Wrangling – Self-‐service, interac7ve data wrangling for Hadoop – Metadata tracked • HiveQL – Joins, unions, aggrega7ons, UDFs – Metadata tracked in Loom
- 13. Teradata Rainstor – Retain data online that is queryable for an indefinite period – Re7re data that are no longer required with auto-‐expira4on policies – Comply with strict government rules and regula7ons – Retain the metadata as it was originally captured – Store tamper-‐proof, immutable (unchangeable) data – Maintain availability to data as RDBMS versions change or expire – Compression, MPP SQL query engine, Encryp7on, Audi7ng
- 14. Big Data Services from ThinkBig Big Data Strategy & Roadmap Data Lake Implementation Analytics & Data Science Training & Support Lack of Clear Big Data Strategy Difficulty Turning Data into Action Missing Big Data Skills Focused exclusively on tying Hadoop and big data solu7ons to measurable business value Data Scattered & Not Well Understood
- 15. Impact • Co-‐loca4on of data provides more efficient workflow for analysts • Hadoop provides scalability at a lower cost than tradi4onal systems • Develop new insights to drive business value Situa4on Problem Lack of centralized metadata repository makes data governance impossible. Enterprise must have transparency into data in the cluster and capability to define extensible metadata. Large scale data lake planned with many heterogeneous sources and many individual analyst users. Solu4on Hadoop provides data lake infrastructure. Loom provides centralized metadata management, with an automa7on framework. Data Governance for Hadoop Bank Holding Company
- 16. Talk Summary • Data governance is cri4cal to building a successful data lake – Fundamental governance capabili7es make data workers more produc7ve – Solu7ons for mee7ng regulatory requirements are also needed • Teradata Loom provides required data cataloging and lineage capabili4es to make hadoop users more produc4ve • RainStor provides advanced archiving solu4on • ThinkBig Data Lake provides the complete package Stop by our booth for more details
- 17. We Love Feedback Questions/Comments Email: [email protected] Follow Me Twitter @kirankamreddy Rate This Session with the PARTNERS Mobile App Remember To Share Your Virtual Passes Follow Teradata 2015 PARTNERS www.teradata-partners.com/social