Automated Metadata Management in Data Lake – A CI/CD Driven Approach
0 likes497 views
The document discusses an automated approach to metadata management in data lakes utilizing a CI/CD framework. It highlights the need for efficient metadata management, outlines the tool's architecture, and includes a live demo showcasing the creation and management of database configurations. Key components include a configuration file, governance, and integration with tools like Python and GitLab CI.
Automated Metadata Management in Data Lake – A CI/CD Driven Approach
- 1. Automated Metadata Management in Data Lake – A CI/CD Driven Approach Keyuri Shah, Lead Engineer Josh Reilly, Lead Engineer
- 2. Agenda § Introduction § Need for Metadata Management § Architecture § Overview on Tool § Live Demo
- 3. Need for Metadata Management • What is Metadata Management • Motivation for a config driven tool • Governance • Easy to Maintain • Development Stack • Python • Gitlab CI • Use Cases • Enterprise Data Lake • Sharing Lake across different teams
- 4. Config File Options ▪ name ▪ owning_team ▪ description ▪ access: ▪ type: ad_group ▪ <env>: AD_GRP_NAM ▪ type: data • name • description • schema • encrypted_columns • masked_columns • Tables • Database • name • database • description • query • Views
- 5. Design
- 6. Live Demo • Create a database/table/view config • Check in to Git • Run CICD pipeline to plan and apply to int • Verify Database/Table/View in Databricks • Update Schema & Run Pipeline
- 7. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions. Keyuri Shah: https://www.linkedin.com/in/keyuri-shah Josh Reilly: https://www.linkedin.com/in/josh-reilly-51052996/