Architecting a datalake
3 likes•616 views
This document discusses architecting a data lake. It begins by introducing the speaker and topic. It then defines a data lake as a repository that stores enterprise data in its raw format including structured, semi-structured, and unstructured data. The document outlines some key aspects to consider when architecting a data lake such as design, security, data movement, processing, and discovery. It provides an example design and discusses solutions from vendors like AWS, Azure, and GCP. Finally, it includes an example implementation using Azure services for an IoT project that predicts parts failures in trucks.
Architecting a datalake
- 1. Architecting a Datalake Laurent Léturgez – Sep 2019 Big Data Meetup - Lille
- 2. Whoami • Database and BigData Architect (Hadoop, Data Science and other cool topics) • Former Developer and Consultant • Owner@Premiseo: Data Management on Premises and in the Cloud • Blogger since 2004 • http://laurent-leturgez.com • Twitter : @lleturgez
- 3. What’s on the menu ? • What is a Datalake ? • Keys to architect a Datalake • Design, Security • Data movement, Data Processing • Discovery • Solutions available • Example • Datalake Implementation driven by IoT
- 4. What is a Datalake ? • Repository of data stored in natural format • Single Store of Enterprise data • Raw Data • Transformed Data : Reports, DataViz, Results (AI, ML …) • Data Structure: • Structured Data : Row, Columns, Relational Data • Semi Structured Data: CSV, XML, JSON, log files • Unstructured Data: Mails, Documents, Binaries (Images, Videos)
- 5. What is a Datalake ? • Features • Data are usually integrated unprocessed • Processed data can be kept in the Datalake • Data are kept … ready to be transformed • Data are saved as long as possible • A Datalake is • Organized • Managed
- 6. What is a Datalake ? • A Datalake is not a datawarehouse Source: martinfowler.com
- 7. Keys to architect a Datalake • A well thought design • Vital for • Success • Discovery efficiency • ETL development effort • Coupled with Security and business process
- 8. Keys to architect a Datalake • A well thought design … example • Operational Areas • Raw Area • Data landing zone in native raw format • Data are kept indefinitely in this area • Data Tagging • Folder Structure organized by Source, Dataset, Date etc. • Staging Area • Data Preparation Area : Decompression, cleansing, aggregation • Data Quality Management is usually made here • Hub Area • Trusted layer of data • Data is ready for analytics organized functionaly
- 9. Keys to architect a Datalake • A well thought design … example • (Extra) Supported Area • Master Data Area • Customer, Products, Financial Data • Used by Analytics • Exploratory Area • Playground for Data Scientists and Analysts • Temporary Area • Testing Data decompression • Single point of data storage before move accross network
- 10. Keys to architect a Datalake • Security • Data Access Control • By User • By Application • ETL Softwares • Analytics • … • By Operational zone • By Source Key Point: IAM Integration
- 11. Keys to architect a Datalake • Security • Data Security • Data Lake Management (Role Control) • Data Resilience • Disaster recovery • Backup / Restore • SLA: Availability, RTO, RPO • Data Encryption • At rest • In transit
- 12. Keys to architect a Datalake • Data Movement, Data Processing • Consider the Data Lake as central point for • Data Ingestion • Data Processing
- 13. Keys to architect a Datalake • Data Movement, Data Processing • Consider the Data Lake as central point • Data Ingestion • Tools / ETL • Metadata strategy should be in place (Data Catalog for tagging) • Data Format • Naming convention for files/directories: ingestion date, format, source etc. • Batch or real time • Many small files or few big files • Data Partitioning Maximum query and processing performance • Cloud or OnPrem ? • Network issues, hybrid Cloud considerations • Data Processing
- 14. Keys to architect a Datalake • Data Movement, Data Processing • Consider the Data Lake as central point • Data Ingestion • Data Processing • Tools • Hadoop (on Prem / Cloud) • Legacies Database systems (SQL Server PolyBase, Oracle Connector for Hadoop, AWS Spectrum/Athena etc.) • Analytics, DataViz and ML • Data Bricks, Power BI, SAS, Qlik etc. • Data Colocation • Data Format • Compressed / Uncompressed • Column oriented
- 15. Keys to architect a Datalake • Orchestration • Cloud Automation or Job Automation ? • Batch or real time • Batch automation • Monitoring • Data volume • Real Time (Usually used for IoT) • How is built the pipeline ? • Event based or not ? • Monitoring
- 16. Keys to architect a Datalake • Discovery • Tagging and Metadata management : Similar … but different • MetaData management : • Data about data : creation and modification date, source, format etc. • Traditional metadata: source, connection string, data type, length, versions etc. • Modern metadata: included in files (AVRO For example) or a database • Advanced metadata: automated processing of metadata • Tagging • Set of tag to understand/describe datasets in the datalake • Usually stored in a Catalog or KV database or through Naming conventions • Key points: When the data has been tagged ? Who owns the tagging system ?
- 17. Solutions available • Solutions available • On Prem : • Hadoop / HDFS • Cloud • AWS : S3 Buckets • Azure : Azure Datalake Store Gen1/Gen2, Storage Accounts • GCP: Google Cloud Storage • Oracle Cloud Infrastructure: Object Storage
- 18. Implementation • Example : Solution • Customer : Industry, Trucks maker • Project : Parts failure prediction • Sensors are embedded in trucks • Data collection for parts health • Data are integrated real time in the Datalake • Legacy data are integrated into the datalake (batch mode) • Parts related data (mostly coming from ERPs) : Serial number, provider, purchases etc. • Predictive algorithms are designed to replace parts before they broke
- 19. Implementation • Example: Solution • Azure Datalake Store / Storage Accounts closely integrated with MS SQL Databases • Why not on Prem ? • Infrastructure costs • Fuzzy Data volume prediction • Hadoop management
- 20. Implementation • Example: Solution • Why Azure ? • Microsoft long time customer • Many services already used (Legacy databases: MS SQL DWH, Power BI etc.) • Active Directory Integration: Security, ACL and • Batch Integration by Talend • Real Time Integration by Azure Products (Iot Hub + Azure Functions) • Close integration with DataBricks for Analytics and Data Processing
- 21. Conclusion • DataLake are now central components for enterprises • Without … • Organized Data • Managed Data (Security, design etc.) • High volume of Data • No powerful AI or ML algorithms • No powerful Analytic processes
- 22. Questions ?