Modern Data architecture Design
2 likes839 views
The document discusses modern data architectures. It presents conceptual models for data ingestion, storage, processing, and insights/actions. It compares traditional vs modern architectures. The modern architecture uses a data lake for storage and allows for on-demand analysis. It provides an example of how this could be implemented on Microsoft Azure using services like Azure Data Lake Storage, Azure Data Bricks, and Azure Data Warehouse. It also outlines common data management functions such as data governance, architecture, development, operations, and security.
Modern Data architecture Design
- 2. CONCEPTUAL DW ARCHITECTURE Ingestion Storage Processing Insights Actions Data sources •Structured •Unstructured Acquisition layer •Real time •Near Real time •Batch Foundation layer •Data lake •Data refinery •Data archival Optimization layer •Data marts •Cubes •Data science Consumption layer •Business applications •Reporting •API’s Source systems Data integration layer Data storage layer Reporting & Visualization layer ETL ELT
- 3. Visualization RDBMS Source Flat files XML E T L Analysis Reporting Enterprisedatawarehouse Staging Data warehouse Data mart Cubes TRADITIONAL DATA ARCHITECTURE Data Science Algorithms ADS
- 4. Visualization RDBMS Source Flat files Data lake Analytics sandbox Traditional EDW Unstructured data HDFS MDM, Security & Governance MODERN ARCHITECTURE- LOGICAL On-demand analysis Canned Reporting Adhoc Reporting Datavirtualization Data Science Algorithms
- 5. RDBMS Source Flat files Unstructured data Security & Governance MODERN ARCHITECTURE - AZURE Business Apps API’s Reporting Azure Data Factory Azure Data Lake Storage Azure Data Bricks Azure Cosmos DB Ingest Store Prepare Model Serve Azure Data warehouse Data Virtualization * Proposed Architecture using Azure Cloud, if the business is subscriber of other cloud providers then propose services with similar functionalities
- 6. DATA MANAGEMENT FUNCTIONS Data Governance Data Architecture Data Development Data Operations Data Security Reference & Master DM DW & BI Document & Content Metadata Data Quality •Authority to plan, monitor and enforce standards and policies for data assetsData Governance •Enterprise modelling •Value chain analysisData Architecture •Analysis •Data modelling & DesignData development •Acquisition, Recovery, Tuning •Retention and archivalData Operations •Standards, Administration •Authentication, AuditingData Security •External & Internal codes •Primary data: Customer, ProductReference & MDM •Architecture, Implementation •Training, Monitoring & TuningDW & BI •Acquisition & Storage, •Backup, Retrieval. Content mgmtDocument & Content •Architecture, Integration •Control, MonitorMetadata •Specification, Analysis •Measurement, ImprovementDQM