Building Serverless Data Infrastructure in the AWS Cloud
1 like686 views
The document discusses building serverless data infrastructure using AWS cloud, highlighting the shift from traditional data warehousing to a more flexible 'data lake' architecture. It emphasizes the importance of automation, event-driven handling, and the use of serverless technologies to manage data ingestion, processing, and analytics efficiently. Key principles include avoiding state in event handling and utilizing services like AWS Glue, Lambda, and various storage options to optimize data flows and reduce costs.
![StorageIngestion
SQS
SNS
Kinesis
DynamoDB/RDS
event triggers y = f (x)
y = f (x, y)
y = f ([x, y])
event handlers
AWS Glacier
(archival)
/{source}-raw/{key}/YYYY-MM-DD
/{source}-refined/{key}/YYYY-MM-DD
AWS Lambda AWS S3
(ready)
KMS
(encryption) lifecycle policies
IAM + Directory
(access control)
CloudWatch/Trail
to S3 direct
AWS Step Functions
(coordinated state)](https://image.slidesharecdn.com/serverlessdata-awsbmddc2017-v1alt-171115040834/85/Building-Serverless-Data-Infrastructure-in-the-AWS-Cloud-32-320.jpg)
Building Serverless Data Infrastructure in the AWS Cloud
- 1. Building Serverless Data Infrastructure in the AWS Cloud Ryan Plant @ryan_plant November 10, 2017
- 4. WHAT WE’LL COVER The New Data Economy Reference Architecture Using the AWS Cloud
- 5. The world’s most valuable resource is no longer oil, but data… May 6th, 2017
- 6. Data => Revenue (but extraction, refinement, packaging, and distribution needed)
- 7. DW Traditional Data Warehousing Volume, variety, and velocity… Advanced analytics… Artificial intelligence… ”What got us here won’t (entirely) get us there…” Mostly proprietary… Costly and complex to scale…
- 8. Next Generation Data Infrastructure (i.e. the “data lake”)
- 9. James “Data Lake” Dixon
- 10. If you think of a datamart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state…
- 11. From Data Warehouses to Lakes A data pond, lake, ocean is not a product it’s an architecture… (and architecture is a principled and pattern-oriented approach to building systems) Any and all data… Any source and format… Any time…
- 12. WHAT WE’LL COVER The New Data Economy Reference Architecture Using the AWS Cloud
- 13. APPS & SOURCES STORAGE AND PROCESSING LAYER SERVING LAYER Storage Catalog Processing Analytics & Artificial Intelligence Ingestion Models & Marts DATA OPS API Search Security Config Telemetry Cost Mgmt
- 14. DATA OPS Security Config Telemetry Cost Mgmt SERVING LAYER Models & Marts API Search APPS & SOURCES STORAGE AND PROCESSING LAYER StorageIngestion Catalog Processing Analytics & Artificial Intelligence
- 15. Data Ingestion Pipelines SERVICE SERVICE SERVICE MONOLITH MONOLITH MONOLITH Change Data Capture (CDC) STREAMS MESSAGING FILE EXTRACTS STORAGE source data aggregated, stored indefinitely many supported formats append append PUT Security segregation & encryption
- 16. Storage and Catalog STORAGE RAW REFINED Catalog • Register source and schema • Data attribute inventory • Relationships and dependencies • Etc… dataIngestion
- 17. Catalog Raw to Refined Processing Pipelines STORAGE RAW REFINED Processing Pipelines dataIngestion C1 C2 C3 C..n • Preserve RAW data; enrich only • Apply transforms to create new, REFINED datasets (e.g. customer partitioned views) • Catalog new datasets • Enable new use cases: • Reporting/Analytical views • Machine/Deep Learning X Y Z ALL DATA
- 18. Processing Pipelines Catalog Analytics and AI STORAGE RAW REFINED dataIngestion Analytics and Artificial Intelligence C1 C2 C3 C..nALL DATA X Y Z … … …
- 19. DATA OPS Security Config Telemetry Cost Mgmt APPS & SOURCES STORAGE AND PROCESSING LAYER StorageIngestion Catalog Processing Analytics & Artificial Intelligence SERVING LAYER Models & Marts API Search
- 20. Processing Pipelines Catalog Curation and Serving STORAGE RAW REFINED dataIngestion Analytics and Artificial Intelligence C1 C2 C3 C..nALL DATA X Y Z Models and Marts … … … Search … … …
- 21. Processing Pipelines Catalog STORAGE RAW REFINED dataIngestion Analytics and Artificial Intelligence C1 C2 C3 C..nALL DATA X Y Z Models and Marts … … … Search … … … API
- 22. APPS & SOURCES STORAGE AND PROCESSING LAYER SERVING LAYER Storage Catalog Processing Analytics & Artificial Intelligence Ingestion Models & Marts DATA OPS API Search Security Config Telemetry Cost Mgmt
- 23. WHAT WE’LL COVER The New Data Economy Reference Architecture Using the AWS Cloud
- 24. Lots of software, hardware, etc.
- 25. TRADITIONAL INVESTMENT IN NEXT GENERATION DATA
- 26. CAPITAL AND RISK BARRIERS acquire/write and maintain software procure, install, and maintain hardware get commercial real estate license
- 28. PUBLIC CLOUD ECONOMIES OF SCALE
- 29. CLOUD OPTIMIZATION Infrastructure as a Service Someone else’s hardware and real estate Your software, your (virtual) servers Platform as a Service Someone else’s software, servers, hardware and real estate Your custom application software Software as a Service Someone else’s application software, you provide the data (everything else doesn’t matter) Cycle Time Capital Optimization Differentiation Focus High Higher Highest
- 30. Go Serverless! (as much as possible)
- 31. everything is an event: messages, log entries, file I/Os, clock alarms, etc. listen for events: trigger a handler with an event stateless event handling: avoid state, persist as event source, handoff as soon as possible automation through orchestration and coordination Principles for event-driven, reactive data infrastructure primed for serverless architectures
- 32. StorageIngestion SQS SNS Kinesis DynamoDB/RDS event triggers y = f (x) y = f (x, y) y = f ([x, y]) event handlers AWS Glacier (archival) /{source}-raw/{key}/YYYY-MM-DD /{source}-refined/{key}/YYYY-MM-DD AWS Lambda AWS S3 (ready) KMS (encryption) lifecycle policies IAM + Directory (access control) CloudWatch/Trail to S3 direct AWS Step Functions (coordinated state)
- 33. Catalog Storage Sources Ingestion AWS Glue (serverless ETL/ELT) source crawlers metadata classifier classifier doSomething(…) {…}trigger Processing Pipelines jobs and job runner To Targets
- 34. Catalog Storage Sources & Targets Ingestion Processing Pipelines AWS Glue (serverless ETL/ELT) AWS EMR (Managed Hadoop) Streaming Kinesis Batch AWS Batch Targets & SourcesIngestion Serving Layer
- 35. Catalog Storage Processing Pipelines AWS Glue (serverless ETL/ELT) Serving Layer AWS ElasticSearch (managed ES) AWS RedShift Spectrum (Parallel DW) Sources Ingestion AWS Athena (Ad-hoc Query)
- 36. Catalog Storage Processing Pipelines Serving Layer Sources Ingestion AWS API Gateway (serverless APIs) AWS QuickSight (visualization) AWS Cognito (Web/Mobile Identity and SSO)
- 37. WHAT WE’LL COVER The New Data Economy Reference Architecture Using the AWS Cloud
- 38. CLOUD OPTIMIZATION Infrastructure as a Service Someone else’s hardware and real estate Your software, your (virtual) servers Platform as a Service Someone else’s software, servers, hardware and real estate Your custom application software Software as a Service Someone else’s application software, you provide the data (everything else doesn’t matter) Cycle Time Capital Optimization Differentiation Focus High Higher Highest
- 39. CLOUD OPTIMIZATION Infrastructure as a Service Someone else’s hardware and real estate Your software, your (virtual) servers Platform as a Service Someone else’s software, servers, hardware and real estate Your custom application software Software as a Service Someone else’s application software, you provide the data (everything else doesn’t matter) You are likely here… Aim here… TBD Opportunity! Public Cloud R&D Investment
- 40. SERVERLESS: USE CAUTION The floor is wet (and is constantly getting mopped!) The edges are sharp: • Development, Test, Debug tools and experience • Configuration and Deployment challenges • Variable, non-deterministic performance Extremely new (but inevitable) paradigm…