8 ways to leverage AWS Lambda in your Big Data workloads
2 likes•807 views
The document discusses 8 ways to leverage AWS Lambda for Big Data workloads. It provides examples of architectures using AWS Lambda for real-time processing of streaming data from sources like Kinesis, applying custom logic to data uploaded to services like S3, DynamoDB, and IoT, and simplifying resource management through automated tasks run by Lambda functions.
8 ways to leverage AWS Lambda in your Big Data workloads
- 1. 8 ways to leverage AWS Lambda in your Big Data workloads Adrian Hornsby Solutions Architect with AWS [email protected] @adhorn
- 2. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 1- Overview of AWS Lambda and Big Data. 2- AWS Lambda usage in Big Data architectures
- 3. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Lambda
- 4. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is Lambda Functions are the unit of deployment and scaling.
- 5. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How does a Lambda function looks like? def hello(event, context): return { "message": ”Hello World!", "event": event } 'use strict'; handlermodule.exports.hello = (event, context, cb) => cb(null, { message: ’Hello World!', event } );
- 6. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How Lambda works S3 event notifications DynamoDB Streams Kinesis events Cognito events SNS events Custom events CloudTrail events LambdaDynamoDB Kinesis S3 Any custom Redshift SNS CloudWatch events
- 7. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Continuous scaling No servers to manage Never pay for idle – no cold servers Benefits of AWS Lambda
- 8. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Key scenarios Data Triggers Customize behavior on data updates in S3, SNS and DDB Control systems Customize responses and response workflows to state changes within AWS Serverless backends Execute server side backend logic in a cross platform fashion Big data Realtime processing of streaming data updates using Kinesis
- 9. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Big Data
- 10. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data Gap 1990 2000 2010 2020 Generated Data Available for Analysis Data Volume Sources: Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011 IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares
- 11. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data for Competitive Advantage • Customer segmentation • Marketing spend optimization • Financial modeling & forecasting • Ad targeting & real-time bidding • Clickstream analysis • Fraud detection • Security threat detection
- 12. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data Lake How Big Data workloads look like Collect Store Analyze Visualize
- 13. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Divide and conquer your Big Data workloads Data Sources Speed/Stream (Real-‐time) Processing Pipeline Batch Processing Pipeline
- 14. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Architectures for Big Data using AWS lambda
- 15. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. #1 Data Sharing Platform using AWS Lambda
- 16. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Lambda-based fan-out function S3 Bucket AWS Lambda Kinesis Stream Amazon SNS DynamoDB Source data
- 17. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. #2 Amazon S3 input validation and conversion layer.
- 18. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Event-Driven Batch controlled with AWS Lambda Source data Source data Source data Source data Data Staging Layer /data/source-‐raw Data Staging Layer /data/source-‐validated Input Validation and Conversion layer Input Tracking layer State Management Store
- 19. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. #3 Indexing Amazon S3 metadata using AWS Lambda.
- 20. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3 Metadata Index Source data Source data Source data Source data S3 Bucket DynamoDB ObjectCreate Event Put Index Entry
- 21. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. #4 Indexing Amazon DynamoDB Content using AWS Lambda
- 22. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Indexing DynamoDB content using ElasticSearch DynamoDB DynamoDB Streams Push changes AWS Lambda Amazon Elasticsearch Service Index Record Search Query
- 23. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. #5 Apply custom logic to data being uploaded into Kinesis.
- 24. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-time Clickstream Processing with Amazon Kinesis Web Clients Users API Gateway Kinesis Stream Input JSON payload Kinesis Stream Output CSV payload AWS Lambda Amazon SNS Kinesis Analytics
- 25. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. #6 Apply custom logic to data being uploaded into AWS IoT.
- 26. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Anomaly Detection Using AWS Lambda AWS Lambda Amazon SNS DynamoDB AWS IoTSensors Control System
- 27. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. #7 Simplify Resource Management using AWS Lambda.
- 28. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Automated cluster backups using AWS Lambda AWS Lambda Amazon Redshift Cluster 1 Cluster 2 Cluster 3 Rules: every 15min Amazon Cloudwatch Snapshot Snapshot Snapshot
- 29. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Auto Start/Shutdown of Instances AWS Lambda Amazon Cloudwatch Rules: every day at 21h30 Rules: every day at 6h15 Sleep trigger Wakeup trigger AWS Resources (EC2 instances)
- 30. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Auto Tagging resources as they start Amazon Cloudwatch AWS LambdaEvents: RunInstances EC2 Instances Tag: Owner = userName PrincipalId = aws:userid
- 31. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. #8 Orchestrating Big Data pipelines using AWS Lambda.
- 32. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Pipeline Orchestration S3 Bucket or Manifest File Trigger Launch Data Pipeline EMR Cluster Create AWS Lambda Amazon SNS S3 BucketMetadata Index
- 33. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Big Data Case Studies Learn from other AWS customers aws.amazon.com/solutions/case-studies/big-data
- 34. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thanks!