Building a Serverless Pipeline
5 likes•3,498 views
This document discusses building a serverless data pipeline using AWS Lambda and other AWS managed services. It provides a 12 step process to create a serverless architecture that collects event data via API calls, stores it in DynamoDB, processes it with Lambda, sends it to Kinesis and Kinesis Firehose, and archives it in S3. This allows building scalable and cost-effective data pipelines and applications without having to manage any servers.
![A simple Lambda function in Python
def lambda_handler(event,context):
result = event['value1'] + event['value2']
return result
aws lambda create-function --function-name myFunc
--handler myFunc.lambda_handler --runtime python2.7
--zip-file fileb://myFunc.zip --memory-size 128
--role arn:aws:iam::ACCOUNT_NUMBER:role/lambda_basic_execution
curl -H "Content-Type: application/json"
-X POST -d "{"value1":5, "value2":7}"
https://API_ENDPOINT/STAGE/RESOURCE
12](https://image.slidesharecdn.com/20160904tiad-161004112546/85/Building-a-Serverless-Pipeline-10-320.jpg)
Building a Serverless Pipeline
- 1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Building a serverless data pipeline Julien Simon, Principal Technical Evangelist, AWS [email protected] @julsimon
- 3. AWS Lambda • Deploy pure funcJons in Java, Python and Node.js • Works nicely with AWS managed services: Amazon S3, Amazon DynamoDB, etc. • Build event-driven applicaJons • Build RESTful APIs in conjuncJon with Amazon API Gateway • Pay as you go: number of requests + execuJon Jme (100ms slots)
- 5. Another way to put it… Tim Wagner, General Manager, AWS Lambda Serverless conference, NYC, May 2016
- 6. MOBILE CHAT APP AD DATA ANALYTICS AND ROUTING MOBILE APP ANALYTICS IMAGE CONTENT FILTERING REAL-TIME VIDEO AD BIDDING NEWS CONTENT PROCESSING GENE SEQUENCE SEARCH CLOUD TELEPHONY DATA PROCESSING WEB APPLICATIONS WEB APPLICATIONS THREAT INTELLIGENCE AND ANALYTICS NEWS CONTENT PROCESSING GAME METRICS ANALYTICS Selected serverless customers PRODUCT RECOMMANDATION
- 9. AWS Lambda ‘Hello World’ (Python) 1. Write a simple Lambda function in Python 2. Create a REST API with API Gateway (resource + POST method) 3. Deploy the API 4. Invoke the API with ‘curl’
- 10. A simple Lambda function in Python def lambda_handler(event,context): result = event['value1'] + event['value2'] return result aws lambda create-function --function-name myFunc --handler myFunc.lambda_handler --runtime python2.7 --zip-file fileb://myFunc.zip --memory-size 128 --role arn:aws:iam::ACCOUNT_NUMBER:role/lambda_basic_execution curl -H "Content-Type: application/json" -X POST -d "{"value1":5, "value2":7}" https://API_ENDPOINT/STAGE/RESOURCE 12
- 11. AWS Lambda in Java with Eclipse https://java.awsblog.com/post/TxWZES6J1RSQ2Z/Testing-Lambda-functions-using-the-AWS-Toolkit-for-Eclipse
- 12. AWS Lambda ‘Hello World’ (Java) 1. In Eclipse, write a simple Lambda function triggered by an S3 event 2. Unit-test the function with Junit 3. Using the AWS Eclipse plug-in, upload and run the function in AWS 4. Run the function again in the AWS Console
- 13. AWS Lambda with the Serverless framework http://github.com/serverless/serverless • Run/test AWS Lambda functions locally, or remotely • Auto-deploys & versions your Lambda functions • Auto-deploys your REST API to AWS API Gateway • Auto-deploys your Lambda events • Support for multiple stages • Support for multiple regions within stages • Manage & deploy AWS CloudFormation resources
- 14. Building a serverless data pipeline Lambda DynamoDB Kinesis Firehose API Gateway HTTP POST /prod/logger writeTo Kinesis DynamoDB ToFirehose S3 eventTable DynamoDB streams bucket EMR, Redshift, … firehoseToS3 Kinesis StreamsLambda Lambda KinesisTo DynamoDB Web apps
- 15. Step 1: create DynamoDB table aws dynamodb create-table --table-name eventTable --attribute-definitions AttributeName=userId,AttributeType=N AttributeName=timestamp,AttributeType=N --key-schema AttributeName=userId,KeyType=HASH AttributeName=timestamp,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 --stream-specification StreamEnabled=true,StreamViewType=NEW_IMAGE eventTable DynamoDB
- 16. Step 2: IAM role for Lambda function aws iam create-role --role-name writeToDynamoDB_role --assume-role-policy-document file://lambda_trust_policy.json aws iam create-policy --policy-name writeToDynamoDB_policy --policy-document file://writeToDynamoDB_policy.json aws iam attach-role-policy --role-name writeToDynamoDB_role --policy-arn WRITETODYNAMODB_POLICY_ARN
- 17. Step 3: create Lambda function S3 DynamoDB eventTable Web apps dynamodb. put_item() Lambda aws lambda create-function --function-name writeToDynamoDB --role WRITETODYNAMO_DB_ROLE --zip-file fileb://writeToDynamoDB.zip --handler writeToDynamoDB.lambda_handler --runtime python2.7 --memory-size 128 --description "Write events to DynamoDB”
- 18. Step 4: create Kinesis Stream DynamoDB S3 eventTable Web apps dynamodb. put_item() Lambda aws kinesis create-stream --stream-name APItoDynamoDB --shard-count 1 Kinesis Streams
- 19. Step 5: IAM role for Lambda function aws iam create-role --role-name writeToKinesis_role --assume-role-policy-document file://lambda_trust_policy.json aws iam create-policy --policy-name writeToKinesis_policy --policy-document file://writeToKinesis_policy.json aws iam attach-role-policy --role-name writeToKinesis_role --policy-arn WRITETOKINESIS_POLICY_ARN
- 20. Step 6: create Lambda function DynamoDB S3 eventTable Web apps DynamoDB streams dynamodb. put_item() Lambda aws lambda create-function --function-name writeToKinesis --role WRITETOKINESIS_ROLE --zip-file fileb://writeToKinesis.zip --handler writeToKinesis.lambda_handler --runtime python2.7 --memory-size 128 --description "Write events to Kinesis” Kinesis StreamsLambda KinesisTo DynamoDB writeTo Kinesis
- 21. Step 7: create API DynamoDB S3 eventTable Web apps DynamoDB streams dynamodb. put_item() Lambda Painful to do with the CLI: 9 aws apigateway calls :-/ à Use the console à Use a Swagger File http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-import-api.html à Use the Serverless framework Kinesis StreamsLambda KinesisTo DynamoDB writeTo Kinesis API Gateway
- 22. Step 8: create IAM role aws iam create-role --role-name DynamoDBToFirehose_role --assume-role-policy-document file://lambda_trust_policy.json aws iam create-policy --policy-name DynamoDBToFirehose_policy --policy-document file://DynamoDBToFirehose_policy.json aws iam attach-role-policy --role-name DynamoDBToFirehose_role --policy-arn DYNAMODBTOFIREHOSE_POLICY_ARN
- 23. Step 9: create Lambda funcIon and DynamoDB trigger aws lambda create-function --function-name DynamoDBToFirehose --role DYNAMODBTOFIREHOSE_ROLE_ARN --zip-file fileb://DynamoDBToFirehose.zip --handler DynamoDBToFirehose.lambda_handler --runtime python2.7 --memory-size 128 --description "Write DynamoDB stream to Kinesis Firehose" aws lambda create-event-source-mapping --function-name DynamoDBToFirehose --event-source DYNAMODB_STREAM_ARN --batch-size 10 --starting-position TRIM_HORIZON Lambda DynamoDB ToFirehose eventTable DynamoDB streams DynamoDB
- 24. Step 10: create IAM role aws iam create-role --role-name firehoseToS3_role --assume-role-policy-document file://firehose_trust_policy.json aws iam create-policy --policy-name firehoseToS3_policy --policy-document file://firehoseToS3_policy.json aws iam attach-role-policy --role-name firehoseToS3_role --policy-arn FIREHOSETOS3_POLICY_ARN
- 25. Step 11: create S3 bucket aws s3 mb s3://jsimon-public Lambda DynamoDB ToFirehose eventTable DynamoDB streams DynamoDB
- 26. Step 12: create Kinesis Firehose stream aws firehose create-delivery-stream --delivery-stream-name firehoseToS3 --s3-destination-configuration RoleARN=FIREHOSETOS3_ROLE_ARN, BucketARN="arn:aws:s3:::jsimon-public", Prefix="firehose", BufferingHints={SizeInMBs=1,IntervalInSeconds=60}, CompressionFormat="GZIP", EncryptionConfiguration={NoEncryptionConfig="NoEncryption"} Kinesis Firehose firehoseToS3 Lambda DynamoDB ToFirehose eventTable DynamoDB streams DynamoDB
- 27. Building a serverless data pipeline Lambda DynamoDB Kinesis Firehose API Gateway HTTP POST /prod/logger writeTo Kinesis DynamoDB ToFirehose S3 eventTable DynamoDB streams bucket EMR, Redshift, … firehoseToS3 Kinesis StreamsLambda Lambda KinesisTo DynamoDB Web apps Lines of code: 16 Number of servers: zero Performance & scalability: maximum https://github.com/juliensimon/aws/tree/master/serverlessPipeline
- 30. Going further AWS re:Invent 2014 | (MBL202) NEW LAUNCH: Getting Started with AWS Lambda https://www.youtube.com/watch?v=UFj27laTWQA AWS re:Invent 2015 | (DEV203) Amazon API Gateway & AWS Lambda to Build Secure and Scalable APIs https://www.youtube.com/watch?v=ZBxWZ9bgd44 AWS re:Invent 2015 | (DVO209) JAWS: The Monstrously Scalable Serverless Framework https://www.youtube.com/watch?v=D_U6luQ6I90 https://github.com/serverless/serverless AWS re:Invent 2015 | (ARC308) The Serverless Company Using AWS Lambda https://www.youtube.com/watch?v=U8ODkSCJpJU AWS re:Invent 2015 | (CMP407) Lambda as Cron: Scheduling Invocations in AWS Lambda https://www.youtube.com/watch?v=FhJxTIq81AU Reference architectures http://www.allthingsdistributed.com/2016/06/aws-lambda-serverless-reference-architectures.html
- 32. AWS Enterprise Summit – 27/10/2016, Paris http://amzn.to/1X2yp0i