SlideShare a Scribd company logo

Real-World Smart Applications with Amazon Machine Learning - AWS Machine Learning Web Day

AWS Germany
AWS Germany

The document outlines the integration of Amazon Machine Learning with social media listening applications to identify and prioritize customer service interactions. It describes the end-to-end application process, including the preparation of data, model creation, and use of AWS services such as Kinesis and Lambda. Additionally, it highlights machine learning strategies for classifying tweets to enhance customer service response efficiency.

Technology
1 of 66
Downloaded 26 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Alex Ingerman Sr. Manager, Tech. Product Management, Amazon Machine Learning 2/25/2016 Real-World Smart Applications with Amazon Machine Learning
Agenda • Why social media + machine learning = happy customers • Using Amazon ML to find important social media conversations • Building an end-to-end application to act on these conversations
Application details Goal: build a smart application for social media listening in the cloud Full source code and documentation are on GitHub: http://bit.ly/AmazonMLCodeSample Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS Amazon Mechanical Turk
Motivation for listening to social media Customer is reporting a possible service issue
Motivation for listening to social media Customer is making a feature request
Motivation for listening to social media Customer is angry or unhappy
Motivation for listening to social media Customer is asking a question
Why do we need machine learning for this? The social media stream is high-volume, and most of the messages are not CS-actionable
Amazon Machine Learning in one slide • Easy to use, managed machine learning service built for developers • Robust, powerful machine learning technology based on Amazon’s internal systems • Create models using your data already stored in the AWS cloud • Deploy models to production in seconds
Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents.
Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API
Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API Amazon Kinesis
Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API Amazon Kinesis AWS Lambda
Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning
Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
Picking the machine learning strategy Question we want to answer: Is this tweet customer service-actionable, or not? Our dataset: Text and metadata from past tweets mentioning @awscloud Machine learning approach: Create a binary classification model to answer a yes/no question, and provide a confidence score
Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
Retrieve past tweets Twitter API can be used to search for tweets containing our company’s handle (e.g., @awscloud) import twitter twitter_api = twitter.Api(**twitter_credentials) twitter_handle = ‘awscloud’ search_query = '@' + twitter_handle + ' -from:' + twitter_handle results = twitter_api.GetSearch(term=search_query, count=100, result_type='recent’) # We can go further back in time by issuing additional search requests
Retrieve past tweets Twitter API can be used to search for tweets containing our company’s handle (e.g., @awscloud) import twitter twitter_api = twitter.Api(**twitter_credentials) twitter_handle = ‘awscloud’ search_query = '@' + twitter_handle + ' -from:' + twitter_handle results = twitter_api.GetSearch(term=search_query, count=100, result_type='recent') # We can go further back in time by issuing additional search requests Good news: data is well-structured and clean Bad news: tweets are not categorized (labeled) for us
Labeling past tweets Why label tweets? (Many) machine learning algorithms work by discovering patterns connecting data points and labels How many tweets need to be labeled? Several thousands to start with Can I pay someone to do this? Yes! Amazon Mechanical Turk is a marketplace for tasks that require human intelligence
Creating the Mechanical Turk task
Creating the Mechanical Turk task
Creating the Mechanical Turk task
Creating the Mechanical Turk task
Creating the Mechanical Turk task
Publishing the task
Publishing the task
Preview labeling results Sample tweets from our previously collected dataset + their labels This column was created from Mechanical Turk responses
Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
Amazon ML process, in a nutshell 1. Create your datasources Two API calls to create your training and evaluation data Sanity-check your data in service console 2. Create your ML model One API call to build a model, with smart default or custom setting 3. Evaluate your ML model One API call to compute your model’s quality metric 4. Adjust your ML model Use console to align performance trade-offs to your business goals
Create the data schema string { "dataFileContainsHeader": true, "dataFormat": "CSV", "targetAttributeName": "trainingLabel", "attributes": [ { "attributeName": "description", "attributeType": "TEXT" }, <additional attributes here>, { "attributeName": "trainingLabel", "attributeType": "BINARY" } ] } Schemas communicate metadata about your dataset: • Data format • Attributes’ names, types, and order • Names of special attributes
Create the training datasource import boto ml = boto.connect_machinelearning() data_spec = { 'DataLocationS3’ : s3_uri # E.g.: s3://my-bucket/dir/data.csv 'DataSchema’ : data_schema } # Schema string (previous slide) # Use only the first 70% of the datasource for training. data_spec['DataRearrangement'] = ‘{ "splitting”: {"percentBegin": 0, "percentEnd”: 70 } }’ ml.create_data_source_from_s3( data_source_id = “ds-tweets-train”, data_source_name = “Tweet training data (70%)”, data_spec, compute_statistics = True)
Create the evaluation datasource import boto ml = boto.connect_machinelearning() data_spec = { 'DataLocationS3’ : s3_uri # E.g.: s3://my-bucket/dir/data.csv 'DataSchema’ : data_schema } # Schema string (previous slide) # Use the last 30% of the datasource for evaluation. data_spec['DataRearrangement'] = ‘{ "splitting”: {"percentBegin": 70, "percentEnd”: 100 } }’ ml.create_data_source_from_s3( data_source_id = “ds-tweets-eval”, data_source_name = “Tweet evaluation data (30%)”, data_spec, compute_statistics = True)
Visually inspecting training data
Create the ML model import boto ml = boto.connect_machinelearning() ml.create_ml_model( ml_model_id = “ml-tweets”, ml_model_name = “Tweets screening model”, ml_model_type = “BINARY”, training_data_source_id = “ds-tweets-train”) Input data location is looked up from the training datasource ID Default model parameters and automatic data transformations are used, or you can provide your own
Evaluate the ML model import boto ml = boto.connect_machinelearning() ml.create_evaluation( evaluation_id = “ev-tweets”, evaluation_name = “Evaluation of tweet screening model”, ml_model_id = “ml-tweets”, evaluation_data_source_id = “ds-tweets-eval”) Input data location is looked up from the evaluation datasource ID Amazon ML automatically selects and computes an industry-standard evaluation metric based on your ML model type
Visually inspecting and adjusting the ML model
Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
Reminder: Our data flow Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
Create an Amazon ML endpoint for retrieving real- time predictions import boto ml = boto.connect_machinelearning() ml.create_realtime_endpoint(“ml-tweets”) # Endpoint information can be retrieved using the get_ml_model() method. Sample output: #"EndpointInfo": { # "CreatedAt": 1424378682.266, # "EndpointStatus": "READY", # "EndpointUrl": ”https://realtime.machinelearning.us-east-1.amazonaws.com", # "PeakRequestsPerSecond": 200} Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
Create an Amazon Kinesis stream for receiving tweets import boto kinesis = boto.connect_kinesis() kinesis.create_stream(stream_name = ‘tweetStream’, shard_count = 1) # Each open shard can support up to 5 read transactions per second, up to a # maximum total of 2 MB of data read per second. Each shard can support up to # 1000 records written per second, up to a maximum total of 1 MB data written # per second. Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
Set up AWS Lambda to coordinate the data flow The Lambda function is our application’s backbone. We will: 1. Write the code that will process and route tweets 2. Configure the Lambda execution policy (what is it allowed to do?) 3. Add the Kinesis stream as the data source for the Lambda function Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
Create Lambda functions Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS // These are our function’s signatures and globals only. See GitHub repository for full source. var ml = new AWS.MachineLearning(); var endpointUrl = ''; var mlModelId = ’ml-tweets'; var snsTopicArn = 'arn:aws:sns:{region}:{awsAccountId}:{snsTopic}'; var snsMessageSubject = 'Respond to tweet'; var snsMessagePrefix = 'ML model '+mlModelId+': Respond to this tweet: https://twitter.com/0/status/'; var processRecords = function() {…} // Base64 decode the Kinesis payload and parse JSON var callPredict = function(tweetData) {…} // Call Amazon ML real-time prediction API var updateSns = function(tweetData) {…} // Publish CS-actionable tweets to SNS topic var checkRealtimeEndpoint = function(err, data) {…} // Get Amazon ML endpoint URI
Create Lambda functions Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS // These are our function’s signatures and globals only. See GitHub repository for full source. var ml = new AWS.MachineLearning(); var endpointUrl = ''; var mlModelId = ’ml-tweets'; var snsTopicArn = 'arn:aws:sns:{region}:{awsAccountId}:{snsTopic}'; var snsMessageSubject = 'Respond to tweet'; var snsMessagePrefix = 'ML model '+mlModelId+': Respond to this tweet: https://twitter.com/0/status/'; var processRecords = function() {…} // Base64 decode the Kinesis payload and parse JSON var callPredict = function(tweetData) {…} // Call Amazon ML real-time prediction API var updateSns = function(tweetData) {…} // Publish CS-actionable tweets to SNS topic var checkRealtimeEndpoint = function(err, data) {…} // Get Amazon ML endpoint URI
Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] }
Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] } Allow request logging in Amazon CloudWatch
Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] } Allow publication of notifications to SNS topic
Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] } Allow calls to Amazon ML real-time prediction APIs
Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] } Allow reading of data from Kinesis stream
Connect Kinesis stream and Lambda function import boto aws_lambda = boto.connect_awslambda() aws_lambda.add_event_source( event_source = 'arn:aws:kinesis:' + region + ':' + aws_account_id + ':stream/' + “tweetStream”, function_name = “process_tweets”, role = 'arn:aws:iam::' + aws_account_id + ':role/' + lambda_execution_role) Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
Amazon ML real-time predictions test Here is a tweet:
Amazon ML real-time predictions test Here is the same tweet…as a JSON blob: { "statuses_count": "8617", "description": "Software Developer", "friends_count": "96", "text": "`scala-aws-s3` A Simple Amazon #S3 Wrapper for #Scala 1.10.20 available : https://t.co/q76PLTovFg", "verified": "False", "geo_enabled": "True", "uid": "3800711", "favourites_count": "36", "screen_name": "turutosiya", "followers_count": "640", "user.name": "Toshiya TSURU", "sid": "647222291672100864" }
Amazon ML real-time predictions test Let’s use the AWS Command Line Interface to request a prediction for this tweet: aws machinelearning predict --predict-endpoint https://realtime.machinelearning.us-east-1.amazonaws.com --ml-model-id ml-tweets --record ‘<json_blob>’
Amazon ML real-time predictions test Let’s use the AWS Command Line Interface to request a prediction for this tweet: aws machinelearning predict --predict-endpoint https://realtime.machinelearning.us-east-1.amazonaws.com --ml-model-id ml-tweets --record ‘<json_blob>’ { "Prediction": { "predictedLabel": "0", "predictedScores": { "0": 0.012336540967226028 }, "details": { "PredictiveModelType": "BINARY", "Algorithm": "SGD" } } }
Recap: Our application’s data flow Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
End-to-end application demo
Generalizing to more feedback channels Amazon Kinesis AWS Lambda Model 1 Amazon SNS Model 2 Model 3
What’s next? Try the service: http://aws.amazon.com/machine-learning/ Download the Social Media Listening application code: http://bit.ly/AmazonMLCodeSample Get in touch! ingerman@amazon.com
Thank you!

More Related Content

Viewers also liked (16)

PPTX
Medical Conditions2
Robert Strachan
 
PDF
Reference - Dirk van Schalkwyk
Pieter Duvenhage
 
PDF
20141212165557913
Milind Tiwari, CFE
 
PPT
Gry planszowe w edukacji, o doświadczeniach z Trzęsienia Danych, Avangarda X
Kamil Śliwowski
 
PDF
Policy matchmaking
kauffmanfdn
 
PDF
Designed Art
edvinas280
 
PDF
Community Leaders Support Omar
Peter Ortiz
 
DOCX
Busia 1
Jerusa Achieno
 
PDF
HCL Recognition
KRISHNAKUMAR TAMILSELVAN
 
PDF
Security Boundaries and Functions of Services for Serverless Architectures on...
AWS Germany
 
PPTX
SOAP2015 - Key Challenges in Global Content Development
Piotr Peszko
 
PPTX
Can you afford not to consider sustainable ICT?
Nordic Digital Business Summit
 
PPTX
IoT Security, Threats and Challenges By V.P.Prabhakaran
Koenig Solutions Ltd.
 
PPTX
Basic Computer Architecture
Yong Heui Cho
 
PDF
500 Kobe Pre-Accelerator Demo Day >> Neil Dugal of 500 Startups
500 Startups
 
KEY
Big Data in Real-Time at Twitter
nkallen
 
Medical Conditions2
Robert Strachan
 
Reference - Dirk van Schalkwyk
Pieter Duvenhage
 
20141212165557913
Milind Tiwari, CFE
 
Gry planszowe w edukacji, o doświadczeniach z Trzęsienia Danych, Avangarda X
Kamil Śliwowski
 
Policy matchmaking
kauffmanfdn
 
Designed Art
edvinas280
 
Community Leaders Support Omar
Peter Ortiz
 
Busia 1
Jerusa Achieno
 
HCL Recognition
KRISHNAKUMAR TAMILSELVAN
 
Security Boundaries and Functions of Services for Serverless Architectures on...
AWS Germany
 
SOAP2015 - Key Challenges in Global Content Development
Piotr Peszko
 
Can you afford not to consider sustainable ICT?
Nordic Digital Business Summit
 
IoT Security, Threats and Challenges By V.P.Prabhakaran
Koenig Solutions Ltd.
 
Basic Computer Architecture
Yong Heui Cho
 
500 Kobe Pre-Accelerator Demo Day >> Neil Dugal of 500 Startups
500 Startups
 
Big Data in Real-Time at Twitter
nkallen
 

Similar to Real-World Smart Applications with Amazon Machine Learning - AWS Machine Learning Web Day (20)

PDF
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
PAPIs.io
 
PDF
Machine Learning for Developers
Danilo Poccia
 
PPTX
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
HostedbyConfluent
 
PDF
Unleash the Power of ML with AWS | AWS Summit Tel Aviv 2019
AWS Summits
 
PPTX
WhereML a Serverless ML Powered Location Guessing Twitter Bot
Randall Hunt
 
PPTX
Presentazione tutorial
dariospin93
 
PPTX
Where ml ai_heavy
Randall Hunt
 
PDF
Machine Learning for everyone
Julien SIMON
 
PPTX
Building Machine Learning Inference Pipelines at Scale (July 2019)
Julien SIMON
 
PDF
AI Services for Developers - Floor28
Boaz Ziniman
 
PDF
MLops workshop AWS
Gili Nachum
 
PPTX
Big Data and Machine Learning on AWS
CloudHesive
 
PDF
엔터프라이즈를 위한 머신러닝 그리고 AWS (김일호 솔루션즈 아키텍트, AWS) :: AWS Techforum 2018
Amazon Web Services Korea
 
PPTX
Intro to SageMaker
Soji Adeshina
 
PPTX
AWS re:Invent 2018 - Machine Learning recap (December 2018)
Julien SIMON
 
PDF
AWS Summit Singapore 2019 | Build, Train and Deploy Deep Learning Models on A...
AWS Summits
 
PDF
AI & Machine Learning at AWS - An Introduction
Daniel Zivkovic
 
PDF
AWS reinvent 2019 recap - Riyadh - AI And ML - Ahmed Raafat
AWS Riyadh User Group
 
PDF
Zenyk Matchyshyn "Doing Machine Learning on Amazon Stack"
Lviv Startup Club
 
PDF
Artificial intelligence in actions: delivering a new experience to Formula 1 ...
GoDataDriven
 
Building a Production-ready Predictive App for Customer Service - Alex Ingerm...
PAPIs.io
 
Machine Learning for Developers
Danilo Poccia
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
HostedbyConfluent
 
Unleash the Power of ML with AWS | AWS Summit Tel Aviv 2019
AWS Summits
 
WhereML a Serverless ML Powered Location Guessing Twitter Bot
Randall Hunt
 
Presentazione tutorial
dariospin93
 
Where ml ai_heavy
Randall Hunt
 
Machine Learning for everyone
Julien SIMON
 
Building Machine Learning Inference Pipelines at Scale (July 2019)
Julien SIMON
 
AI Services for Developers - Floor28
Boaz Ziniman
 
MLops workshop AWS
Gili Nachum
 
Big Data and Machine Learning on AWS
CloudHesive
 
엔터프라이즈를 위한 머신러닝 그리고 AWS (김일호 솔루션즈 아키텍트, AWS) :: AWS Techforum 2018
Amazon Web Services Korea
 
Intro to SageMaker
Soji Adeshina
 
AWS re:Invent 2018 - Machine Learning recap (December 2018)
Julien SIMON
 
AWS Summit Singapore 2019 | Build, Train and Deploy Deep Learning Models on A...
AWS Summits
 
AI & Machine Learning at AWS - An Introduction
Daniel Zivkovic
 
AWS reinvent 2019 recap - Riyadh - AI And ML - Ahmed Raafat
AWS Riyadh User Group
 
Zenyk Matchyshyn "Doing Machine Learning on Amazon Stack"
Lviv Startup Club
 
Artificial intelligence in actions: delivering a new experience to Formula 1 ...
GoDataDriven
 
Ad

More from AWS Germany (20)

PDF
Analytics Web Day | From Theory to Practice: Big Data Stories from the Field
AWS Germany
 
PDF
Analytics Web Day | Query your Data in S3 with SQL and optimize for Cost and ...
AWS Germany
 
PDF
Modern Applications Web Day | Impress Your Friends with Your First Serverless...
AWS Germany
 
PDF
Modern Applications Web Day | Manage Your Infrastructure and Configuration on...
AWS Germany
 
PDF
Modern Applications Web Day | Container Workloads on AWS
AWS Germany
 
PDF
Modern Applications Web Day | Continuous Delivery to Amazon EKS with Spinnaker
AWS Germany
 
PDF
Building Smart Home skills for Alexa
AWS Germany
 
PDF
Hotel or Taxi? "Sorting hat" for travel expenses with AWS ML infrastructure
AWS Germany
 
PDF
Wild Rydes with Big Data/Kinesis focus: AWS Serverless Workshop
AWS Germany
 
PDF
Log Analytics with AWS
AWS Germany
 
PDF
Deep Dive into Concepts and Tools for Analyzing Streaming Data on AWS
AWS Germany
 
PDF
AWS Programme für Nonprofits
AWS Germany
 
PDF
Microservices and Data Design
AWS Germany
 
PDF
Serverless vs. Developers – the real crash
AWS Germany
 
PDF
Query your data in S3 with SQL and optimize for cost and performance
AWS Germany
 
PDF
Secret Management with Hashicorp’s Vault
AWS Germany
 
PDF
EKS Workshop
AWS Germany
 
PDF
Scale to Infinity with ECS
AWS Germany
 
PDF
Containers on AWS - State of the Union
AWS Germany
 
PDF
Deploying and Scaling Your First Cloud Application with Amazon Lightsail
AWS Germany
 
Analytics Web Day | From Theory to Practice: Big Data Stories from the Field
AWS Germany
 
Analytics Web Day | Query your Data in S3 with SQL and optimize for Cost and ...
AWS Germany
 
Modern Applications Web Day | Impress Your Friends with Your First Serverless...
AWS Germany
 
Modern Applications Web Day | Manage Your Infrastructure and Configuration on...
AWS Germany
 
Modern Applications Web Day | Container Workloads on AWS
AWS Germany
 
Modern Applications Web Day | Continuous Delivery to Amazon EKS with Spinnaker
AWS Germany
 
Building Smart Home skills for Alexa
AWS Germany
 
Hotel or Taxi? "Sorting hat" for travel expenses with AWS ML infrastructure
AWS Germany
 
Wild Rydes with Big Data/Kinesis focus: AWS Serverless Workshop
AWS Germany
 
Log Analytics with AWS
AWS Germany
 
Deep Dive into Concepts and Tools for Analyzing Streaming Data on AWS
AWS Germany
 
AWS Programme für Nonprofits
AWS Germany
 
Microservices and Data Design
AWS Germany
 
Serverless vs. Developers – the real crash
AWS Germany
 
Query your data in S3 with SQL and optimize for cost and performance
AWS Germany
 
Secret Management with Hashicorp’s Vault
AWS Germany
 
EKS Workshop
AWS Germany
 
Scale to Infinity with ECS
AWS Germany
 
Containers on AWS - State of the Union
AWS Germany
 
Deploying and Scaling Your First Cloud Application with Amazon Lightsail
AWS Germany
 
Ad

Recently uploaded (20)

PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PDF
IoT-Powered Industrial Transformation – Smart Manufacturing to Connected Heal...
Rejig Digital
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
PDF
Biography of Daniel Podor.pdf
Daniel Podor
 
PDF
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
PDF
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
Jak MŚP w Europie Środkowo-Wschodniej odnajdują się w świecie AI
dominikamizerska1
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PPTX
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
PDF
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
PDF
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PDF
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
PDF
What Makes Contify’s News API Stand Out: Key Features at a Glance
Contify
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
IoT-Powered Industrial Transformation – Smart Manufacturing to Connected Heal...
Rejig Digital
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
Biography of Daniel Podor.pdf
Daniel Podor
 
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
Jak MŚP w Europie Środkowo-Wschodniej odnajdują się w świecie AI
dominikamizerska1
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
POV_ Why Enterprises Need to Find Value in ZERO.pdf
darshakparmar
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
What Makes Contify’s News API Stand Out: Key Features at a Glance
Contify
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 

Real-World Smart Applications with Amazon Machine Learning - AWS Machine Learning Web Day

  • 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Alex Ingerman Sr. Manager, Tech. Product Management, Amazon Machine Learning 2/25/2016 Real-World Smart Applications with Amazon Machine Learning
  • 2. Agenda • Why social media + machine learning = happy customers • Using Amazon ML to find important social media conversations • Building an end-to-end application to act on these conversations
  • 3. Application details Goal: build a smart application for social media listening in the cloud Full source code and documentation are on GitHub: http://bit.ly/AmazonMLCodeSample Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS Amazon Mechanical Turk
  • 4. Motivation for listening to social media Customer is reporting a possible service issue
  • 5. Motivation for listening to social media Customer is making a feature request
  • 6. Motivation for listening to social media Customer is angry or unhappy
  • 7. Motivation for listening to social media Customer is asking a question
  • 8. Why do we need machine learning for this? The social media stream is high-volume, and most of the messages are not CS-actionable
  • 9. Amazon Machine Learning in one slide • Easy to use, managed machine learning service built for developers • Robust, powerful machine learning technology based on Amazon’s internal systems • Create models using your data already stored in the AWS cloud • Deploy models to production in seconds
  • 10. Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents.
  • 11. Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API
  • 12. Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API Amazon Kinesis
  • 13. Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API Amazon Kinesis AWS Lambda
  • 14. Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning
  • 15. Formulating the problem We would like to… Instantly find new tweets mentioning @awscloud, ingest and analyze each one to predict whether a customer service agent should act on it, and, if so, send that tweet to customer service agents. Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
  • 16. Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
  • 17. Picking the machine learning strategy Question we want to answer: Is this tweet customer service-actionable, or not? Our dataset: Text and metadata from past tweets mentioning @awscloud Machine learning approach: Create a binary classification model to answer a yes/no question, and provide a confidence score
  • 18. Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
  • 19. Retrieve past tweets Twitter API can be used to search for tweets containing our company’s handle (e.g., @awscloud) import twitter twitter_api = twitter.Api(**twitter_credentials) twitter_handle = ‘awscloud’ search_query = '@' + twitter_handle + ' -from:' + twitter_handle results = twitter_api.GetSearch(term=search_query, count=100, result_type='recent’) # We can go further back in time by issuing additional search requests
  • 20. Retrieve past tweets Twitter API can be used to search for tweets containing our company’s handle (e.g., @awscloud) import twitter twitter_api = twitter.Api(**twitter_credentials) twitter_handle = ‘awscloud’ search_query = '@' + twitter_handle + ' -from:' + twitter_handle results = twitter_api.GetSearch(term=search_query, count=100, result_type='recent') # We can go further back in time by issuing additional search requests Good news: data is well-structured and clean Bad news: tweets are not categorized (labeled) for us
  • 21. Labeling past tweets Why label tweets? (Many) machine learning algorithms work by discovering patterns connecting data points and labels How many tweets need to be labeled? Several thousands to start with Can I pay someone to do this? Yes! Amazon Mechanical Turk is a marketplace for tasks that require human intelligence
  • 29. Preview labeling results Sample tweets from our previously collected dataset + their labels This column was created from Mechanical Turk responses
  • 30. Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
  • 31. Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
  • 32. Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
  • 33. Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
  • 34. Preview labeling results Sample tweets and labels (most metadata fields removed for clarity)
  • 35. Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
  • 36. Amazon ML process, in a nutshell 1. Create your datasources Two API calls to create your training and evaluation data Sanity-check your data in service console 2. Create your ML model One API call to build a model, with smart default or custom setting 3. Evaluate your ML model One API call to compute your model’s quality metric 4. Adjust your ML model Use console to align performance trade-offs to your business goals
  • 37. Create the data schema string { "dataFileContainsHeader": true, "dataFormat": "CSV", "targetAttributeName": "trainingLabel", "attributes": [ { "attributeName": "description", "attributeType": "TEXT" }, <additional attributes here>, { "attributeName": "trainingLabel", "attributeType": "BINARY" } ] } Schemas communicate metadata about your dataset: • Data format • Attributes’ names, types, and order • Names of special attributes
  • 38. Create the training datasource import boto ml = boto.connect_machinelearning() data_spec = { 'DataLocationS3’ : s3_uri # E.g.: s3://my-bucket/dir/data.csv 'DataSchema’ : data_schema } # Schema string (previous slide) # Use only the first 70% of the datasource for training. data_spec['DataRearrangement'] = ‘{ "splitting”: {"percentBegin": 0, "percentEnd”: 70 } }’ ml.create_data_source_from_s3( data_source_id = “ds-tweets-train”, data_source_name = “Tweet training data (70%)”, data_spec, compute_statistics = True)
  • 39. Create the evaluation datasource import boto ml = boto.connect_machinelearning() data_spec = { 'DataLocationS3’ : s3_uri # E.g.: s3://my-bucket/dir/data.csv 'DataSchema’ : data_schema } # Schema string (previous slide) # Use the last 30% of the datasource for evaluation. data_spec['DataRearrangement'] = ‘{ "splitting”: {"percentBegin": 70, "percentEnd”: 100 } }’ ml.create_data_source_from_s3( data_source_id = “ds-tweets-eval”, data_source_name = “Tweet evaluation data (30%)”, data_spec, compute_statistics = True)
  • 41. Create the ML model import boto ml = boto.connect_machinelearning() ml.create_ml_model( ml_model_id = “ml-tweets”, ml_model_name = “Tweets screening model”, ml_model_type = “BINARY”, training_data_source_id = “ds-tweets-train”) Input data location is looked up from the training datasource ID Default model parameters and automatic data transformations are used, or you can provide your own
  • 42. Evaluate the ML model import boto ml = boto.connect_machinelearning() ml.create_evaluation( evaluation_id = “ev-tweets”, evaluation_name = “Evaluation of tweet screening model”, ml_model_id = “ml-tweets”, evaluation_data_source_id = “ds-tweets-eval”) Input data location is looked up from the evaluation datasource ID Amazon ML automatically selects and computes an industry-standard evaluation metric based on your ML model type
  • 43. Visually inspecting and adjusting the ML model
  • 44. Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
  • 45. Reminder: Our data flow Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
  • 46. Create an Amazon ML endpoint for retrieving real- time predictions import boto ml = boto.connect_machinelearning() ml.create_realtime_endpoint(“ml-tweets”) # Endpoint information can be retrieved using the get_ml_model() method. Sample output: #"EndpointInfo": { # "CreatedAt": 1424378682.266, # "EndpointStatus": "READY", # "EndpointUrl": ”https://realtime.machinelearning.us-east-1.amazonaws.com", # "PeakRequestsPerSecond": 200} Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
  • 47. Create an Amazon Kinesis stream for receiving tweets import boto kinesis = boto.connect_kinesis() kinesis.create_stream(stream_name = ‘tweetStream’, shard_count = 1) # Each open shard can support up to 5 read transactions per second, up to a # maximum total of 2 MB of data read per second. Each shard can support up to # 1000 records written per second, up to a maximum total of 1 MB data written # per second. Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
  • 48. Set up AWS Lambda to coordinate the data flow The Lambda function is our application’s backbone. We will: 1. Write the code that will process and route tweets 2. Configure the Lambda execution policy (what is it allowed to do?) 3. Add the Kinesis stream as the data source for the Lambda function Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
  • 49. Create Lambda functions Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS // These are our function’s signatures and globals only. See GitHub repository for full source. var ml = new AWS.MachineLearning(); var endpointUrl = ''; var mlModelId = ’ml-tweets'; var snsTopicArn = 'arn:aws:sns:{region}:{awsAccountId}:{snsTopic}'; var snsMessageSubject = 'Respond to tweet'; var snsMessagePrefix = 'ML model '+mlModelId+': Respond to this tweet: https://twitter.com/0/status/'; var processRecords = function() {…} // Base64 decode the Kinesis payload and parse JSON var callPredict = function(tweetData) {…} // Call Amazon ML real-time prediction API var updateSns = function(tweetData) {…} // Publish CS-actionable tweets to SNS topic var checkRealtimeEndpoint = function(err, data) {…} // Get Amazon ML endpoint URI
  • 50. Create Lambda functions Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS // These are our function’s signatures and globals only. See GitHub repository for full source. var ml = new AWS.MachineLearning(); var endpointUrl = ''; var mlModelId = ’ml-tweets'; var snsTopicArn = 'arn:aws:sns:{region}:{awsAccountId}:{snsTopic}'; var snsMessageSubject = 'Respond to tweet'; var snsMessagePrefix = 'ML model '+mlModelId+': Respond to this tweet: https://twitter.com/0/status/'; var processRecords = function() {…} // Base64 decode the Kinesis payload and parse JSON var callPredict = function(tweetData) {…} // Call Amazon ML real-time prediction API var updateSns = function(tweetData) {…} // Publish CS-actionable tweets to SNS topic var checkRealtimeEndpoint = function(err, data) {…} // Get Amazon ML endpoint URI
  • 51. Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] }
  • 52. Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] } Allow request logging in Amazon CloudWatch
  • 53. Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] } Allow publication of notifications to SNS topic
  • 54. Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] } Allow calls to Amazon ML real-time prediction APIs
  • 55. Configure Lambda execution policy Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS { "Statement": [ { "Action": [ "logs:*” ], "Effect": "Allow", "Resource": "arn:aws:logs:{region}:{awsAccountId}:log-group:/aws/lambda/{lambdaFunctionName}:*" }, { "Action": [ "sns:publish” ], "Effect": "Allow", "Resource": "arn:aws:sns:{region}:{awsAccountId}:{snsTopic}" }, { "Action": [ "machinelearning:GetMLModel”, "machinelearning:Predict” ], "Effect": "Allow", "Resource": "arn:aws:machinelearning:{region}:{awsAccountId}:mlmodel/{mlModelId}” }, { "Action": [ "kinesis:ReadStream”, "kinesis:GetRecords”, "kinesis:GetShardIterator”, "kinesis:DescribeStream”,"kinesis:ListStreams” ], "Effect": "Allow", "Resource": "arn:aws:kinesis:{region}:{awsAccountId}:stream/{kinesisStream}" } ] } Allow reading of data from Kinesis stream
  • 56. Connect Kinesis stream and Lambda function import boto aws_lambda = boto.connect_awslambda() aws_lambda.add_event_source( event_source = 'arn:aws:kinesis:' + region + ':' + aws_account_id + ':stream/' + “tweetStream”, function_name = “process_tweets”, role = 'arn:aws:iam::' + aws_account_id + ':role/' + lambda_execution_role) Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
  • 57. Building smart applications Pick the ML strategy 1 Prepare dataset 2 3 Create ML model 4 Write and configure code 5 Try it out!
  • 58. Amazon ML real-time predictions test Here is a tweet:
  • 59. Amazon ML real-time predictions test Here is the same tweet…as a JSON blob: { "statuses_count": "8617", "description": "Software Developer", "friends_count": "96", "text": "`scala-aws-s3` A Simple Amazon #S3 Wrapper for #Scala 1.10.20 available : https://t.co/q76PLTovFg", "verified": "False", "geo_enabled": "True", "uid": "3800711", "favourites_count": "36", "screen_name": "turutosiya", "followers_count": "640", "user.name": "Toshiya TSURU", "sid": "647222291672100864" }
  • 60. Amazon ML real-time predictions test Let’s use the AWS Command Line Interface to request a prediction for this tweet: aws machinelearning predict --predict-endpoint https://realtime.machinelearning.us-east-1.amazonaws.com --ml-model-id ml-tweets --record ‘<json_blob>’
  • 61. Amazon ML real-time predictions test Let’s use the AWS Command Line Interface to request a prediction for this tweet: aws machinelearning predict --predict-endpoint https://realtime.machinelearning.us-east-1.amazonaws.com --ml-model-id ml-tweets --record ‘<json_blob>’ { "Prediction": { "predictedLabel": "0", "predictedScores": { "0": 0.012336540967226028 }, "details": { "PredictiveModelType": "BINARY", "Algorithm": "SGD" } } }
  • 62. Recap: Our application’s data flow Twitter API Amazon Kinesis AWS Lambda Amazon Machine Learning Amazon SNS
  • 64. Generalizing to more feedback channels Amazon Kinesis AWS Lambda Model 1 Amazon SNS Model 2 Model 3
  • 65. What’s next? Try the service: http://aws.amazon.com/machine-learning/ Download the Social Media Listening application code: http://bit.ly/AmazonMLCodeSample Get in touch! [email protected]