Borys Rybak “Azure Machine Learning Studio & Azure Workbench & R + Python”
Download as PPTX, PDF1 like380 views
Machine learning enables computers to learn from data and experiences to act without being explicitly programmed. The goal of machine learning is to use example data or past experience to solve problems. There are different styles of machine learning algorithms such as supervised learning where the training data is labeled, and unsupervised learning where the training data is unlabeled. Machine learning problems can involve regression, classification, or clustering. The machine learning process involves preparing data, applying learning algorithms to create models, and deploying chosen models through applications and APIs.
Borys Rybak “Azure Machine Learning Studio & Azure Workbench & R + Python”
- 8. What is Machine Learning?
- 10. What exactly is Machine Learning?
- 11. “Machine learning enables computers to learn from data and experiences and to act without being explicitly programmed.”
- 12. 130 exabytes of data (in 2005) • 2005 - 130 exabytes of data • 2010 - 1200 exabytes • 2015 - 7900 exabytes • 2020 - 40900 exabytes
- 14. What is the goal of Machine Learning?
- 15. “The goal of machine learning is to program computers to use example data or past experience to solve a given problem.
- 16. Machine Learning in a Nutshell Machine learning algorithm Model Application Data Contains patterns Finds patterns Recognizes patterns Provides new data to see if it matches known patterns
- 17. Finding Patterns: A Simple Example Name $2,600.45 $2,294.58 $1,003.30 $8,488.32 Amount Fraudulent Smith Potter Peters Adams No Yes Yes No What’s the pattern for fraudulent transactions?
- 18. Finding Patterns: Another Example $2,600.45 $2,294.58 $1,003.30 $8,488.32 Name Amount Fraudulent Smith Potter Peters Adams No Yes Yes No Where Issued Where Used Age $200.12 $3,250.11 $8,156.20 $7,475.11 Pali Jones Hanford Marx USA RUS USA FRA AUS USA USA UK 22 29 25 64 58 43 27 32 No No Yes No USA USA RUS USA JAP RUS UK GER $540.00 $7,475.11 Norse Edson USA USA 27 20 No Yes ITA RUS What’s the pattern for fraudulent transactions?
- 19. P(A) P(B|A) P(B) P(A|B) = Deep learning uses this Styles of ML Algorithms: Examples
- 20. Terminology Training data The prepared data used to create a model Creating a model is called training a model Supervised learning The value you want to predict is in the training data The data is labeled Unsupervised learning The value you want to predict is not in the training data The data is unlabeled The most common approach
- 21. Categorizing ML Problems: Regression Goal: Predict a value Example question:
- 22. Categorizing ML Problems: Classification Goal: Predict a class Example question: Class 1 Class 2 Can be more than two classes
- 23. Categorizing ML Problems: Clustering Goal: Discover structure Example question: Cluster 1 Cluster 2 Cluster 3
- 24. Hey Machine! What to learn?
- 25. Too complex: (When you can’t code it….) NLP H-W recognition CV Too specialized: (When you have to adapt...) Amazon Netflix Predictive typing Too much: (When you can’t scale it….) Spam detection Fraud detection Healthcare Too robotic: (When you can’t track it….) AI gaming Robot control
- 27. Collaborative Filtering: (Scenario) Lucy: Bob: John: Content-Based Filtering: (Scenario) Lucy: Bob: John: What to recommend to John?
- 28. Collaborative Filtering: Not relying on machine analyzable content Similarity alg's: k-NN & Pearson Explicit vs. Implicit data collection • Explicit: rating movies after watch • Implicit: how often you watch movies, how often you have watched season in period of time. Used in: Facebook, MySpace, LinkedIn, Twitter Challenges: Cold start, scalability, sparsity Algorithms: matrix factorization Content-Based Filtering: History: Information Retrieval Algorithms for abstracting: tf-idf Algorithms: Bayesian classifiers, cluster analysis, decision trees, articial neural networks Scenarios: Netflix
- 29. Time for Action!
- 30. Deploy chosen model Chosen Model Apply learning algorithm to data Candidate Model Prepared Data Apply pre- processing to data Iterate to find the best model Data Preprocessing Modules Iterate until data is ready Preprocessing Modules Machine Learning Algorithms Applications The goal: Smarter applications Raw Data Raw Data Choose data The Machine Learning Process
- 31. Candidate Model Raw Data Raw Data Prepared Data Apply pre- processing to data Deploy chosen model Apply learning algorithm to data Chosen Model ML Studio Preprocessing Modules Machine Learning Algorithms Data Preprocessing Modules Azure ML API Applications Azure ML Again
- 32. Azure ML Studio Allows running “experiments” Candidate Model Raw Dat a Raw Dat a Prepared Data Apply pre- processing to data Deploy chosen model Apply learning algorithm to data Chosen Model ML Studio Preprocessing Modules Machine Learning Algorithms Data Preprocessing Modules Azure ML API
- 33. Azure ML Data Preprocessing Example modules Candidate Model Raw Dat a Raw Dat a Prepared Data Apply pre- processing to data Deploy chosen model Apply learning algorithm to data Chosen Model Preprocessing Modules Machine Learning Algorithms Data Preprocessing Modules Azure ML API Clean Missing Data Removes or fills in missing values in a dataset Example: Replace each missing value with the mean of the other values in this column Select Columns in Dataset Creates a view of a dataset that includes or excludes specific columns Example: Delete a column whose data is highly correlated with data in another column Partition and Sample Divides or extracts a subset of a dataset Example: Select a specific number of rows from the data There are dozens more data preprocessing modules
- 34. Azure ML Learning Example algorithms Candidate Model Raw Dat a Raw Dat a Prepared Data Apply pre- processing to data Deploy chosen model Apply learning algorithm to data Chosen Model Preprocessing Modules Machine Learning Algorithms Data Preprocessing Modules Azure ML API Regression Linear regression Ordinal regression Bayesian linear regression Neural network regression Decision forest regression Boosted decision tree regression Classification Two-class neural network Two-class decision forest Multiclass neural network Multiclass decision forest Multiclass decision jungle Clustering K-means Azure ML is designed for data scientists
- 38. Microsoft Azure Azure ML API Chosen Model Azure ML API Deploying and using a model 1) Deploy chosen model ML Studio 2) Call model with values for features the model requires 3) Get back value predicted by the model using those features Application Candidate Model Raw Dat a Raw Dat a Prepared Data Apply pre- processing to data Deploy chosen model Apply learning algorithm to data Chosen Model Preprocessing Modules Machine Learning Algorithms Data Preprocessing Modules Azure ML API
- 39. Time for Something Different!
- 40. Scenario Detecting credit card fraudMICROSOFTAZUREONPREMISES Model Azure ML Azure Blobs Historical Transaction Data Credit Card Customers Payment Application
- 41. Scenario Predicting customer churnMICROSOFTAZUREONPREMISES Call Center Staff Call Center Application Model Azure ML Azure Blobs Detailed Call Data ML Prep Application CRM Data Aggregated Call Data Data for ML Azure HDInsight Aggregation Application
- 42. Scenario Predicting equipment failure MICROSOFTAZURE Azure ML Model Azure IoT Hub Streaming Data ONPREMISES INTERNETCONNECTED Maintenance StaffDevices Azure Stream Analytics, HDInsight Storm, Spark Streaming Notification Application
- 44. The Reality of Machine Learning No model is an island Azure ML is commonly used with other Azure technologies For ingesting data For storing data For displaying data More … A complete solution often contains many different parts Which can make the story complex for customers Solution: Cortana Intelligence Suite A group of related Azure data technologies for analytics and intelligence
- 45. Cortana Intelligence Suite Originally known as Cortana Analytics Suite
- 46. Cortana Intelligence Gallery Allows access to ML APIs, e.g., Face API, Translator API, etc. Offers example models for many industries, e.g., retail, healthcare, etc.
- 47. Challenges for Machine Learning
- 48. Business Leaders Want solutions to business problems Data Scientists Want powerful, easy- to-use tools Software Developers Want to create better applications Who can be interested in ML?
- 50. Lots od data Lots of computer power Effective algorithms.
- 51. Very well defined question.
- 52. • Choosing what question to ask is the most important part of the process • Ask yourself: Do you have the right data to answer this question? • Ask yourself: Do you know how you’ll measure success? ? $ % +
- 53. Iterate until you have a model that makes good predictions.
- 54. “Make machine learning accessible to every enterprise, data scientist, developer, information worker, consumer, and device anywhere in the world.”
- 56. ~ Stephen Hawking, Web Summit 2017
Editor's Notes
- #2: Thanks for invitation, excitement about city, venue and confernce
- #3: Tell what will be said. --> Machine Learning - Why? How? What? --> Machine Learning in action - Short Demo (basic algorithm, Python or R) --> Azure Machine Learning - Why? How? What? --> Azure Machine Learning Studio - showcase (demo1, demo2, demo3, summary) --> Azure Machine Learning Workbench - showcase (demo1, demo2, summary)
- #4: Introduction of myself (background, profession )
- #5: Introduction of myself (personal) -> Ask Questions for better communication and knowledge about the public.
- #6: How many of you already tried Machine Learning? (basics scenario, complex one) Which one basics? So the rest complex? So how many of you want to find out something new about Machine Learning and maybe Learn about it?
- #12: Customers can build Artificial Intelligence (AI) applications that intelligently: sense, process, and act on information (augmenting human capabilities, increasing speed and efficiency, and helping organizations achieve more)
- #13: Literka 'A' = 1 byte Strona z literami = 1000x Literka 'A' = 1 kilobyte Ksiazka ze stronami = 1000x strona z literkami (500 obustronnie) = 1 megabyte Human genome 1000x 1 megabyte (you can encode a whole human being on) = 1 gigabyte Putting every second of human life for 80 years in HD video = 1 terabyte 1.4 miliarda akrów (1 acre = ~4k m^2), kazdy akr ma ~500 drzew, wiec w sumie 700 miliardow drzew w lasach tropikalnych Amazonii- jesli wiec wytnie sie wszystkie drzewa i zamieni na kartki papieru, umiesci na kazdej kartce po literce 'A' x1000 to bedzie sie miec od 1 do 2 petabyte danych. [zdjecie ziemii] jesli pomnozymy to x1000 to bedziemy miec exabyte.
- #16: ~Introduction to Machine Learning, 2nd Edition, MIT Press
- #26: Too complex: When you can't code it. (NLP, hand-writing recognition, Computer Vision…) Too much: When you can't scale it. (e.g. Spam & fraud detection, healthcare) Too specialized: When you have to adapt/personalize. (Amazon, Netflix, predictive typing) Autonomous: When you can't track it. (AI gaming, robot control)
- #27: Netflix Example Movies already watched Same actor Same director Period of time (watched whole season per night or whole series) Basic Recommenders Systems - Collaborative Filtering Scanario - Content-Based Filtering Scenario The Netflix Prize was an open competition for the best collaborative filtering algorithm to predict user ratings for films, based on previous ratings without any other information about the users or films, i.e. without the users or the films being identified except by numbers assigned for the contest. The competition was held by Netflix, an online DVD-rental and video streaming service, and was open to anyone who is neither connected with Netflix (current and former employees, agents, close relatives of Netflix employees, etc.) nor a resident of certain blocked countries (such as Cuba or North Korea).[1] On September 21, 2009, the grand prize of US$1,000,000 was given to the BellKor's Pragmatic Chaos team which bested Netflix's own algorithm for predicting ratings by 10.06%.[2]
- #28: What Music Do I Want to Listen to? last.fm: Collaborative filtering User's past behavior Cold start PANDORA: Content-based filtering Characteristics of items, i.e. properties of songs, artists,… Little information needed at the beginning
- #29: TFIDFIn - information retrieval, tf–idf or TFIDF, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.
- #30: Targeted Marketing Collaborative Filtering Collaborative + Content-based Filtering = Hybrid Recommender
- #40: Azure ML Workbench Azure for Students Showing Portal (AZURE) Showing DSVM + DLVM Showing demo Workbench Why Deep Learning Virtual Machine? Increasingly, deep learning algorithms / deep neural networks are becoming one of the popular methods employed in many machine learning problems. They are especially good at machine cognition tasks like image, text, audio/video understanding often approaching human cognitive levels in some specific domains with advanced deep neural network architectures and access to large set of data to train models. Deep learning requires large amount of computational power to train models with these large datasets. With the cloud and availability of Graphical Processing Units (GPUs), it is becoming possible to build sophisticated deep neural architectures and train them on a large data set on powerful computing infrastructure on the cloud. The Data Science Virtual Machine has provided a rich set of tools and samples for data preparation, machine learning, and deep learning. But one of the challenges faced by the users is to discover the tools and samples for specific scenarios like deep learning easily and also more easily provision GPU-based VM instances. This Deep Learning Virtual Machine (DLVM) addresses these challenges. What is Deep Learning Virtual Machine? The Deep Learning Virtual Machine is a specially configured variant of the Data Science Virtual Machine (DSVM) to make it more straightforward to use GPU-based VM instances for training deep learning models. It is supported on Windows 2016 and the Ubuntu Data Science Virtual Machine. It shares the same core VM images (and hence all the rich toolset) as the DSVM but is configured to make deep learning easier. We also provide end-to-end samples for image and text understanding, that are broadly applicable to many real life AI scenarios. The deep learning virtual machine also tries to make the rich set of tools and samples on the DSVM more easily discoverable by surfacing a catalog of the tools and samples on the virtual machine. In terms of the tooling, the Deep Learning Virtual Machine provides several popular deep learning frameworks, tools to acquire and pre-process image, textual data. For a comprehensive list of tools, you can refer to the Data Science Virtual Machine Overview Page.
- #42: Simply put, customer churn occurs when customers or subscribers stop doing business with a company or service. Also known as customer attrition, customer churn is a critical metric because it is much less expensive to retain existing customers than it is to acquire new customers
- #48: Challenges for ML -Skilled Data Scientists Like unicorns, PhD in maths, collaborative, cs, statistics background, business approach understanding. -Infrastructure Cheaper to store data, you can scale it – cloud -Time Running algorithms at classes so long (leaving laptop by night) - with cloud you can increment. -Global scalable How to scale those algorithms globaly
- #50: What Machine Learning Does? Finds patterns in data Then uses those patterns to predict the future Examples: Detecting credit card fraud Determining whether a customer is likely to switch to a competitor Deciding when to do preventive maintenance on a factory robot
- #51: Why is Machine Learning So Hot? Doing machine learning well requires: Lots of data Lots of compute power Effective machine learning algorithms All of those things are now more available than ever
- #52: What Do I Need for Data Science? Very well defined question / business objective "which of my machines / is this part of my machine going to fail within the next two weeks?" //what is the probability of failing in next two weeks? You need to have data - Relevant data Data that is relevant to the business objective That data need to be accurate - Accurate data Connected data Large data It is not necessarily Big Data, but large is good.
- #53: What Do I Need for Data Science? Very well defined question / business objective "which of my machines / is this part of my machine going to fail within the next two weeks?" //what is the probability of failing in next two weeks? You need to have data - Relevant data Data that is relevant to the business objective That data need to be accurate - Accurate data Connected data Large data It is not necessarily Big Data, but large is good.
- #54: Machine learning has its own jargon Such as training data, features, and supervised learning Machine learning problems often fall into three areas: Regression Classification Clustering Many different types of machine learning algorithms are used today Using these algorithms to create good models requires effort
- #55: Azure Machine Learning Machine learning isn’t hard to understand Although it can be hard to do well Azure ML is a strong offering Especially for data scientists Machine learning can probably help your customers
- #57: Watch Stephen Hawking speak about the future of AI at Web Summit.