Data analytics on Azure
- 1. Analítica Avanzada Azure SQL Data Warehouse Azure Machine Learning Elena López
- 2. ¿Dónde ves tu compañía el día de hoy? ¿Dónde ves tu competencia al día de hoy? ¿Dónde ves tu compañía en el futuro?
- 3. Las Investigaciones dicen: Mi compañía HOY* Mi compañía EN EL FUTURO (24-36 meses)* Mis mayores competencias HOY* CAPACIDADES ANALÍTICAS BASICO AVANZADO *Based on independent market research in the form of focus groups and in-depth-interviews
- 4. Helping you innovate across your business Customer Insights Sales Insights Virtual Assistants Cash Flow Forecasting HR Insights Churn Analytics Dynamic Pricing Waiting line optimization Risk Management Quality Assurance Resource Planning Lead Scoring Intelligent chatbots Financial Forecasting Employee Insights Marketing Sales Service Finance Operations Workforce Product RecommendationProduct Recommendation Predictive MaintenancePredictive Maintenance Demand ForecastingDemand Forecasting
- 5. Transformaciones Innovadoras Hybrid solution predicts onboard water usage, saving $200k/ship/year The average size of a single cart has decreased Provide personalized digital content to shoppers Increase cart size Unplanned downtime results in cost overruns Predict when maintenance should be performed Minimize downtime Solar energy production is inconsistent Align energy supply with the optimal markets Maximize revenue Product recommendation Predictive maintenance Demand forecasting Distributed power generation increases revenue by over €100 million ASOS delivers 15.4 million personalized experiences with 33 orders per second
- 6. ¿Cómo las compañías se están transformando? Construyendo flujos de datos unificados y utilizables Entrenando modelos de Machine Learning y Deep Learning para lograr inteligencia Optimizando los modelos y distriuyéndolos a gran escala Desplegando aplicaciones dinámicas e inteligentes a los usuarios
- 7. Complejidad de las soluciones Muchas Opciones en el mercado Data silos Datos incongruentes Dificultad de escalar efectivamente Rendimiento NegativoY muchos no están satisfechos con sus Soluciones actuales “What it Takes to be Data-Driven: Technologies and Best Practices for Becoming a Smarter Organization”, TDWI, 2017. Están satisfechos con la facilidad de uso del software de análisis 46% 21% Están satisfechos con el acceso a datos estructurados y no estructurados 28% Están satisfechos con la facilidad para escalar cambios y Soluciones de imprevisto Pero…
- 8. Los datos son la clave de cualquier iniciativa de Transformación Digital, la materia prima por excelencia en la 4ta. Revolución industrial Data AI Data modernization to Azure Globally distributed data Cloud Scale Analytics Data Modernization on- premises AI apps & agents Knowledge mining Machine learning
- 9. Los datos deben estar a tiempo La velocidad es crítica pero no suficiente GOBERNANZA DE DATOS Grupo de procesos, roles, políticas, estándares y métricas para asegurar el uso eficiente de la información. Define quién debe tomar cada acción, bajo cuál circunstancia y usando cuáles métodos
- 10. La tierra prometida La realidad Imaginemos encontrar una pieza en particular para cada escenario
- 12. Let’s walk through an actual machine learning scenario…
- 14. Flammable equipment near heat source No hardhat Missing equipment
- 15. No hardhat
- 16. How can we accurately identify hazardous situations with machine learning?
- 17. Machine Learning on Azure Domain specific pretrained models To simplify solution development Azure Databricks Machine Learning VMs Popular frameworks To build advanced deep learning solutions TensorFlowPyTorch ONNX Azure Machine Learning LanguageSpeech … SearchVision Productive services To empower data science and development teams Powerful infrastructure To accelerate deep learning Scikit-Learn Azure Notebooks Jupyter Familiar Data Science tools To simplify model development Visual Studio Code Command line CPU GPU FPGA From the Intelligent Cloud to the Intelligent Edge
- 18. Sophisticated pretrained models Infuse apps with powerful, pre-trained AI models Customize easily and tailor to your needs Vision Speech Language Bing Search … Computer Vision | Video Indexer | Face | Content Moderator Speech to Text | Text to Speech | Speech Translation | Speaker Recognition Text Analytics | Spell Check | Language Understanding | Text Translation | QnA Maker Big Web Search | Video Search | Image Search | Visual Search | Entity Search | News Search | Autosuggest
- 19. Machine Learning on Azure Domain specific pretrained models To simplify solution development Azure Databricks Machine Learning VMs Popular frameworks To build advanced deep learning solutions TensorFlowPyTorch ONNX Azure Machine Learning LanguageSpeech … SearchVision Productive services To empower data science and development teams Powerful infrastructure To accelerate deep learning Scikit-Learn Azure Notebooks Jupyter Familiar Data Science tools To simplify model development Visual Studio Code Command line CPU GPU FPGA From the Intelligent Cloud to the Intelligent Edge
- 20. Machine Learning on Azure Domain specific pretrained models To simplify solution development Azure Databricks Machine Learning VMs Popular frameworks To build advanced deep learning solutions TensorFlowPyTorch ONNX Azure Machine Learning LanguageSpeech … SearchVision Productive services To empower data science and development teams Powerful infrastructure To accelerate deep learning Scikit-Learn Familiar Data Science tools To simplify model development CPU GPU FPGA From the Intelligent Cloud to the Intelligent Edge Azure Notebooks JupyterVisual Studio Code Command line
- 21. Familiar Data Science toolsChoose any python development environment And improve data science productivity PyCharmAzure NotebooksVisual Studio Code Command lineZeppelin Interactive widgets for Jupyter Notebooks Azure Machine Learning for Visual Studio Code extension Jupyter Get started with AML on Azure Notebooks: http://aka.ms/aznotebooks-aml
- 22. Next challenge is to build a model RPN RoIP R-CNN Needs specialized knowledgeTime-consuming Complex
- 23. Machine Learning on Azure Domain specific pretrained models To simplify solution development Popular frameworks To build advanced deep learning solutions Productive services To empower data science and development teams Powerful infrastructure To accelerate deep learning Familiar Data Science tools To simplify model development From the Intelligent Cloud to the Intelligent Edge Azure Databricks Machine Learning VMs TensorFlowPyTorch ONNX Azure Machine Learning LanguageSpeech … SearchVision Scikit-Learn Azure Notebooks JupyterVisual Studio Code Command line CPU GPU FPGA
- 24. Powerful frameworks • Build advanced deep learning solutions Use your favorite deep learning frameworks without getting locked into one framework ONNX Community project created by Facebook and Microsoft Use the best tool for the job. Train in one framework and transfer to another for inference TensorFlow PyTorch Scikit-Learn MXNet Chainer Keras
- 25. Machine Learning on Azure Domain specific pretrained models To simplify solution development Popular frameworks To build advanced deep learning solutions Productive services To empower data science and development teams Powerful infrastructure To accelerate deep learning Familiar Data Science tools To simplify model development From the Intelligent Cloud to the Intelligent Edge Azure Databricks Machine Learning VMs TensorFlowPyTorch ONNX Azure Machine Learning LanguageSpeech … SearchVision Scikit-Learn Azure Notebooks JupyterVisual Studio Code Command line CPU GPU FPGA
- 26. + Productive Services To empower data science and development teams Develop models faster with automated machine learning Use any Python environment and ML frameworks Manage models across the cloud and the edge. Prepare data clean data at massive scale Enable collaboration between data scientists and data engineers Access machine learning optimized clusters Azure Machine Learning Python-based machine learning service Azure Databricks Apache Spark-based big-data service
- 27. Bring AI to everyone with an end-to-end, scalable, trusted platform Built with your needs in mind Support for open source frameworks Managed compute DevOps for machine learning Simple deployment Tool agnostic Python SDK Automated machine learning Seamlessly integrated with the Azure Portfolio Boost your data science productivity Increase your rate of experimentation Deploy and manage your models everywhere
- 28. Fast, easy, and collaborative Apache Spark™-based analytics platform Built with your needs in mind Optimized Apache Spark environmnet Collaborative workspace Integration with Azure data services Autoscale and autoterminate Optimized for distributed processing Support for multiple languages and libraries Seamlessly integrated with the Azure Portfolio Increase productivity Build on a secure, trusted cloud Scale without limits
- 29. Leverage your favorite deep learning frameworks AZURE ML SERVICE Increase your rate of experimentation Bring AI to the edge Deploy and manage your models everywhere TensorFlow MS Cognitive Toolkit PyTorch Scikit-Learn ONNX Caffe2 MXNet Chainer AZURE DATABRICKS Accelerate processing with the fastest Apache Spark engine Integrate natively with Azure services Access enterprise-grade Azure security
- 30. Productive Services • What to use when? + Customer journey Data Prep Build and Train Manage and Deploy Apache Spark / Big Data Python ML developer Azure ML service (Pandas, NumPy etc. on AML Compute) Azure ML service (OSS frameworks, Hyperdrive, Pipelines, Automated ML, Model Registry) Azure ML service (containerize, deploy, inference and monitor) Azure ML service (containerize, deploy, inference and monitor) Azure Databricks (Apache Spark Dataframes, Datasets, Delta, Pandas, NumPy etc.) Azure Databricks + Azure ML service (Spark MLib and OSS frameworks + Automated ML, Model Registry)
- 31. Machine Learning on Azure Domain specific pretrained models To simplify solution development Popular frameworks To build advanced deep learning solutions Productive services To empower data science and development teams Powerful infrastructure To accelerate deep learning Familiar Data Science tools To simplify model development From the Intelligent Cloud to the Intelligent Edge Azure Databricks Machine Learning VMs TensorFlowPyTorch ONNX Azure Machine Learning LanguageSpeech … SearchVision Scikit-Learn Azure Notebooks JupyterVisual Studio Code Command line CPU GPU FPGA
- 33. Flammable equipment near heat source Missing equipment No hardhat
- 34. Differentiators • Automated machine learning • Machine learning DevOps • Machine Learning 95 % Accelerated model building Azure DevOps integration for CI/CD
- 35. How much is this car worth? Azure Machine Learning Automated machine learning
- 36. Model creation is typically a time consuming process Mileage Condition Car brand Year of make Regulations … Parameter 1 Parameter 2 Parameter 3 Parameter 4 … Gradient Boosted Nearest Neighbors SGD Bayesian Regression LGBM … Mileage Gradient Boosted Criterion Loss Min Samples Split Min Samples Leaf XYZ Model Which algorithm? Which parameters?Which features? Car brand Year of make
- 37. Which algorithm? Which parameters?Which features? Mileage Condition Car brand Year of make Regulations … Gradient Boosted Nearest Neighbors SGD Bayesian Regression LGBM … Nearest Neighbors Criterion Loss Min Samples Split Min Samples Leaf XYZ Model Iterate Gradient Boosted N Neighbors Weights Metric P ZYX Mileage Car brand Year of make Model creation is typically a time consuming process Car brand Year of make Condition
- 38. Which algorithm? Which parameters?Which features? Iterate Model creation is typically a time consuming process
- 39. Enter data Define goals Apply constraints Azure Machine Learning accelerates model development with automated machine learning Input Intelligently test multiple models in parallel Optimized model
- 40. 70%95% Azure Machine Learning accelerates model selection with model explainability Feature importance Mileage Condition Car brand Year of make Regulations Model B (70%) Mileage Condition Car brand Year of make Regulations Feature importance Model A (95%)
- 41. Machine Learning DevOps End-to-end ML lifecycle management CI/CD for ML pipelines Feedback loop Register and Manage Model Build Image Build model (your favorite IDE) Deploy Service Monitor Model Train & Test Model Integrated with Azure DevOps
- 43. ServeStore Prep and trainIngest Batch data Streaming data Azure Kubernetes service Power BI Azure analysis services Azure SQL data warehouse Cosmos DB, SQL DB Azure Data Lake Storage Azure Data Factory Azure Event Hubs Azure Databricks Azure Machine Learning service Apps Model Serving Ad-hoc Analysis Operational Databases
- 44. PREP & TRAIN Collect and prepare data Train and evaluate model A B C Operationalize and manage
- 45. Ingest Azure Data Factory Store Azure Blob Storage Understand and transform Azure Databricks • Leverage open source technologies • Collaborate within teams • Use ML on batch streams • Build in the language of your choice • Leverage scale out topology • Scale compute and storage separately • Integrate with all of your data sources • Create hybrid pipelines • Orchestrate in a code-free environment Leverage best-in-class analytics capabilities Scale without limits Connect to data from any source
- 46. • Easily scale up or scale out • Autoscale on serverless infrastructure • Leverage commodity hardware • Determine the best algorithm • Tune hyperparameters to optimize models • Rapidly prototype in agile environments • Collaborate in interactive workspaces • Access a library of battle-tested models • Automate job execution Scale compute resources to meet your needs Quickly determine the right model for your data Simplify model development Automated ML capabilities Automated ML Scale out clusters Infrastructure Machine learning Tools Azure Databricks Azure ML service Azure ML service Azure Databricks Azure ML service
- 47. • Identify and promote your best models • Capture model telemetry • Retrain models with APIs • Deploy models anywhere • Scale out to containers • Infuse intelligence into the IoT edge • Build and deploy models in minutes • Iterate quickly on serverless infrastructure • Easily change environments Proactively manage model performance Deploy models closer to your data Bring models to life quickly Train and evaluate models Model MGMT, experimentation, and run history Azure ML service Containers AKS ACI IoT edge Docker Azure ML service
- 48. 2022202120202019 Deep neural networks will be a standard tool for 80% of data scientists1 More than 40% of data science tasks will be automated1 20% of companies will dedicate workers to monitor neural networks1 90% of modern analytics platforms will feature natural- language generation1 30% of net new revenue growth from industry-specific solutions will include AI1 1 in 5 workers engaged in mostly nonroutine tasks will rely on AI to do their jobs2 1 “100 Data and Analytics Predictions Through 2021”, Gartner, 2017. 2 “Predicts 2018: AI and the Future of Work”, Gartner, 2018. 2018
- 49. • Choose VMs for your modeling needs • Process video using GPU-based VMs • Run experiments in parallel • Provision resources automatically • Leverage popular deep learning toolkits • Develop your language of choice Scale compute resources in any environment Quickly evaluate and identify the right model Streamline AI development efforts AI Ready Virtual Machines Data Science VMs Notebooks, IDEs, Command Line Scale out clusters AML Compute MS Cognitive Toolkit Keras TensorFlow PyTorch Azure ML service SDK
- 50. Deploy Azure ML models at scale Azure Machine Learning service
- 51. Model management in Azure Machine Learning Create/retrain model Create scoring files and dependencies Create and register image Monitor Register model Cloud Light Edge Heavy Edge Deploy image
- 52. © Microsoft Corporation Get started quickly with AI
- 53. © Microsoft Corporation Intelligent apps Leverage AI to create the future of business applications Conversational agents Transform your engagements with customers and employees Business processes Transform critical business processes with AI Of customer interactions powered by AI bots by 2025 Applications to include AI by the end of this year 75% 85% Of enterprises using AI by 202095%
- 54. © Microsoft Corporation Smart factory Smart kitchen Smart bath Voice-activated speakers Smart cameras Drones Smart kiosks
- 55. © Microsoft Corporation Employment Facilitating development of professional skills Modern life Delivering personalized experiences to improve independence Human connection Providing equal access to information and opportunity How Microsoft is helping the visually-impaired How Helpicto is helping non-verbal children How RIT is helping the hearing-impaired Empowering people with tools that amplify human capability
- 57. Customer analytics Financial modeling Risk, fraud, threat detection Credit analytics Marketing analytics Faster innovation for a better customer experience Improved consumer outcomes and increased revenue Enhanced customer experience with machine learning Transform growth with predictive analytics Improved customer engagement with machine learning Customer profiles Credit history Transactional data LTV Loyalty Customer segmentation CRM data Credit data Market data CRM Credit Risk Merchant records Products and services Clickstream data Products Services Customer service data Customer 360 degree evaluation Customer segmentation Reduced customer churn Underwriting, servicing and delinquency handling Insights for new products Commercial/retail banking, securities, trading and investment models Decision science, simulations and forecasting Investment recommendations Real-time anomaly detection Card monitoring and fraud detection Security threat identification Risk aggregation Enterprise DataHub Regulatory and compliance analysis Credit risk management Automated credit analytics Recommendation engine Predictive analytics and targeted advertising Fast marketing and multi- channel engagement Customer sentiment analysis Transaction data Demographics Purchasing history Trends Effective customer engagement Decision services management Risk and revenue management Risk and compliance management Recommendation engine
- 58. Genomics and precision medicine Clinical and claims data GPU image processing IoT device analytics Social analytics Faster innovation for drug development Improved outcomes and increased revenue Diagnostics leveraging machine learning Predictive analytics transforms quality of care Improved patient communications and feedback FAST-Q BAM SAM VCF Expression HL7/CCD 837 Pharmacy Registry EMR Readings Time series Event data Social media Adverse events Unstructured Single cell sequencing Biomarker, genetic, variant and population analytics ADAM and HAIL on Databricks Claims data warehouse Readmission predictions Efficacy and comparative analytics Prescription adherence Market access analysis Graphic intensive workloads Deep learning using Tensor Flow Pattern recognition Aggregation of streaming events Predictive maintenance Anomaly detection Real-time patient feedback via topic modelling Analytics across publication data MRI X-RAY CT Ultrasound DNA sequences Real world analytics Image deep learning Sensor data Social data listening
- 59. Content personalization Customer churn prevention Recommendation engine Predictive analytics Sentiment analysis Faster innovation for customer experience Improved consumer outcomes and increased revenue Enhance user experience with machine learning Predictive analytics transforms growth Improved consumer engagement with machine learning Customer profiles Viewing history Online activity Content sources Channels Customer profiles Online activity Content distribution Services data Transactions Subscriptions Demographics Credit data Content metadata Ratings Comments Social media activity Personalized viewing and engagement experience Click-path optimization Next best content analysis Improved real time ad targeting Quality of service and operational efficiency Market basket analysis Customer behavior analysis Click-through analysis Ad effectiveness Content monetization Fraud detection Information-as-a-service High value user engagement Predict audience interests Network performance and optimization Pricing predictions Nielsen ratings and projections Mobile spatial analytics Demand-elasticity Social network analysis Promotion events time-series analysis Multi-channel marketing attribution Consumption logs Clickstream and devices Marketing campaign responses Personalized recommendations Effective customer retention Information optimization Inventory allocation Consumer engagement analysis
- 60. Next best and personalized offers Store design and ergonomics Data-driven stock, inventory, ordering Assortment optimization Real-time pricing optimization Faster innovation for customer experience Improved consumer outcomes and increased revenue Omni-channel shopping experience with machine learning Predictive analytics transforms growth Improved consumer engagement with machine learning Customer profiles Shopping history Online activity Social network analysis Shopping history Online activity Floor plans App data Demographics Buyer perception Consumer research Market/competitive analysis Historical sales data Price scheduling Segment level price changes Customer 360/consumer personalization Right product, promotion, at right time Multi-channel promotion Path to purchase In-store experience Workforce and manpower optimization Predict inventory positions and distribution Fraud detection Market basket analysis Economic modelling Optimization for foot traffic, Online interactions Flat and declining categories Demand-elasticity Personal pricing schemes Promotion events Multi-channel engagement Demand plans Forecasts Sales history Trends Local events/weather patterns Recommendation engine Effective customer engagement Inventory optimization Inventory allocation Consumer engagement
- 61. Customer value analytics Next best and personalized offers Risk and fraud management Sales and campaign optimization Sentiment analysis Faster innovation for customer growth Improved outcomes and increased revenue Risk management with machine learning Predictive analytics transforms growth Improved customer engagement with machine learning Customer profiles Online history Transaction data Loyalty Customer segmentation CRM data Credit data Market data CRM Merchant records Products Services Marketing data Social media Online history Customer service data Customer 360, segmentation aggregation and attribution Audience modelling/index report Reduce customer churn Insights for new products Historical bid opportunity as a service Right product, promotion, at right time Real time ad bidding platform Personalized ad targeting Ad performance reporting Real-time anomaly detection Fraud prevention Customer spend and risk analysis Data relationship maps Optimizing return on ad spend and ad placement Multi-channel promotion Ideal customer traits Optimized ad placement Opinion mining/social media analysis Deeper customer insights Customer loyalty programs Shopping cart analysis Transaction data Demographics Purchasing history Trends Effective customer engagement Recommendation engine Risk and fraud analysis Campaign reporting analytics Brand promotion and customer experience
- 62. Digital oil field/ oil production Industrial IoT Supply-chain optimization Safety and security Sales and marketing analytics Faster innovation for revenue growth Improved outcomes and increased revenue Optimizing supply- chain with machine learning Predictive analytics transforms safety and security Improved customer engagement with machine learning Field data Asset data Demographics Production data Sensor stream data UAVs images Inventory data Production data Sensor stream data Transport Retail data Grid production data Refinery tuning parameters Clickstream data Products Services Market data Competitive data Demographics Production optimization Integrate exploration and seismic data Minimize lease operating expenses Decline curve analysis Pipeline monitoring Preventive maintenance Smart grids and microgrids Grid operations, field service Asset performance as a service Trade monitoring, optimization Retail mobile applications Vendor management - construction, transportation, truck and delivery optimization Real-time anomaly detection Predictive analytics Industrial safety Environment health and safety Fast marketing and multi-channel engagement Develop new products and monitor acceptance of rates Predictive energy trading Deep customer insights Transaction data Demographics Purchasing history Trends Upstream optimization, maximize well life Grid operations, asset inventory optimization Supply-chain optimization Risk optimization Recommendations engine
- 63. Intrusion detection and predictive analytics Security intelligence Fraud detection and prevention Security compliance reporting Fine-grained data analytics security Prevent complex threats with machine learning Faster innovation for threat prevention Risk management with machine learning Transform security with improved visibility Limit malicious insiders to transform growth Firewall/network logs Apps Data access layers Firewall/network logs Network flows Authentications Firewall/network logs Web Applications Devices OS Files Tables Clusters Reports Dashboards Notebooks Prevention of DDoS attacks Threat classifications Data loss/anomaly detection in streaming Cybermetrics and changing use patterns Real-time data correlation Anomaly detection Security context, enrichment Offence scoring, prioritization Security orchestration e-Tailing Inventory monitoring Social media monitoring Phishing scams Piracy protection Ad-hoc/historic incident reports SOC/NOC dashboards Deep OS auditing Data loss detection in IoT User behavior analytics Role-based access controls Auditing and governance File integrity monitoring Row level and column level access permissions Firewall/network logs Web/app logs Social media content Security controls to leverage all data Actionable threat intelligence Risk and fraud analysis Compliance management Identity and access management for analytics
- 65. Use any language, any development tool and any framework >90% of Fortune 500 companies use Microsoft Cloud Benefit from industry-leading security, privacy, compliance, transparency, and AI ethics standards Accelerate time to value with agile tools and services Powerful tools Pretrained AI services Comprehensive platform On-premisesEdgeCloud Innovate with AI everywhere – in the cloud, at edge and on-premises
- 66. © Microsoft Corporation It’s all on MicrosoftAzure
- 68. © Microsoft Corporation System integrationData integration BI and analytics