[db tech showcase Tokyo 2018]　#dbts2018 #B27 『Discover Machine Learning and ADWC - The Perfect Combination』

[db tech showcase Tokyo 2018]　#dbts2018 #B27 『Discover Machine Learning and ADWC - The Perfect Combination』

• 1.
  Discover Machine Learningand ADWC The Perfect Combination
• 2.
  Francisco Munoz Alvarez •Oracle ACE Director • 8/9/10g/11g/12c OCP, RAC OCE, AS OCA, E-Business OCP, • SQL/PLSQL OCA, Oracle 7 OCM • Oracle 7, 11GR2, 12cR1, 12cR2, ADWCS and OVM 3.1 and 3.2 and 3.3 Beta Tester/Early Adopter • IOUC LA and APAC Spokesperson, President of APACOUC,IAOUG, CLOUG and NZOUG • ITIL Certified • 2008 Top OTN Forum Contributor (All year #1) • 2008 Oracle ACE Award Winner • 2010 Oracle Excellence Award Winner • 2010 Oracle Magazine Editors Choice Award Winner • 2012 LAOUC Architect of the year Winner • 2017 APAC Webinar Tour Best Session Winner • Presented over 400 sessions at 47 Countries around the world Blog: oraclenz.com - Email: [email protected] Twitter : fcomunoz Data Intensity Director of Innovation www.dataintensity.com
• 3.
  2001 - 20042005 2006 - 2008 2009 2010 - 2012 2013 - 2014 2015 - 2016 2017 - 2018 Corporate Evolution & Strategy 3 Company founded in 2001 Provide remote managed services for Oracle applications and databases Established Eastern European outsourcing partnership Introduced Oracle hosted application offerings Launched Oracle Engineered Systems Practice RedStack Acquisition Enrich & Inatech Acquisitions EQT Investment Oracle Cloud Specializations - Financials - Procurement - Infrastructure Oracle Managed Services Provider Credentials Added Oracle EBS Functional Services Became Platinum Partner Added support for expanding Oracle ecosystem OBI, Hyperion, Agile, GRC, Siebel, BRM Introduced dedicated private and shared cloud computing solutions Audax Group Investment Clear Measures Acquisition
• 4.
  Delivery & ServiceCapabilities That Span The Globe 4
• 5.
  5 Oracle Credentials &Deep Oracle Expertise SSAE 16 SOC 1 Type II ISO 9001 Certified ITIL & ITSM Certified Oracle Cloud Applications Financials Cloud Procurement Cloud Infrastructure Cloud Oracle Applications E-Business Suite R12 Financials E-Business Suite R12 Supply Chain Oracle Engineered Systems Exadata Database Machine X2 Exadata Database Machine Oracle Database Appliance Oracle Platform Linux 5 / 6 OVM 2 / 3 CustomerAdvisory Board Oracle EBS ATG Oracle Cloud Infrastructure 2017 Oracle Cloud Infrastructure Partner of the Year – UK 2015 Engineered Systems Partner of the Year - Global BETAProgram Participant Oracle Technology Database 11g / 12c Database 11g / 12c Security Database 11g / 12c Data Warehousing Database 11g / 12c Performance Tuning Enterprise Manager 11g / 12c Real Application Clusters 11g / 12c Business Intelligence Foundation 10 / 11g Data Integration 11g / 12c WebLogic Server 12c Identity Management 12c SOA Suite 12c Big Data Appliance Oracle Cloud Infrastructure Associate Architect Certifications
• 6.
  Oracle Database & EngineeredSystems Oracle Applications ERP / Procurement Oracle BI & Analytics Global Portfolio of Oracle Solutions and Services Delivered Your Way 6 DELIVERYMODELS Remote Managed Services Extension of Customer IT Team | 24x7 Monitoring & Critical Response | Daily Management & Maintenance Cloud Managed Services On-Premises/Private Cloud | Public Cloud | Hybrid Cloud Professional Services / Consulting Assessments/Design | Migration | Implementations | Upgrades | Quick Start Programs
• 7.
  The Data IntensityOracle Advantage 7 We are an Oracle First service provider and believe in Oracle on Oracle for Oracle We have been selling enterprise cloud services for over 12 years and understand the sales process We architected OCI for our own EBS environment and presented it at Oracle Cloud Days Comprehensive portfolio of solutions from Oracle Cloud to On-Premises Applications and technology. Oracle Platinum and Managed Services Partner focused on accelerated migration and management services for Oracle Cloud Infrastructure. Proven with hundreds of Oracle clients worldwide who have entrusted us to handle the implementation, management, and support of their mission-critical Oracle applications and technologies
• 8.
  Applications (On-Premises) • OracleE-Business Suite - Financials, SCM, Procurement • Hyperion • PeopleSoft • ATG Web Commerce • OBIEE • BRM • Agile • GRC • Siebel • Fusion Middleware Database & Advanced Technology • Oracle Database • Advanced Security • Advanced Compression • RAC • Oracle Streams • WebLogic Suite • Active • Database Vault • WebCenter • Data Guard • Advanced Compression • ODI • GoldenGate • TDE Infrastructure • Oracle Exadata Machine • Oracle Database Appliance • Exalogic Elastic Cloud • Oracle Private Cloud Appliance • Exalytics • ZFS • Oracle SuperCluster • VM • OEL SaaS • Oracle ERP Cloud • Financials • HCM • Procurement • PPM • OM • EPM PaaS • Oracle Database Cloud • Java • Oracle SOA Cloud • BI • Oracle Identity Cloud • Oracle Data Integrator • Oracle Management Cloud IaaS • Oracle Cloud Infrastructure Oracle Capabilities & Alignment Partner for Growth Move Workloads to the Cloud Modernize Data Management Connect & Extend Apps Technical & Functional Integration Delivery Capabilities Alignment to 3 Oracle Sales Plays Confidential 8
• 9.
  Machine Learning andADWC 9 The Perfect Combination What is Machine Learning Introduction to Autonomous Data Warehouse Cloud Service Introduction to Oracle Machine Learning Running SQL Statements and Scripts Creating Interactive Notebooks Scheduling Notebooks to Refresh Conclusion
• 10.
  Machine Learning andADWC 10 The Perfect Combination What is Machine Learning Introduction to Autonomous Data Warehouse Cloud Service Introduction to Oracle Machine Learning Running SQL Statements and Scripts Creating Interactive Notebooks Scheduling Notebooks to Refresh Conclusion
• 11.
  Machine Learning First ofall, let’s review what machine learning is. • Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data and use it learn for themselves. • The process of learning begins with observations or data, such as examples, direct experience, or instruction, in order to look for patterns in data and make better decisions in the future based on the examples that we provide. The primary aim is to allow the computers learn automatically without human intervention or assistance and adjust actions accordingly. • Machine learning enables analysis of massive quantities of data. While it generally delivers faster, more accurate results in order to identify profitable opportunities or dangerous risks, it may also require additional time and resources to train it properly. Combining machine learning with AI and cognitive technologies can make it even more effective in processing large volumes of information. 11 Data Science
• 12.
  Machine Learning Algorithms • Supervisedmachine learning algorithms can apply what has been learned in the past to new data using labeled examples to predict future events. Starting from the analysis of a known training dataset, the learning algorithm produces an inferred function to make predictions about the output values. The system is able to provide targets for any new input after sufficient training. The learning algorithm can also compare its output with the correct, intended output and find errors in order to modify the model accordingly. • In contrast, unsupervised machine learning algorithms are used when the information used to train is neither classified nor labeled. Unsupervised learning studies how systems can infer a function to describe a hidden structure from unlabeled data. The system doesn’t figure out the right output, but it explores the data and can draw inferences from datasets to describe hidden structures from unlabeled data. • Semi-supervised machine learning algorithms fall somewhere in between supervised and unsupervised learning, since they use both labeled and unlabeled data for training – typically a small amount of labeled data and a large amount of unlabeled data. The systems that use this method are able to considerably improve learning accuracy. Usually, semi-supervised learning is chosen when the acquired labeled data requires skilled and relevant resources in order to train it / learn from it. Otherwise, acquiringunlabeled data generally doesn’t require additional resources. • Reinforcement machine learning algorithms is a learning method that interacts with its environment by producing actions and discovers errors or rewards. Trial and error search and delayed reward are the most relevant characteristics of reinforcement learning. This method allows machines and software agents to automatically determine the ideal behavior within a specific context in order to maximize its performance. Simple reward feedback is required for the agent to learn which action is best; this is known as the reinforcement signal. 12 Data Science
• 13.
  Machine Learning Example: The bestway to explain what machine learning is would be to give you a simple example. Let’s say you want to develop a program that automatically detects what’s in a picture, and we show a picture of a dog. However, this kind of approach gets tricky pretty quickly. What if there’s a white dog in the picture with no brown hair? What if the picture shows only the round parts of the table? This is where machine learning comes in. Machine learning typically implements an algorithm that automatically detects a pattern in the given input. You can give, say, 1,000 pictures of a dog and 1,000 pictures of a table to a machine learning algorithm. Then, it will learn the difference between a dog and a table. When you give it a new picture of either a dog or a table, it will be able to recognize which one it is. 13 Data Science
• 14.
  Machine Learning Machine learningalgorithms work much the same way. You can apply the same idea to: • Recommendation systems (think YouTube, Amazon, and Netflix) • Face recognition • Voice recognition • Among other applications. Popular machine learning algorithms you might have heard about include: • Neural networks • Deep learning • Support vector machines • Random forest You can use any of the above algorithms to solve the picture-labeling problem I mentioned before. 14 Data Science
• 15.
  Machine Learning andADWC 15 The Perfect Combination What is Machine Learning Introduction to Autonomous Data Warehouse Cloud Service Introduction to Oracle Machine Learning Running SQL Statements and Scripts Creating Interactive Notebooks Scheduling Notebooks to Refresh Conclusion
• 16.
  Autonomous Data WarehouseCloud • Easy o Fully-managed, pre-configured and optimized for DW workloads o Simply load data and run • No need to define indexes, create partitions, etc. • Fast o Based on Exadata technology • Elastic o Instant scaling of compute or storage with no downtime
• 17.
  ADW – In-DatabaseSQL Analytics Hierarchical Analytics Window Functions Forecasting Statistics Approximate Analytics Pattern Matching SQL Modeling Advanced Aggregations Ranking Pivoting Used-Defined PTFs Text Analytics
• 18.
  ADW – In-DatabaseSQL Analytics Oracle Machine Learning SQL notebooks provide easy access to Oracle's parallelized, scalable in-database implementations of a library of Oracle Advanced Analytics' machine learning algorithms (classification, regression, anomaly detection, clustering, associations, attribute importance, feature extraction, times series, etc.), SQL, PL/SQL and Oracle's statistical and analytical SQL functions. Oracle Machine Learning SQL notebooks and Oracle Advanced Analytics' library of machine learning SQL functions combined with PL/SQL allow companies to automate their discovery of new insights, generate predictions and add "AI" to data viz dashboards and enterprise applications.
• 19.
  ADW : In-DatabaseMachine Learning Classification Regression Anomaly Detection Attribute Importance Association Rules Clustering Feature Extraction Descriptive statistics Hypothesis Testing ANOVA Test Distribution Fit Text Mining
• 20.
  ADW : In-DatabaseMachine Learning Oracle 18c get the following new features, that include new machine learning algorithms, improvements to machine learning algorithms, and meta-data improvements for registering new R based algorithms: • New Time-Series function : This new function forecasts target value based solely on a known history of target values and uses the popular auto-regressive modelling method. • New Model Detail Views : Previously you could inspect the details of a model using a function. This is being phased out and replaced by model view, with the format DM$VA • New Neural Networks Algorithm : With the growing interest in deep learning, Oracle have now included a neural network algorithm into the database, thus providing SQL and PL/SQL interfaces to all for easy of use and easy of integration into applications. • New Random Forest Algorithm : Random Forests has been proven over the past few years to be very accurate for certain types of classification problems. This algorithm has now been included in the database, with SQL and PL/SQL interfaces. • Improved Sampling for Association Rules : A new specialised sampling approach is introduced for Association Rules. This is to improve performance, while maintaining accuracy, for large/big data sets. • Algorithm Meta Data Registration : Simplifies the integration of new algorithms in the R extensibility framework. This feature allows a uniform consistent approach of registering new algorithm functions and their settings.
• 21.
  Machine Learning andADWC 22 The Perfect Combination What is Machine Learning Introduction to Autonomous Data Warehouse Cloud Service Introduction to Oracle Machine Learning Running SQL Statements and Scripts Creating Interactive Notebooks Scheduling Notebooks to Refresh Conclusion
• 22.
  Oracle ML –SQL Notebooks For Everyone • Oracle Machine Learning is a SQL notebook interface for data scientists to perform machine learning in the Oracle Autonomous Data Warehouse Cloud (ADWC). • Notebook technologies support the creation of scripts while supporting the documentation of assumptions, approaches and rationale to increase data science team productivity. • Oracle Machine Learning SQL notebooks, based on Apache Zeppelin technology, enable teams to collaborate to build, evaluate and deploy predictive models and analytical methodologies in the Oracle ADWC. • Multi-user collaboration enables the same notebook to be opened simultaneously by different users. • Changes made by one user are immediately updated for other team members. e notebook to be opened SQL
• 23.
  Oracle ML –SQL Notebooks For Everyone • Easy o Integrated for Data Visualization (Reporting), Data Analysis (BI) o Web based SQL notebook integrated into ADW (Web Service) o Autonomous setup and configuration (Fully integrated, no connection to setup) o Sample gallery of notebooks: machine learning • Flexible o SQL access to 18c in-database analytics o Markdown tags for dynamic visualizations (Markdown Language that comes with OML) • Collaborative o Collaboration and sharing built-in o Notebooks support versioning SQL
• 24.
  What is OracleML • Built-in, web-based SQL Notebook • Bundled with Autonomous Data Warehouse • Derived from open-source Apache Zeppelin o Extensive additions that are being fed back into the Apache Zeppelin project • Provides web-based SQL access to ADW • Simple but powerful set of data visualizations • Includes sharing and collaboration framework • All workspaces, projects, notebooks etc saved inside database o Automatically (i.e. autonomously) managed and backed up by database 25
• 25.
  Autonomous Data WarehouseCloud: Architecture Overview Data Warehouse Services (EDWs, DW, departmental marts and sandboxes) Oracle Machine Learning SQL Notebook SQL ANSI 2016 Compliant Analytic SQL Functions Machine Learning Statistics Time Series Pattern Matching Analytic Views Mark- down HTML tags Dynamic forms URL tags Image tags HIGH MEDIUM LOW
• 26.
  Smart Way toUse Oracle ML 1. Write SQL using 18c in-database analytics 2. Document and annotate using the HTML, URL, image markdown tags 3. Build interactive visualizations using markdown tags 4. Collaborate and share • Version notebooks for team work • Share entire notebook with report consumers • Export specific paragraphs or visualizations 27
• 27.
  Oracle ML KeyConcepts And Getting Started
• 28.
  Key Concepts • Workspace oFolderfor organizing projects oControls user access to projects • Project oFolder for organizing and storing SQL notebooks, SQL scripts and jobs • Notebook oContains one or more paragraphs oIncludes data visualizations (tables and graphs) oTwo built-in shortcuts • SQL Query Scratchpad • SQL Script Scratchpad 29
• 29.
  Key Concepts forNotebooks • Interpreters o Bindings that connect a notebook to ADW resource group (high, medium, low) o Markdown is automatically enabled • Paragraph o Contains a single SQL statement (default) or • SQL Query offers multiple data visualizations o Contains a SQL script (%script) o Contains markdown tags + text (%md) 30
• 30.
  Logging into Oracle MachineLearning Logging in and using the Oracle ML home page
• 31.
  • Pop-out mainmenu • Shortcuts, links, history 32 • Online help Oracle ML Home Page
• 32.
  33 Editor Area (Show/Hide) Output Area (Show/Hide) ParagraphNotebook
• 33.
  34 Paragraph menu area •Run/refresh paragraphs • Show/hide code • Show/output • Settings Notebook menu area • Run/refresh all paragraphs • Show/hide code • Show/output • Clear output • Clear notebook • Export notebook • List of Shortcuts • Interpreter bindings • Report type
• 34.
  Using ADW Resource Interpreters Connectingto high, medium or low ADW resource groups
• 35.
  ADW Resource Interpreters •Three interpreters linked to three ADW resource groups: o High o Medium o Low • Normally only need to select two interpreters: • md – to access markdown tags • ADW resource group interpreter 36
• 36.
  Interpreters Linked toDatabase Resource Groups • 3 pre-defined database services o Choice of performance and concurrency • HIGH o Highest resources, lowest concurrency o Queries run in parallel • MEDIUM o Less resources, higher concurrency o Queries run in parallel • LOW o Least resources, highest concurrency o Queries run serially No of concurrent queries Max idle time CPU shares HIGH 3 5 mins 4 MEDIUM 20 5 mins 2 LOW 32 1 hour 1 Example for a database with 16 OCPUs
• 37.
  Machine Learning andADWC 38 The Perfect Combination What is Machine Learning Introduction to Autonomous Data Warehouse Cloud Service Introduction to Oracle Machine Learning Running SQL Statements and Scripts Creating Interactive Notebooks Scheduling Notebooks to Refresh Conclusion
• 38.
  39 SQL Editor Area (Show/Hide) OutputArea (Show/Hide) Paragraph
• 39.
  Documenting SQL Using MarkdownTags Using markdown tags to document your SQL statements
• 40.
  OML Markdown KeyConcepts • Set of predefined markdown tags oIncludes basic HTML tags, e.g. <H1>, <H2>… • Markdown tags can be used in: oColumn selection oSetting aggregation methods oWHERE, GROUP BY, ORDER BY clause oFROM clause • Note: Markdown is always parsed before SQL is executed oCan’t create dynamic input to populate structure of markdown tag 41
• 41.
  Simple Documentation MarkdownTags • Useful tags for dynamic content generation ohttps://sourceforge.net/p/zeppelin/wiki/markdown_syntax/ oLinks • [click here for more information](http://your-specific-url) oReference Links • Use references to [link first URL][1] and like to another URL like [this][2] with hover-text [1]: http://url-link-1 [2]: http://url-link-2 ”My hover-over text message" oImages •  oBasic formatting tags includes HTML heading tags and font-format tags: • *use this for italic* • **use this for bold** • ***use this for bold and italic*** 42
• 42.
  43 Image tag URL linktag Reference link tag HTML tag Markdown binding Paragraph title
• 43.
  44 Copyright © 2018,Oracle and/or its affiliates. All rights reserved. | 19
• 44.
  Machine Learning andADWC 45 The Perfect Combination What is Machine Learning Introduction to Autonomous Data Warehouse Cloud Service Introduction to Oracle Machine Learning Running SQL Statements and Scripts Creating Interactive Notebooks Scheduling Notebooks to Refresh Conclusion
• 45.
  OML Markdown AdditionalKey Concepts • Set of predefined markdown tags oIncludes basic HTML tags, e.g. <H1>, <H2>… • Markdown tags can be used in: oColumn selection oSetting aggregation methods oWHERE, GROUP BY, ORDER BY clause oFROM clause • Note: Markdown is always parsed before SQL is executed o Can’t create dynamic input to populate structure of markdown tag 46
• 46.
  More Sophisticated MarkdownTags • Useful tags for dynamic content generation ohttps://zeppelin.apache.org/docs/latest/manual/dynamicform.html#select-form oSimple input box • ${formName} templates. oSelect form/pulldown selection • ${formName=defaultValue,option1(DisplayName)|option2(DisplayName)...} oCheckbox • ${checkbox:formName=defaultValue1|defaultValue2...,option1|option2...} oURL Links • [text-for-link] <http://url-to-content> 47
• 47.
  48 Building More SophisticatedInteractive Reports
• 48.
  49 Documenting Interactive Reports
• 49.
  Download Sample Notebooks •Sales Analysis series of notebooks oBased on sales history schema (SH) oIncludes 5 notebooks: • Overview, Channel Analysis Dashboard, Customer Analysis Dashboard, Product Analysis Dashboard, adhoc analysis o Download links - https://www.dropbox.com/s/2if07dja88y8s1u/Sales-Analysis.zip?dl=0 • Customer Insight Analytics series of notebooks oMachine learning using sales history schema (SH) oIncludes 8 notebooks covering machine learning examples o Download links - https://www.dropbox.com/s/47oj06yw4g0e654/Deep-Customer-Analytics.zip?dl=0 50
• 50.
  Machine Learning andADWC 51 The Perfect Combination What is Machine Learning Introduction to Autonomous Data Warehouse Cloud Service Introduction to Oracle Machine Learning Running SQL Statements and Scripts Creating Interactive Notebooks Scheduling Notebooks to Refresh Conclusion
• 51.
  Why Schedule aNotebook to Refresh? • Data for each visualization is cached within notebook. • After data loads/changes notebook are not updated. • Two ways to refresh a notebook: 1. Manually refreshing the queries 2. Using a job to schedule when notebook should be refreshed 52
• 52.
  Scheduling a Notebookto Refresh • Enter name for job • Select the notebook to refresh • Set start date + time • Set frequency of refresh • Determine when refresh ends: oX number of runs oX number of errors oTimeout is reached 53
• 53.
  Scheduling a Notebookto Refresh 54 • Job status page gives snapshot of refreshes • Each run records the details of each error within notebook • “View” button opens notebook o See next slide
• 54.
  Viewing Details ofRefresh Job 55
• 55.
  Machine Learning andADWC 56 The Perfect Combination What is Machine Learning Introduction to Autonomous Data Warehouse Cloud Service Introduction to Oracle Machine Learning Running SQL Statements and Scripts Creating Interactive Notebooks Scheduling Notebooks to Refresh Conclusion
• 56.
  Oracle Machine Learning •Easy o Web based SQL notebook integrated into ADW o Autonomous setup and configuration o Sample gallery of notebooks: machine learning • Flexible o SQL access to 18c in-database analytics o Markdown tags for dynamic visualizations • Collaborative o Collaboration and sharing built-in o Notebooks support versioning SQL
• 57.
  Where to getmore information…
• 58.
  Where to getmore information • Product information: cloud.oracle.com/datawarehouse • Documentation oADW: https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse- cloud/index.html oOracle ML: https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse- cloud/omlug/getting-started-oracle-machine-learning1.html • Hands-on Workshop ohttps://oracle.github.io/learning-library/workshops/journey4-adwc 59
• 59.
  Where to getmore information • New Q&A Forum on Cloud Customer Connect ohttps://cloudcustomerconnect.oracle.com/resources/32a53f8587/summary • Forbes: Autonomous Capabilities Will Make DBAs More Valuable ohttps://www.forbes.com/sites/oracle/2018/03/21/autonomous-capabilities-will- make-data-warehouses-and-dbas-more-valuable/#73b134c6624e 60
• 60.
  Email to: [email protected] AnyQuestions?