Innovating With Data and Analytics
1 like353 views
1. The document discusses how organizations can leverage data, analytics, and insights to fundamentally change and pioneer new business models. 2. It emphasizes that data analytics cannot be accomplished in a silo and must involve the entire organization. Modern cloud platforms, software methodologies, and data tools are needed. 3. Examples are provided of how various organizations have used tools like Pivotal Greenplum to gain insights from data to solve problems in areas like predictive maintenance, risk management, and national security.
Innovating With Data and Analytics
- 1. Innovating with Data and Analytics Jacque Istok @jstok
- 2. © Copyright 2017 Dell Inc.2 The New Normal DATA DEVICES Law Enforcement Media Banks Delivery Services Marketers Government Private Investigators /Lawyers Individuals Employers Data Users/Buyers Analytic Services Advertising Catalog Co-ops List Brokers Websites Information Brokers Credit BureausMedia Archives Data Aggregators FINANCIAL GOVERNMENT PHONE/ TV INTERNETMEDICAL RETAIL
- 3. 3© 2017 Pivotal Software, Inc. All rights reserved. Great organizations leverage software, analytics, and insights to take better actions and fundamentally change and pioneer entirely new operational business models
- 4. Analytics Within Digital Transformation Cannot Be Accomplished In A Silo
- 5. © 2017 Pivotal Software, Inc. All rights reserved. Open Source Innovation Massively Parallel Processing Data Science and Machine Learning Multi-Cloud and Flexible Deployment Agile Development Our View on Next Generation Data Platforms
- 6. DATA IS IMPORTANT How can my organization capitalize on this trend?
- 7. Using Our Process For Solving Analytics Pair Programming / Solutioning Retros Iterative Development Greenplum Open Data Platform ANSI-compliant SQLStandups User Centric Data Science
- 8. TRANSFORMING INTO A MODERN SOFTWARE COMPANY Building high-quality software at start-up speed requires modern software methodologies, cloud platform, and data tools Modern Cloud Native Platform 01 Modern Software Methodology 02 Modern Data Tools 03
- 9. Modern Cloud-Native Platform Building high-quality software at start-up speed requires modern software methodologies, cloud platform, and data tools 01
- 10. Provide the Business With Solutions “You’ve got to start with the customer experience and work back toward the technology - not the other way around.” - Steve Jobs 02
- 11. Modern Software Methodologies A modern platform requires a modern approach to software development Building a strategy for product development Building a quality product at startup speed An organization optimized to respond to disruption 1MODERN SOFTWARE METHODOLOGY Agile Software Methodologies 2MODERN SOFTWARE METHODOLOGY Lean Startup Techniques 3MODERN SOFTWARE METHODOLOGY Optimized for Change 02
- 12. METHODOLOGIES TIP Start with a technology solution and look for a problem to solve Start with a business problem or opportunity and map to technology solution 02
- 13. User Centered Design “A design approach that supports the entire development process with user-centered activities, in order to create a product that is easy to use and of added value to the intended users.” www.usabilitynet.org 02
- 14. Is It Useful? usage = value rarely used = waste 02
- 15. © Copyright 2017 Dell Inc.15 03
- 16. ANALYTICAL APPLICATIONS NATIVE INTERFACES MULTI- STRUCTURED DATA SOURCES & PIPELINES Structured Data JDBC, ODBC SQL ANSI SQL USERS FLEXIBLE DEPLOYMENT Local Storage Other RDBMSes SparkGemFire Cloud Object Storage HDFS JSON, Apache AVRO, Apache Parquet, XML, & More Teradata SQL Other DB SQL Apache MADlib ML/Statistics/Graph Python. R, Java, Perl, C Programmatic Apache SOLR Text PostGIS GeoSpatial Custom Apps BI / Reporting Machine Learning AI IT Dev Business Analysts Data Scientists On-Premises Public Clouds Private Clouds Fully Managed Clouds MODERN CLOUD ANALYTICS PLATFORM KafkaETL Spring Cloud Data Flow Massively Parallel (MPP) PostgresSQL Kernel Petabyte Scale Loading Query Optimizer (GPORCA) Workload Manager Polymorphic Storage Command Center SQL Compatibility (Hyper-Q) Modern Cloud Analytics Platform03
- 17. Common Business Problems With Potential for High ROI Using Data FRAUD MANAGEMENT RISK MANAGEMENT CYBERSECURITY MANUFACTURING PREDICTIVE MAINTENANCE ELECTRICITY GRID
- 18. Modern Data Tools Common Functional Requirements 1. High speed ingestion (sensors, financial transactions, etc) 2. Data consolidation (join and cross reference) 3. SQL for structured data 4. Process higher level data constructs: Graph, Geospatial, Text 5. SLA driven queries on big data 6. Real time access for consumers and applications 03 CAPTURE ANALYZE APPLY
- 19. Modern Data Tools Common Non-Functional Requirements 1. Open Source Software 2. Platform Agnostic 3. Production Proven 4. Scale out Growth 5. Security Centric 6. Cloud Ready OPEN PRODUCTION PROVEN SCALE OUT SECURE 03
- 20. Pivotal Data Suite Maps to Common Business Requirements Pivotal Greenplum Comprehensive Open source Data Warehouse Pivotal HDB Open source Hadoop Native, high-performance, scale-out SQL analytics - Apache HAWQ Pivotal GemFire Open source application & transactional database based on Apache Geode ------- BIG DATA ------- ------- FAST DATA ------- 03
- 22. High Speed Ingestion Data Capture Problem: Fast incoming data needs to be captured Example use cases: 1. Financial transactions 2. Sensors from industrial equipment 3. People using public resources: tracking on Bridge/Tunnel, etc Data needs to be captured fast in memory Data needs to be modified, consolidated, updated in memory Data needs to be streamed and loaded to consolidated platform Sensor Data
- 23. Hot Analytical Processes That Need Fresh Data Overnight ETL / ELT jobs can be too slow App 1 App 3 App 2 Transactional System High Speed Micro Batching TRANSACTIONS ANALYTICS• Analytical processes need access to the latest data • ETL/ELT processes are expensive and hard to maintain • A continuous flow of input keeps analytics up to date MPP
- 24. Consumer Facing Analytics Recommendation Engine Push Problem: Big Data Analytics Access for Consumers Example use cases: 1. eCommerce 2. Financial Recommendations 3. Entertainment Online Big Data Needs to be Stored & Analyzed Machine learning algorithm generates recommendations Consumer scale app needs real time recommendation Consumer Specific Recommendations
- 25. Operationalized insights need to be real time Combination of SQL Analytics and NoSQL event-driven transactions is needed App 1 App 3 App 2 Real Time System TRANSACTIONS ANALYTICS• Data Insights must be immediately pushed to applications • Apps should be able to react in real-time to analytical findings Machine Learning Advanced Analytics ANSI SQL APIs / NoSQL Data Insights MPP
- 26. Exemplified Use-Case - Predictive Maintenance Evaluates live data “According to historical trends, there’s an 80% chance this equipment would fail in the next 12 hours” Learns with historical trends "How were the temperature and vibration sensors reading when the latest failures happened?" Live data becomes historical over time Sensor data Historical Real-time Take action Smart system
- 27. Reference Architecture Sensors Update with findings Consumer Application Machine Learning Real-time data streams Apply Improve Management Dashboard Reporting
- 28. Global Energy Corp. – Predictive Maintenance Challenges • Failing turbines causing issues with power generation • Unable to store & process fire-hose of data Solution • High velocity data ingestion from Turbines • Store PBs of turbine data • Machine learning and SQL analytics • very low latency and high speed data access Time EquipmentCondition Broken Cost to repair Failure starts to occur Early signal 1 Early signal 2 Early signal 3 Audible noise What we want to avoid
- 29. Telco Network Quality KPIs 29 Fast Data Ingest and Real-time Analytics Challenge: • Provide network quality KPIs to Brazil's National Telco Agency • Capture, analyze and guarantee data consistency for millions of data points / sec. • Enable near real-time analytics over the high volume and fast data being transacted Solution: • Pivotal GemFire and Greenplum allow combined fast low-latency in-memory transactions and near real-time analytics • Real-time data insights on cell phone network quality KPIs • Data Science engagement for improving the network quality
- 30. Firm Wide Risk Management Customer: US Based Top Investment Bank Business Problem Global platform for storage, query and analytics of firm’s trades, orders, assets, and risk Challenges High speed incoming transactions Petabytes of data storage Advanced machine learning and data science Operational risk fed back to trading systems real time Firm wide risk view to executive team Solution Gemfire for high speed transaction data Greenplum for storage, reporting, analytics Live feedback loop
- 31. National Security Customer: Mid Size Country With Security Concern Government Problem Need to monitor national security for protection of citizens Challenges Petabytes of data storage Analysis of structured data, geospatial data, text data and social graphs in consolidation Need real time response for public safety officers Solution Greenplum for data consolidation and joining Greenplum for geospatial, text, graph and machine learning Gemfire for distribution of analytics result to field officers
- 33. Breakfast • Gets people into the office at the same time • People tend to have lunch at the same time • Maximizes pairing and collaboration time
- 34. Pair Programming ● Two heads are better than one ● Combination of experience ● Continuous code review ● Great for training ● Rotation of pairs → no knowledge silos
- 35. Continuous Integration ● Information radiators ● Test Driven Development ● Source of truth ● Smart dependency test graph ● Unified code base
- 36. Standup ● Minimize the meetings ● Focus on shared understanding ● Share concerns ● Encourage continuous improvement
- 37. 1. Data is a growing asset 2. Need tools to derive business value from data 3. Platform Assumptions: Security, Open Source, Platform Agnostic, Scalable, Performant 4. Agile Methodology: fast feedback, learn, adapt 5. Let’s work together to add value Key Points Summary
- 38. Pivotal Greenplum: Learn More Find out more about Pivotal Greenplum at https://pivotal.io/pivotal-greenplum OR learn more about the open source at http://greenplum.org/ OR give it a try yourself at Amazon AWS or Microsoft Azure or via Download