DDDP 2019 - Brown to Green
1 like•304 views
The document discusses strategies for improving open source communities and addresses challenges faced in edge computing, including security and technical debt. It highlights the importance of modern infrastructure, automated platforms, and the integration of IoT with enterprise applications to support intelligent decision-making. Additionally, it outlines the necessary functionalities for next-generation architectures, such as secure connectivity, machine learning capabilities, and end-to-end system management.
DDDP 2019 - Brown to Green
- 1. How can we better create OSS communities? 1 Source: My Life Through a Lens @bamagal Unsplash
- 2. Next generation of Technical Debt 2 Source: @ScottAdamsSays Dilbert.com Technical Debt Source: Bart Copeland ActiveState : Technical Debt in High Tech - A CEO’s Perspective
- 4. 4 Brown to Green A Open Platform for Evolutionary Architecture - Connecting a Secure Edge to Intelligent Apps REDHATOPENENERGY John Archer Principal Cloud Platform Solution Architect Red Hat - Energy [email protected]
- 5. Where Innovation is Happening 5 Source: Red Hat success story. “BP modernizes infrastructure, introduces DevOps with self-service platform” accessed May 2019. Source: ExxonMobil - Red Hat Summit keynote video from Summit 2019 The combination of microservices, containers, and a fully automated CI/CD platform provides what developers have been asking for for years. They now have full self-service to deliver change from the initial idea, through the innovation, right through to production, as quickly as humanly possible. Paul Costall Head of Application Engineering Services, BP
- 6. Source: Giancarlo Revolledo Unsplash Islas Ballestas, Peru IT/OT BRIDGE STILL NOT A REALITY 6
- 7. EDGE SECURITY NOT TAKEN SERIOUSLY 7
- 8. SECURE EDGE CHALLENGES 8 ● Cultural ● Insecure ○ Most deployed edge solutions today are at risk ● Data ownership is mixed ● Sunset of 2.5G/3G moving to LTE-M and 5G ● Costly ○ Are metered by tags by traditional SCADA vendors ● Intelligent backhaul still difficult
- 9. DATA SCIENCE CHALLENGES 9 ● Cloud GPUs at needed scale still expensive ● Many efforts stuck on desktops ● Data ○ Can not find curated and authoritative data ○ What they can find is random quality, latency, complete ○ Lacking metadata standards and pedigree ● Most teams still maturing their ML/AL/DL/RL ● Very crowded and fast moving space ● No one wants to be called a Citizen Data Scientist
- 10. 10 KEY FUNCTIONALITY FOR AN END-TO-END ARCHITECTURE Securely connect, authenticate and manage disparate connected devices that speak different protocols Apply analytics at the edge with machine learning and business rules to enable local, low-latency decision making Centralize IoT data processing, analytics and machine learning to enable deep business insights and actionable intelligence Enable integration with enterprise and business applications to bridge the gap between OT and IT and reduce complexity Tools to enable end-to-end data security, compliance, authorization and authentication Device Management & Connectivity Intelligent Edge Processing & Analytics Advanced Analytics & Machine Learning Business & Application Integration End-to-End Security & Compliance
- 11. RED HAT OPEN INNOVATION LABS 1 1 BRIDGING THE SILOS Accelerating innovation CONTEMPORARY NON-LINEAR TRADITIONAL LINEAR
- 12. CAPABILITIES ENGINEERED TO WORK TOGETHER CODE CONTAINER NATIVE PLATFORM MESSAGING API RULES BUSINESS AUTOMATION FUNCTION AS A SVC
- 13. ● Measured Boot ● Secure Boot ● Full End-to-End Disk Encryption ● Signed Containers ● Bashless OS ● OS and Firmware Over-the-Air Style updates WHAT MAKES UP A NEXTGEN SECURE EDGE OS? 13
- 14. SECURE EDGE / INTEGRATION HUB MQTT AMQP 1.0 Edge Private or Public Cloud HTTP LoRaWAN CoAP Enterprise & AI/ ML Applications Device Registry Telemetry Commands
- 15. SECURE EDGE / INTEGRATION HUB MQTT HTTP LoRaWAN CoAP Customer Applications Broker Streams Custom Code Device Registry
- 16. YOUR DIFFERENTIATION DEPENDS ON YOUR ABILITY TO DELIVER INTELLIGENT APPS FASTER CONTAINERS, KUBERNETES, DEVOPS & DATAOPS ARE KEY INGREDIENTS Innovation Culture Cloud-native Applications AI & Machine Learning Internet of Things Virtual GPU
- 17. CONNECTING THE EDGE TO DATA SCIENTISTS Highly Scalable, flexible, elastic, microservice based architecture Fully Portable – On Premise to any public cloud vendor Leverages the power and agility of open source software without lock-in Architecture Tenets Data Scientist Data Manager s Citizen Data Scientist Cognitive AI Vision Speech Face Audio Video Text Data Models Curation Prep Quality Publishing SecurityPython, R, Jupyter.org, Tensorflow, Keras, Pandas, Bokeh, Dash, Prometheus, Grafana, SciPy, NumPy, SumPy, Julia , Spark, PySpark, Theano, Scikit, FaceDetect Packages: AI/ML/Data Science Pods MongoDB, MariaDB, mySQL, Postgres, Couchbase, Redis, MS-SQL, OraclePersistence : SSOandAuthentication OIDC SAML OAuth JWT Kerberos DevOps Node.js, .Net Core, Java, Python, PHP, Ruby, Rails, Javascript, PerlApp Dev: AppDev & App Services and Persistence Pods REST ODBC JDBC WS Predictive Maintenance Autonomous Operations Supply Chain Improvements Downstream Reliability Use Cases Multitenant – CPU and GPU powered workloads REST IoT “Things” MQTT Integration, BPM, Rules, Messaging, API, IoT, Microservices, IstioApp Services: OnPremise Public Cloud WSS Kafka
- 18. DATA SCIENTIST DEVELOPERS NEEDS All Developers need ● Choice of architectures ● Choice of programming languages ● Choice of databases and persistence ● Choice of application services ● Choice of development tools ● Choice of build and deploy workflows Data Science Additional Needs ● Access to GPUs and varied storage ● Access to Curated Data ● Automated ScienceOps pipelines ● Collaboration with the Business ● Access to specific data science languages and toolsets They don’t want to have to deal with the infrastructure.
- 19. MATURING INTO A DATASCIENCEOPS PIPELINE Seeing an emerging notion of Data ScienceOps workflows. Data Prep includes: Ingest, ETL, Pedigree, MetaData, Quality Polyglot Models for ML Training, Data Streaming and Data at Rest concerns Dynamic behaviors depending on compute/GPU/Storage/Memory resources
- 20. MACHINE LEARNING ON OPENSHIFT Unique performance computing requirements for Artificial Intelligence, Machine Learning, Neural Networks and GPUs Multiple Data Science images: • TensorFlow • Pyro/PyTorch • Scikit-learn • CNN/GANs • Keras • Seldon • RAPIDS.AI • Apache Arrow
- 22. MULTI CLOUD OBJECT GATEWAY 22 App Multi-Cloud Buckets Multi-site Buckets S3 APIApp App Hybrid Buckets DEPLOYAND MANAGE DATA SERVICES
- 23. EFFICIENCY AND SECURITY BY DEFAULT S3 Write Fragment Dedupe Encrypt StoreCompress Paris DC London DC New York DC
- 24. HIGH LEVEL ARCHITECTURE 24 HIGH LEVEL ARCHITECTUREApplication meta-data NooBaa Core Optimized utilization, resilience, performance, locality, economics, etc. Heartbeats Instructions Instructions Scalable NooBaa Storage Node stores chunks, runs lambda functions, monitors host, sends heartbeats Scalable NooBaa Endpoint S3 / Lambda API auth, chunking, dedupe, compress, encrypt Cloud Resource AWS S3 AWS S3-compatible, Azure Blob, Google cloud storage Data path
- 25. ● Fedora IoT ● JupyterHub on Openshift ○ Jupyter notebook, JupyterHub, JupyterLab, Openshift Templates ● Kubeflow ○ Kube project for Tensorflow, JupyterHub/Lab, PyTorch, MPI Operator ● Opendatahub.io ○ Ceph, Spark, JupyterHub/Lab, Tensorflow ○ Simplified Multiple Kernel notebook support ○ Simplified GPU Support ○ Resource management and instance culling ● RAPIDS.AI - Open GPU Data Science packages ○ CUDF, DASK, XGBoost, PyTorch ● Seldon.io - Machine Learning Models OSS EDGE and DATA SCIENCE PROJECTS
- 26. ● Join Openshift Commons - Energy and ML SIGs https://commons.openshift.org/ ● Openshift Self Service Education https://learn.openshift.com ● Install Minishift https://docs.okd.io/latest/minishift/getting-started/installing.html ○ MacOS - brew cask install minishift ○ Manual - https://github.com/minishift/minishift/releases ● Install Jupyter and JupyterHub Openshift templates ○ https://github.com/jupyter-on-openshift/jupyterhub-quickstart ● Review the OpenDataHub.io project ● Email me at [email protected] HOW CAN I GET STARTED?
- 27. CONFIDENTIAL Designator linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat 27 Red Hat is the world’s leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you OPTIONALSECTIONMARKERORTITLE
- 28. 28 COMPREHENSIVE CONTAINER SECURITY Container Content Container Registry CI/CD Pipeline Deployment Policies Security Ecosystem CONTROL Application Security DEFEND Infrastructure EXTEND Container Host Multi-tenancyContainer Platform Network Isolation Storage Audit & Logging API Management 28
- 29. BEST EDGE COMPUTING FOUNDATION?
- 30. KNATIVE SERVERLESS BUILDING BLOCKS Build A pluggable model for building artifacts, like jar files, zips or containers from source code. Serving An event-driven model that serves the container with your application and can "scale to zero". Eventing Common infrastructure for consuming and producing events that will stimulate applications. "...an extension to Kubernetes exposing building blocks to build modern, source-centric, and container-based applications that can run anywhere".