Streaming with Oracle Data Integration
3 likes2,350 views
The document discusses streaming transformations using Oracle Data Integration, emphasizing the importance of real-time data processing and its applications, such as retail analysis and IoT data analysis. It outlines the benefits of Oracle Data Integration, specifically its scalability, and interaction with Apache Kafka and Spark Streaming for efficient data handling. Additionally, it highlights the ease of setup and management provided by Goldengate in facilitating data integration from various sources.
Streaming with Oracle Data Integration
- 2. • Michael Rainey - Technical Advisor • Spreading the good word about Gluent products with the world • Oracle Data Integration expertise • Oracle ACE Director • mRainey.co 2 Introduction we liberate enterprise data
- 4. • The processing and analysis of structured or “unstructured” data in real-time • Why Streaming? • When speed (velocity) of data is key • Streaming data is processed in “time windows”, in memory, across a cluster of servers • Examples: • Calculating a retail buying opportunity • Real-time cost calculations • IoT data analysis 4 What is “Streaming”
- 8. • Scalable, fault-tolerant, high-throughput stream processing • Spark Streaming receives live input data streams from various sources • Continuous stream of data is known as a discretized stream or DStream • Data is divided into mini-batches and processed by the Spark engine • Operations such as join, filter, map, count, windowed computations, etc are used to transform data in-flight 7 Stream processing - Apache Spark
- 10. • Enterprise has invested heavily in ODI and/or GoldenGate • Getting started with development languages (Python/pySpark, Java, etc) • Centralized metadata management • Integrate with other data sources using a single interface • Realized cost savings • According to Gartner, 200% increase in maintenance costs when custom coding (https://www.gartner.com/doc/3432617/does-customcoded-data-integration-stack) 9 Why Oracle Data Integration?
- 14. • GoldenGate • …is non-invasive • …has checkpoints for recovery • …moves data quickly • …is easy to setup 12 Why GoldenGate with Kafka?
- 15. • Heterogeneous sources and targets • Built to integrate all data • Flexibility • Reusable code templates (Knowledge Modules) • Reusable Mappings • ODI can adapt to your data warehouse - and not the other way around • Flow based mappings 13 Why Oracle Data Integrator with Spark Streaming?
- 17. • Standard GoldenGate Extract / Pump processes to capture RDBMS data • Replicat for Java parameter file & process group created and setup • Kakfa Producer properties and Kafka Handler configuration setup 15 Oracle GoldenGate for Big Data - Kafka Handler Setup
- 18. • Kafka handler properties • Set properties for how GoldenGate interacts with Kafka • Format, transaction vs operation mode, etc • Kafka producer configuration 16 GoldenGate for Kafka setup http://mrainey.co/ogg-kafka-oow
- 21. • Create Model using Kafka Logical Schema • Create Datastore • Similar to standard “File” datastore, define file format and setup columns • Only support for CSV • Future formats may include JSON, Avro, etc • Add Datastore to mapping 18 Kafka and Oracle Data Integrator
- 22. • Create Spark Data Server, Physical / Logical Schema • Set Hadoop Data Server • Add properties, such as checkpointing, asynchronous execution mode, etc • Additional properties can be added: http://spark.apache.org/docs/latest/configuration.html • Spark Server is setup as Staging location • Source Datastore from Kafka, Oracle DB, etc • Target Datastore is Cassandra, Oracle DB, etc • Code generated by KM is pySpark • pySpark code can be added to filters, joins, other components for transformations • Additional languages (Scala, Java) may be coming soon 19 Spark Streaming and Oracle Data Integrator
- 25. • Streaming is the “velocity” in data. AKA “Fast Data” • Oracle Data Integrator and Oracle GoldenGate provide a framework for development and management of data streaming processes • Big Data add-ons continue to support new technologies • Build a streaming architecture using GoldenGate and ODI: • Metadata management • Integration of RDBMS data with “schema on read” data • Build upon the skills in-house 22 Recap
- 26. 23 we liberate enterprise data thank you!