Microservices Patterns with GoldenGate

Microservices Patterns with GoldenGate:
Deep-dive into Microservices Patterns with Replication and Stream Analytics
O R A C L E D E V E L O P M E N T , M A Y - 2 0 2 0

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Today’s Agenda
Microservices and the Challenge of Data
Data Mesh Concept, How GoldenGate Fits
Microservice Patterns with GoldenGate
Anyone Remember “Complex Event Processing”?
Copyright © 2020 Oracle and/or its affiliates.
1
2
3
4

Microservices are Good!
4
Service Mesh Revolution
Emergence and widespread use of microservices have
directly led to revolution in DevOps, massive uptake in
Kubernetes and by 2020 the Service Mesh revolution
Key Benefits:
• Decomposition, of monolithic architecture
• Modularity, smaller services and improved
speed of initial development
• Independence, loosely-coupled systems that can
be created using different languages or data
• With the loose coupling also comes much
greater flexibility around deployment and
upgrades, eliminating complex dependencies
• Flexibility at Scale, deployments may start small,
run locally and later scale very wide, running
across multi-cloud environments and containers
• Sidecar pattern for “Mesh” frameworks are
further revolutionizing microservices
Chris Richardson: https://www.nginx.com/blog/introduction-to-microservices/

Dogma of Received Wisdom at the Data Tier
5
https://docs.microsoft.com/en-us/dotnet/architecture/microservices/architect-
microservice-container-applications/data-sovereignty-per-microservice
• “DB per Service” is a founding concept of Microservices
Architectures
• Some architects are more dogmatic than others:
• “All microservices must have an isolated data store” vs. “Some
microservices may share a data store”
• At scale, usually means that microservice application
must accept “eventual consistency” for data tier
Graph of Microservices vs. Classical App Architecture
“Drawbacks”: https://www.nginx.com/blog/introduction-to-microservices/
https://microservices.io/patterns/data/database-per-service.html
Encapsulate the data to simplify system
dependencies, create loose-coupling and
empower a good CICD/DevOps process!
Does this pattern always make sense?
Mr. Microservice

6
https://en.wikipedia.org/wiki/Rorschach_test
• Service Granularity, DDD offers a weak heuristic for defining the domain
scope of a microservice (decomposition is hard)
• Enterprise Domains are Big, for larger applications (ERP etc) domains are
inherently large. domain decomposition can create artificial, subjective
boundaries that proliferate incompatible business ontologies
• Business Processes are Cross-Functional, a given business process may
cut across many data domains, corporate business units and even 3rd party
entities altogether (shared kernels and domain aggregates are needed)
• Context Matters, reasonable differences of object definitions and
taxonomy will naturally occur across large organizations (context
mappings)
• Legacy Applications, they exist. And, they don’t fit neatly into idealistic
Bounded Context mappings (bubble contexts for Legacy)
Context mapping is more art than science:
http://www.ouarzy.com/2016/07/25/micro-
service-and-bounded-context-clarification/
Context is in the mind of the modeler:
However…
Domain Driven Design (DDD) principles guide developers to create microservices that align to Bounded
Contexts, which “defines tangible boundaries of applicability of some sub-domain”
https://microservices.io/patterns/decomposition/decompose-by-subdomain.html
Challenges with Bounded Context & DDD
Sounds hard!

Ledger
Service
Command
Model
Domain
Event
Hardest Part About Microservices: Your Data
7
Availability & Durability – what happens with the
data when there is a service failure? How is the state
of data saved and restored? (non-volatility)
CAP Theorem & Network Partitions (CP or AP)
• Strong Consistency – which services require guaranteed
consistency, recovery and a single record of the truth?
• Eventual Consistency – if a microservice end-point is
unavailable, what is the retry protocol? Do all the
services in the Saga/Process adhere to the same
transaction semantics for retry and rollbacks?
ACID – what is the business impact of data
transactions ending in an inconsistent state? Will
business processes fail? Are there any regulatory or
compliance impacts? (ACID vs CAP, Google F1)
findOrderHistory()
findOrder()
Order Events
GL EventsShipping
Events
Logistics
Service
Command
Model
Domain
Event
Order
Service
Command
Model
Domain
Event
Order
History Service
Command
Model
Event
Handlers
Event
Handlers
Event
Handlers
Event
Store
many microservice design patterns
have been created specifically to
try and compensate for the data
consistency guarantees that
databases inherently provide
Eg: CQRS and Event Sourcing
DB encapsulation seems
to shift the very hard data
consistency issues to the
App Tier…
just accept “eventual consistency” and “lack of isolation” for transactions, and we’ll all be ok!

Data and Events are First-Class Citizens
8
Service Service Service
Analytics, Data Science and Data Lakes are too important to my
business’ Digital Transformation and Data-Driven initiatives…
we need architecture focus on Data and Events too
Application Microservices
Data Stores
Event Logs
produce events
consume events
produce events
consume events
read write
App
Events
Data
Events
DB Log
Events
Control Plane
• “State of the Truth”
at a point in time
(current or historic)
• Durable storage used
for Data Recovery /
Archives/years of data
• Polyglot, each service
may determine its own
data structures
• “Narrative of the Truth”
sequence of events
(between data snapshots)
• Days/months of event
data available as Time
Series or Messaging
• Strict ordering of events
& Idempotency
• Strong Consistency of
DB logs (eg; when using
GoldenGate)
• “Systems of Record”
at application tier
• APIs, business rules
and business object
semantics
• Not durable storage
The microservice API is king!

Can we take some of the best ideas of a Service Mesh and apply them to
Data and Events, to create a kind of Data Mesh?
Microservice Mesh and Data Mesh
9
Immutable
Raw Data Events Prepared Data Canonical Data
App
Events
Data
Events
DB Log
Events
Data Domain
Projection …n
Data Domain
Projection 1
Application
Microservices
Control Plane

What is a Data Mesh?
11
Microservice
Patterns
Log-based
Integrations
Polyglot Data
Movement
Data Mesh is a data-tier architecture to integrate and
govern enterprise data assets across distributed multi-
cloud environments
Microservices-centric:
• For the administration, deployment and monitoring of the core
frameworks of data movement and governance
• Aligns with for Service Mesh frameworks (K8S, Istio, etc)
• “Sidecar Proxy” style pattern for Events and Data
Immutable event-logs for data integrations:
• Messaging and data store events are globally accessible via
immutable event logs
• Logs may be used to drive Streaming or Batch integrations
Distributed data movement of all types of data
• A data mesh moves data: Relational, NoSQL, JSON, Graph…
• Relational data consistency (ACID) during data movement
• Must work reliably with enterprise OLTP data sets
https://en.wikipedia.org/wiki/ACID
Data
Mesh
Event
Streaming
Immutable
Logs
Data
Replication
Polyglot
Persistence
Edge / 5G
Frameworks
Domain
Driven
Design
Service Mesh
“Sidecars”
Data
Mesh

Evolution towards Real-Time Data Mesh
Industry 3.0: Hub and Spoke Transitional: Kappa Hub Mature: Distributed Kappa
This data pattern, popularized by Ralph
Kimball and Bill Inmon, has been the
foundation for enterprise data
management since 1993.
It is transaction consistent, can scale up
nicely for most use cases, and is based on
SQL, lingua-franca for most tools.
By 2010, the Lambda (big data) pattern
was common. In 2014, Jay Kreps (of
LinkedIn) questioned the Lambda
Architecture and spawned Kappa.
The Kappa principles consider batch
processing as a special case of stream
processing. Use a historized event log to
process both real-time as well as batch
processing.
By 2020, IT infrastructure has
dramatically changed – networking,
containers, cloud, compute, IoT etc have
all pushed data to the edge.
A mature Kappa architecture is not a
single instance “hub” but rather a
distributed mesh of data logs, stream
data processing, change events, and time
series data.
Kappa: https://www.oreilly.com/radar/questioning-the-lambda-architecture/
https://en.wikipedia.org/wiki/Dimensional_modeling
mesh & microservice controls
12
ETL
ETL
ETL
ETL
Lambda: http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html
Monoliths Distributed

Data Mesh Conceptual View – Data Domains
13Copyright © 2020 Oracle and/or its affiliates.
Enterprise Data
Producers:
ERP Apps, DBs,
Middleware etc.
Data Domain
Consumers
People owners of “Data
Products”, collections of
data sets in various
stages of curation
IoT Data
Producers:
Devices & Things
Raw Data
Prepared Data
Canonical Data Data
Domain A
Data
Domain B
f(x)
f(x)
Data
Domain C
Data Mesh
(distributed Kappa, microservices, cloud agnostic)
Domain-Specific Views of Data
Raw Event Consumers
Automated Devices,
Edge Nodes (5G), Scheduled
Routines (eg; ETL etc)
Data Product-Specific
Storage Choices:
• RDBMS
• Data Lake
• Object Store
• Graph, etc.

Raw data, Time Series & Alerting events are pushed
Direct to Database (high fidelity transaction semantics fully preserved)
Consumer-Driven, Event-Centric Data Mesh
Enterprise Data
Producers
Detect
Event
Logical
Change
Records
(LCRs)
App
DB
committed!
CDC Replication
Data Domain
Consumers
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Raw
Data
Prepared
Data
Canonical
Data
Raw Data (LCR)
Schema Events
(DDL)
Prepared
Data Topics
“Master”
Data Topics
JSON, XML,
Avro, Parquet,
CSV
Prepared data events are pushed
Canonical data events
Speed &
Fidelity
Trusted
Views
Ease of
Consumption
LCR/TFs
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
Data Mesh puts the consumer
needs first – they require data
at different latency, fidelity,
trust levels and views
Data Model
Object Model
System
Of Record
(SoR)
User
Action
App APIs and
system log events
14

Direct to Database (high fidelity transaction semantics fully preserved)
Distributed by Design, Microservices Based
Data Domain
Producers
Detect
Event
Logical
Change
Records
(LCRs)
App
DB
committed!
Data Domain
Consumers
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Data Model
Object Model
System
Of Record
(SoR)
User
Action
CDC Replication
Microservices
Edge Compute
or Cloud for
Raw Data
Events
Prepare
Technical Data
Views
LCRs
Business
Data Views
Raw data, Time Series & Alerting events are pushed
Prepared data events are pushed
Canonical data
Events
(ephemeral or persisted)
Stream
Process
Events
(persisted)
Stream
Process
Events
(persisted)
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
15

Single Pane of Glass for Real-Time Data Mesh
connect
DB2/z
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
Real-Time Stream
Data Processing
Raw
Data
DBAs &
Data Engineers
Data Owners &
Data Products
16
Data Consumer DrivenEvent Centric Pipelines
Deploys in a Mesh
Across Containers, Public Clouds and 5G Edge Devices

How it Works Today: GoldenGate for Big Data
Data Domain
Consumers
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
BYOS (Bring Your Own Spark)
* distributed, may run on any combination of containers and clouds
17
Data Engineer Data AnalystDBA/GG Ops
Capture Pipeline Analyze DeliverIngest
GoldenGate Microservices Applications Stream Analytics Application
BYOM
(Bring Your
Own Messaging)All Data Events
& Transactions

What Exactly Does GoldenGate Do?
Database
DBA
ETL
Tools
continuous stream of events
Database Events:
• Transaction Commits (DML)
• Schema Changes (DDL)
• Procedure Execution
Any Data Store Target
• Databases (any)
• Non-Relational
• NoSQL, HDFS, etc
• Search (Elastic…)
• Cloud (Obj Store…)
• Messaging
• Kafka, JMS etc.
• Cloud (any)
GoldenGate
Trail (protocol)
Online Redo Logs
18
App
Services
S
Q
L
S
Q
L
S
Q
L
GoldenGate Replicats, Event
Handlers, and Data Formatters:
Core Concept:
DB Event Log
Customer
Data Formats
Business Events:
Committed Events
1
2
3
4

First Things First – GoldenGate Microservices
20
data replications
bi-directional
ms/sec updates
consistency guarantees
Cloud, Containers or Edge Devices
Extracts Replicats Client Libraries
Native GUIs and Full REST APIs
API Gateway or Proxy Service
GG GUI
GoldenGate is itself a set of microservices
that human users or other services may
interact with
Embedded User Interface:
• C-based microservices with embedded HTTP client for
native JavaScript based GUI
• Oracle Jet frameworks for intuitive interaction model
REST native APIs:
• Fully REST native
• Also available, a Command Line Interface (CLI) produces
REST calls to the native services
Full GoldenGate Replication Capabilities:
• 100% coverage for all traditional GG replication
patterns; fully capable of HA/DR use cases
GG
Admin
Service
GG
Distro
Service
GG
Metrics
Service
GG
Receiver
Service
GG
Service
Manager
Your
Services

Microservice Design Patterns for Data
21
Patterns for MicroservicesInherent to the Microservice Architecture is the developer
using specific patterns, sometimes the patterns are partially
embodied in a Programming Framework, but typically the
developers must choose to follow certain heuristics while
programming.
This presentation’s focus:
• “Database Patterns” & “Integration Patterns” …using DB Event
Replication (AKA: Change Data Capture) to improve them
• Simplify the pattern, make the microservice application more resilient
and provide better data consistency guarantees
DB Patterns for Discussion:
• Database per Service (covered earlier)
• CQRS – Command Query Responsibility Segregation
• Event Sourcing
• Saga Pattern
• Transactional Outbox
• Aggregates (AKA: Domain Events)
Transaction
Outbox
https://medium.com/@madhukaudantha/microservice-architecture-and-design-patterns-for-microservices-e0e5013fd58a

Transaction Outbox
https://microservices.io/patterns/data/transactional-outbox.html
App
Tables outboxApp
TablesApp
Tables
JSON
Payload
Order
Service
Object
Storage
Update DB *and* Send Domain Events:
• Keep the DB Update and Event as atomic
Other Context:
• “Message Relay” should be Idempotent / Exactly Once in order to
assure that the recipient of the events are not receiving duplicates
• “Message Relay” concept may use (1) polling events, or (2) tail
the DB transaction logs (AKA: Changed Data Capture, CDC)
XYZ
Service
Using Oracle GoldenGate:
• …GG is an exactly-once “Message Relay” microservice
• …also adds benefit of maintaining Transaction Consistency
• …millisecond latencies, trusted as a DR solution for OLTP
• …flexible “payloads” and “broker” variations (see next page)

Baseline Tx Outbox Pattern (microservices.io) Outbox Without the Middleman (No Broker)
Outbox Payload is Tables (not Documents) Outbox from Materialized View
Transaction Outbox Variations
outboxApp
Tables
JSON
Payload
Order
Service
Message
Broker
outboxApp
Tables
JSON
Payload
Order
Service
XYZ
Service JSON
Payload
JSON
Payload
App
Tables
Order
Service
Doc
Payload
Message
Broker
App
Tables
XYZ
Service
JSON
Payload
Object
Storage
Doc
Payload
• Full fidelity database raw data
is needed
• Sharding, Data Science,
ODS/EDW, etc…
• Data binding becomes shape of
table rather than shape of
Service-defined doc
• GG Formatters put Table data
into Broker as JSON, Avro,
Parquet, XML, etc.
• Payload centric, may be used
with Aggregates pattern
• Developer of microservice
defines Payload signature
• Best for smaller services
• See also: CQRS and Event
Sourcing
• Point-to-point replication rather
than Pub/Sub
• Orchestration-based Sagas
• Singleton use cases; eg; always
copying service events to a data
lake
• Useful for micro-monoliths and
monoliths; DB’s with very large
or very 3NF schema
• Views are “interface tables” for
external data consumers
• Views present business-friendly
data objects
• Views must be maintained by
DBAs
Object
Storage
JSON
Payload
App
Tables
Order
Service
View
Tables
Object
Storage
View
Tables
Doc
Payload
Doc
Payload
XYZ
ABC

Transaction Outbox for the Whole Enterprise
DB2/z
Replication of
Real-time Data
Transactions & Events
GoldenGate Stream Analytics
ETL
&ML
Object
Storage
Relational
Non-
Relational
Apps
https://www.oracle.com/middleware/technologies/goldengate.html
Microservices-
Based
DBMS
Cloud
Big Data
NoSQL
Streams

Order
History
Logistics
Service
Order
Service
Ledger
Service
CQRS Pattern
Command Query Responsibility Segregation
Query
Model
Command
Model
SQL Select
Read Side
Write Side
Inserts,
Updates etc.
Basic CQRS (https://martinfowler.com/bliki/CQRS.html)
findOrderHistory()
findOrder()
Event
Handlers
Multi-Service Example (https://microservices.io/patterns/data/cqrs.html)
GL Events
Shipping Events
Order Events
• Separation of duties:
• Writes (Commands) – go against Domain Model
• Reads – optimized for client Queries
• Alternative to typical “CRUD” type interfaces for
Microservice API / signatures
• Popular when:
• Complex Domains – require queries against many
domain objects at same time
• High Performance – to support highly optimized read
interfaces
• Complex Domains:
• Multiple services must update common “view”
• Clients (GUIs) may run queries that require many
joins and/or lots of records fetched
write read
User Interface
GUI

Order
History
Order
Service
Simplified CQRS with Transaction Outbox
findOrderHistory()
findOrder()
Delegate the data event propagation:
• Instead of having the developers (of the
microservices) write the logic, handlers, and
failure conditions for eventing → leverage data-
tier event replication to propagate the DML
events to the “view” data store
Avoid complicated and costly mistakes:
• CAP Theorem is a hard problem to code for in the
microservice itself:
• What happens if the service fails just after
a local update but before the event is
sent?
• What if the “view” Service receives the
event but the data store fails?
• What if the event volume / workload
spikes and the “view” Service goes down?
Data tier “logical replication” is designed to be very fast, work across
WAN/Clouds and to maintain consistency:
• Relational semantics may involve long-running transactions, foreign
key dependent DML operations, and strict transaction ordering
requirements
write
User Interface
ms/sec updates
consistency guarantees
Order Events
read

Order
Service
Logistics
Service
Ledger
Service
Order
History
Simplified CQRS Pattern
findOrderHistory()
findOrder()
Event
Handlers
Order EventsCommand
Model
Command
Model
Command
Model
Event
Store
Event
Handlers
Event
Handlers
GL Events
Shipping
Events
Domain
Event
Domain
Event
Domain
Event
Before After
• Developer responsible for correct
implementation of CAP Theorem
• Data consistency on any outage
(microservice / data tier)
• More code to write and maintain
• Worse performance (latency)
• Impedance, across many services into
view model (semantic misalignment)
✓ Consistency and recoverability
guarantees
✓ Less code to write or maintain
✓ Best possible performance
write read
User Interface User Interface
Order
History
Logistics
Service
Order
Service
Ledger
Service
write read

Event Sourcing Pattern
BIG topic in the Microservice community:
• …and, a point of confusion as well
Pattern espoused for:
• An alternate durable data store (instead of a Database) to
store a services’ state, and/or… In combination with a local
Database, where the Event Store is use primarily for
communication of events (and not for saving state)
• A way to record a sequence of immutable app events to be
used later by the service which generated the events,
and/or (when Event Store is shared) by other services who
wish to subscribe to events that other services create
Implementations:
• DIY using your favorite language; purpose-built event
stores like Eventuate, Axon, Akka or Event Store
• Apache Kafka, Apache Pulsar, RabbitMQ, or NoSQL engines
As a Durable Store ++ For Event Communications (only)
Local Events Shared Events
Event Store
➢ ALL events…
➢ Order Created
➢ Order Approved
➢ …
➢ Order Shipped
Event Store
all durable
events
➢ only the
➢ events that
➢ need to be
➢ communicated
➢ ….
Order
Events
Logistics
Events
Ledger
Events
Event Store
Order
Events
Logistics
Events
Ledger
Events
Event Store
Circa 2005: https://martinfowler.com/eaaDev/EventSourcing.html
• https://microservices.io/event-sourcing.html
• https://microsoft.com/patterns/event-sourcing
https://dzone.com/articles/microservices-with-
cqrs-and-event-sourcing
https://www.infoq.com/news/2017/11/eve
nt-sourcing-microservices/
Order
Service
Order
Service
Order
Service
Logistics
Service
Ledger
Service
Order
Service
Logistics
Service
Ledger
Service

Critiques of Event Sourcing
Exposing the Persistence Tier:
• Taken too far (Why Event Sourcing is an Anti-Pattern), developers wind up using the
Event Store as a Shared Persistence model, and other microservice now have hard-
coupled binding to the message formats of the originating service
Whole System Fallacy:
• Some microservices leaders (Udi and Greg Reach CQRS Agreement) say to narrow
the aperture on when to use CQRS + Event Sourcing → only within a Business
Component and a Single Bounded Context
• Minimizes utility of pattern for Communications
Forcing Eventual Consistency on Developers:
• The propensity to over-use CQRS & Event Sourcing at the at the whole system level
forces developers to manage eventual consistency in the Application tiers (What
they don’t tell you about event sourcing)
• “…they will make your life a living hell” doing DevOps, debugging and
system recovery when a “Mesh” of services are interacting via Event Store and
message signatures can lead to disaster

Event Sourcing with Transaction Outbox
Before After
Service Service Service
Application Microservices
Data Stores
Events
produce events
consume events
produce events
consume events
read write
App
Events
Data
Events
DB Log
Events
Control Plane
Data Mesh
https://dzone.com/articles/microservices-with-cqrs-and-event-sourcing
• Microservice developers must code for consistency
• Data may be stale or incorrect
• Many tiers for failure, difficult to trouble-shoot
• Useful for “communications” and “shared events” requirements
• Delegate data tier consistency to “Events” tier
• Using DB Replication tools (eg; GoldenGate) for ACID
• Events may be used “system-wide” rather than per Context
per ‘bounded context’ and not used system-
wide for communications
Domain z
Domain X
Domain Y

Saga Pattern
https://livebook.manning.com/book/microservices-patterns/chapter-4/17
Popular, long-standing pattern:
• First paper back in 1987 (Garcia-Molina and Salem, 1987)
• Used for long-lived transactions in distributed systems, that cannot
be completed using multi-phase commits
In Microservices:
• Frequent requirement due to “one DB per service” pattern
• Multi-phase commits at microservice tier are uncommon
• Typically see either (a) Choreography or (b) Orchestration based
approaches (https://microservices.io/patterns/data/saga.html)
Drawbacks:
• Lack of Isolation (ACD not ACID) and Eventual Consistency lag, system
state can be inconsistent for seconds, minutes or hours
• Complex programming model, depends on “Compensating
Transactions”
• Requires other Patterns (eg; Event Sourcing, Aggregates) for
maintaining DB consistency
Additional logic for
compensation /
rollbacksChoreography or
Orchestration logic

We don’t want to lose any events
and leave the system inconsistent.
How to atomically update the
store and send the event?
Saga Pattern with Transactional Outbox
Ticket
Service
Insurance
Service
Payment
Service
Outbox Pattern
Outbox Pattern
Outbox Pattern
Event Hub
Ticket
Topic
Order
Topic
Payment
Topic
Publish Events:
Subscribe Events:
Red Hat Summit 2019: The New Era of Transactions…
Saga Approach:
• Choreography
Outbox with Saga:
• Publish committed events to event hub
(eg; Kafka or other); very low risk of
inconsistency due to network or service
failures (Dzone: Data Consistency…)
• GoldenGate enforces Idempotency /
Exactly Once and may preserve intra-table
Consistency as well
Subscribers:
• All services in the Saga can be assured
that events in the Hub are fully committed
• Compensating Txs happen per normal
Saga pattern guidelines

Broker
Aggregate is a collection of objects grouped together:
• Typically grouped by Transaction boundaries
• Used for local persistence
• *within* a specific Bounded Context (always!)
Motivation:
• A core purpose of this pattern (within a specific context) is to
ensure that invariants and domain rules are always applied to the
whole transaction holistically
Domain Event (Aggregate) vs Integration Event
Domain Service
Aggregate 1
Aggregate 2
Root Entity
Entity Value
Domain Event
GUI Domain Context xyz
Order
Events
Chris Richardson: Using Aggregates with Events and CQRS
Integration Event is a collection of objects grouped together:
• Also, typically grouped by Transaction boundaries
• Used mainly for events and external communications
• Usually designed for bindings outside of local Context
See Also:
• This pattern is mainly focused on the organization of data posted
via Event Sourcing, Saga, or Transaction Outbox
Domain Service
Aggregate 1
Aggregate 2
Root Entity
Entity Value
Domain Event
GUI Domain Context xyz
Microsoft Patterns: Using Integration Events (with Event Brokers)
Integration Event

Non-Relational
Integrations
Data Lake Ingest Streaming Ingest Cloud Ingest Messaging Replication NoSQL Replication SaaS Replication
DB Event
Replication
Unidirectional Bi-Directional Peer-to-Peer Broadcast Consolidation Distribution
Event Stream
Processing
Data Pipelines Data Transformation GoldenGate Integrations Time Series Analysis Geo-Fencing Predictive Analytics
Integration Events with GoldenGate
Copyright © 2020 Oracle and/or its affiliates. 34

Hype Cycle 2009
Complex Event Processing:
• CEP is a kind of computing in which incoming data about
events is distilled into more useful, higher level event data
that provides insight into what is happening. […] CEP is
used for highly demanding, continuous-intelligence
applications that enhance situation awareness and
support real-time decisions. Gartner
20 Years Too Early?
• CEP dates back to the 1990’s (history of CEP engines)
• CEP came before “Event Stream Processing” and generally
has covered more complex use cases (eg; handling of out-
of-order events and more complicated correlation
semantics) (what’s the difference, 2019 and mythbuster
CEP vs ESP, 2008)
• Largely overtaken by Big Data stream processing
technologies that are open-source, massively-parallel, and
widely available as cloud-native
2009!

Stream Processing 2020
Time
Series
DB/OLAP
Big Data Event
Stream Processing
Complex
Event
Processing
Becoming more aligned to open source
/ apache frameworks
Becoming more capable of rich windowing functions
and time-clock correlation semantics
Complex Event Processing:
• Traditionally running in “scale-up”
SMP in-memory framework
• Many CEP engines are moving
toward MPP “scale-out”
architectures
• Programming-centric
historically,
some CEP
engines
were
also query-
centric as far
back as 2009
• Time-clock semantics
and advanced cache
may be used for “state
machine” type use cases
Stream Processing:
• Built around MPP frameworks and
typically Apache open-sourced
• Genesis was around simplistic use
cases on high volumes of events
• All SP engines began with
rudimentary windowing and
correlation semantics, but most
frameworks are gradually
becoming more functionally
comparable to classic CEP
• Simplistic windowing and caching
for basic stream-clock events
Time Series Databases:
• Optimized for persistence and historic analytics
on time-ordered events…data is often sourced
from CEP or SP engines
Technical and Functional Differences between CEP and SP: https://complexevents.com/2019/07/15/whats-the-difference-between-esp-and-cep-2/

Stream Processing/CEP for Event Driven Architectures
There has been a widespread
awakening to the benefits of Event
Drive Architecture (EDA) for
increasing the scalability and agility of
business systems. […] Stream
analytics is based on the mathematics
of complex-event processing (CEP).
CEP is a computing technique in
which incoming data about what is
happening (event data) is processed
as it arrives (data in motion or
recently in motion) to generate
higher level, more useful, summary
information (complex events).
W. Roy Schulte (of Gartner), March 2020:
EDA is Suddenly Popular Will Stream Analytics be Next?
Event Stream Analytics (& CEP)
Data & Microservice Events
Event/Data
Pipelines
Time-Series
Analysis
Geospatial
Analysis
Real-time
AI/ML
Continious
ETL
Use Cases:

Oracle GoldenGate Stream Analytics
39Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2018
Ingest Events Select Processing Patterns Build Event Pipelines Serve Data Downstream
GoldenGate events from any DB or
NoSQL; also supports events from
Kafka, JMS, AQ and MQTT
Rich set of pre-built patterns will
dramatically improve developer
efficiency and time-to-value
Easily leverage geo-fencing,
machine-learning, and other
reference data within the stream
Data can be delivered out to Kafka,
databases, or easily staged for
external ETL jobs
Data Owners &
Data Products

Significant Intellectual Property
More than 70 patents on stream processing
Mature tech stack for Event Processing
Over 10 years of IP investment
Continuous Query Language (CQL)
12.2
12.3
18c
19c
11g

GoldenGate Stream Analytics
Interactive, Low-Code Designer Time Series Analytics Event Driven Dashboards
Patterns / Accelerators Geo-Spatial Analysis & Geo-Fencing Predictive Analytics / ML

Upgrade to Spark, Better than Flink or KSQL
[best] [worst] GoldenGate
Spark
Streaming
Apache
Flink / Beam
Confluent
KSQL
User Experience
Low Code Development (with built-in patterns/accelerators)    
Interactive/Live Edits (browser based, view changes immediately)    
Built-in Live Dashboards (event-driven charts/graphs)    
Core Streaming Semantics
What is Being Computed (transforms, joins, flatten, statefulness etc)    
Time Windows(global, fixed, sliding, tumbling, custom etc)    
When in Processing Time (triggers – event, time, count, timers, etc)    
How do Refinements Relate (discarding, accumulating, retracting)    
Analytics
Robust CEP Capabilities (complex event correlations, native time clock)    
Geo-Fencing & Spatial (lat/long, built in maps, custom map tiles, etc)    
Machine Learning (native scala, PMML, python support etc)    
Time Series Analysis (built-in interval patterns, thresholding etc)    
Other Features
Backpressure (dynamic ingest per pipeline) Automatic Custom Custom Custom
State Management (automation across streams & native cache) Data Grid N/A RocksDB RocksDB
Data Consistency (OLTP Change Events, Inserts/Updates/Deletes) Automatic Custom Custom Custom
GoldenGate Stream Type (aware of SCN/CSN, transactions, order, etc) Native Custom Custom Custom

Today’s Agenda
Microservices and the Challenge of Data
Data Mesh Concept, How GoldenGate Fits
Microservice Patterns with GoldenGate
Anyone Remember “Complex Event Processing”?
Conclusion!
1
2
3
4
5

GoldenGate Stream Analytics for Data Mesh
connect
DB2/z
Data
Objects
Table
Data
Raw Data
/ Alerts
SQL
Consumers
Applications,
Data Services
Biz Consumers
Analytics &
Data Marts
Data Science
& Streaming
Applications
DBAs for HA,
DR and OLTP
Real-Time Stream
Data Processing
Raw
Data
DBAs &
Data Engineers
Data Owners &
Data Products
44
Data Consumer DrivenEvent Centric Pipelines
Deploys in a Mesh
Across Containers, Public Clouds and 5G Edge Devices

This is not a Metamorphosis, it is a Paradigm Shift
The Success Paradox
Data success factors that did well in
Industry 3.0 will not be the factors
that create success in Industry 4.0
Next Gen Data Architecture
ETL Vendors
1990 – 2010’s Gen1 :
• Replication
• Messaging
• Streaming
• Pipelines
Next-Gen has
new DNA not
tied to old ETL tools
It is impossible to evolve older Batch
Processing tools into a modern Event-
Centric Stream Processing solution; the
underlying paradigms are fundamentally
different
45

Microservices Patterns with GoldenGate

Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Microservices Patterns with GoldenGate

More Related Content

What's hot (20)

Similar to Microservices Patterns with GoldenGate (20)

More from Jeffrey T. Pollock (20)

Recently uploaded (20)

Microservices Patterns with GoldenGate