Ekon21 Microservices - Event Driven Design

Microservices – Event Driven Systems
Arnaud Bouchez - Synopse
MICROSERVICES
Event Driven Design

• Arnaud Bouchez
– Delphi Expert
• Various solutions (from Vatican to gaming industry)
• IoT solution (RSI)
• Real-time Monitoring solution (LiveMon)
– Open Source
• mORMot (SOA ORM MVC framework)
• SynPDF SynMustache SynDB SynCrypto
– Training and consultancy
• Synopse one-man company

Event-Driven Systems
• Using Events?
• Event-Driven Design
• Event-Driven mORMot

Using Events?
• Imperative Architecture
– Events are high-level commands

Using Events?
• Easy modelization into regular code
procedure TZen.HandleIAmGoingOut(Sender: TPeople);
var temp: TTemperature;
begin
temp := TemperatureSensor.GetTemperature;
if temp > Threshold then
RoboValet.FetchDownJacket(Sender);
end;

Using Events?
– State is stored in a main shared DB
translated into SQL statements

Using Events?
– State is stored in a main shared DB
translated into SQL statements
• DB-centric approach
• Atomicity is ensured, but history is lost
• DB becomes a bottleneck

Using Events?
• Event-Driven Architecture
– Collaboration is done using Events

Using Events?
– Collaboration is
done using Events
Just like
VCL/FMX /LCL !

Using Events?
– System consists in uncoupled nodes
which could be added or removed on-the-fly
– Nodes publish events, and subscribe to them
– Each node stores the information it needs
– Some API gateways expose state via REST

Event-Driven Architecture
• Nodes implemented as services
procedure TZen.Init;
begin
TemperatureSensor.SubscribeToTempChange(OnTempChanged);
end;
procedure TZen.OnTempChanged(NewTemp: TTemperature);
begin
CurrentTemp := NewTemp;
end;
begin
if CurrentTemp > Threshold then
RoboValet.NotifyNeedsJacket(Sender);
end;

Not natural to imperative code (Delphi, C#, Java)
– System consists in uncoupled nodes
Architecture becomes complex
– Nodes publish events, and subscribe to them
May be subject to unexpected loops
– Each node stores the information it needs
Duplicated data, with eventual consistency

• More complex than imperative code
var temp: TTemperature;
begin
temp := TemperatureSensor.GetTemperature;
if temp > Threshold then
RoboValet.FetchDownJacket(Sender);
end;

• Events may be received for no use
e.g. the following method is always called
procedure TZen.OnTempChanged(NewTemp: TTemperature);
begin
CurrentTemp := NewTemp;
end;
but this value may never be used in
begin
end;

• Danger of unexpected infinite loop
begin
end;
What happens
if RoboValet.NotifyNeedsJacket
triggers (indirectly) TZen.HandleIAmGoingOut ?
Since nodes may be added at any time,
how to prevent it to happen?

• Danger of unexpected infinite loop
begin
end;
What happens
if RoboValet.NotifyNeedsJacket
triggers (indirectly) TZen.HandleIAmGoingOut ?
Enter the Event-Driven debugging hell:
actual event process depends on the system it runs on

• So, when to use it?
– Your are modeling evolving information
• e.g. scale to a lot of devices or users
– You need to track changes of state
• A monolithic DB only stores the current state
(via Update) - otherwise it may sink under Inserts
• Event-Sourcing and CQRS (see later)

– You expect a modular design
• Process not written in stone during compilation
(as in imperative programming): add/enable nodes
to change the system behavior at runtime
• Microservices, SaaS, horizontal scaling

– You defined a stateless domain
• From state comes complexity,
so DDD usually modelizes a time snapshot
• DDD events are a way to introduce temporality
– Never 100% Event-Driven
• You will eventually define REST API gateways

• Message Bus
– Most common way to implement Event-Driven
• Cloud, Local or in-house Queues
• Like VCL TApplication.ProcessMessages
– Publish/Subscribe mechanism
• Nodes get only needed information
• Nodes could be added/removed
– Used in conjunction with regular DB and REST

• Message Bus

• Peer to peer communication
– Nodes communicate directly between them
• Each node manages its own queue(s)
• Good integration with REST
• Need a centralized catalog service, P2P ZeroMQ,
or convention-over-configuration setup
• As offered by mORMot over WebSockets

• Event Sourcing
– Business logic is implemented through Events
– Events are persisted, not state
• Play the events to compute a state
• Intermediate states may be cached/persisted
– Events are transmitted, not state
• Publish/Subscribe instead of REST
• Define API gateways for the outer world

• Event Sourcing
– Business logic is implemented through Events
– Convenient way to define Microservices
• Subscribe to events, to get the information
• Publish events, as result of the process
– Microservices persistence
• Most Microservices will use a regular state storage
• Some Microservices will use Event Sourcing

• Events and CQRS
– Command Query Responsibility Segregation
• Commands triggers events
• Events are gathered and stored
• State/Events store is uncoupled from Commands
– CQRS follows a non CRUD pattern
• Create Read Update Delete

• CRUD
DB

• CQRS
DB

Event-Driven mORMot

Event-Driven mORMot
• Interface-based services
type
ICalculator = interface(IInvokable)
['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
/// add two signed 32 bit integers
function Add(n1,n2: integer): integer;
end;
– Defines the contract of the service
– Microservice: should follow SOLID principles

Event-Driven mORMot
procedure ResolveCalculator(out: ICalculator);
...
function TMyClient.MyAdd(a,b: integer): integer;
var Calculator: ICalculator;
begin
Client.Services.Resolve(ICalculator, Calculator);
result := Calculator.Add(a,b);
end;
– Runtime injection of the actual implementation
• May be in-process, local or remote

Event-Driven mORMot
var Calculator: ICalculator;
begin
Client.Services.Resolve(ICalculator, Calculator);
result := Calculator.Add(a,b);
end;
– Client is a mORMot TSQLRestClientURI
– Will communicate using JSON over HTTP(S)

Event-Driven mORMot
• RESTful Local and Client-Server
– In-process
Stand-alone client, fast server-side access
– Named pipes or Windows messages
Stand-alone client, fast server-side access
– HTTP/1.1 via kernel-mode http.sys API
• Kernel-mode execution, IOCP driven, https – Windows specific
– HTTP/1.1 via OS socket API
• Windows or Linux, using a Thread Pool, https via Reverse Proxy
• Upgradable to WebSockets

Event-Driven mORMot
• RESTful Client
TSQLRestServer
TSQLRest
TSQLRestClientURI
TSQLRestClient

Event-Driven mORMot
• RESTful Server
TSQLRestServerDB
TSQLRestServer
TSQLRest
TSQLRestServerRemoteDB TSQLRestServerFullMemory

Event-Driven mORMot
• RESTful Local and Client-Server
– Needed methods are available at TSQLRest
• REST auto-routing
• JSON high-performance serialization
• Following Liskov Substitution Principle
– Focus on the logic
– Abstract from the plumbing

Event-Driven mORMot
• REST/JSON Persistence via ORM/ODM
– Agnostic Storage for Events or State
• ORM over local SQlite3 store
• ORM over remote SQL databases
• ODM over TObjectList
• ODM over MongoDB
Switching from one to another
in a few lines of code, or through settings

Event-Driven mORMot
• REST/JSON Persistence via ORM/ODM
– Advanced ORM features
• REST Convention-Over-Configuration routing
• Direct DB-to-JSON serialization (no TDataSet)
• Built-in statement and JSON cache
• Batch (Unit-Of-Work) writes: bulk SQL/NoSQL insert
• SQL logging/timing, real-time replication,
data sharding, remote proxying

Event-Driven mORMot
• Cross-platform
– Main mORMot units
• Delphi: Windows x86/x64
Best IDE experience for coding and debugging
• FPC: Windows, MacOS, Android, Linux, BSD
x86/x64, ARM32/AARCH64 (PPC32/PPC64)
• (Cross)Kylix: Linux x86

Event-Driven mORMot
• Cross-platform
– Generated client code using Mustache templates
• Delphi: Windows, MacOS, Android, iOS
• FPC: Windows, MacOS, Android, iOS, Linux, …
• (Cross)Kylix: Linux
• SmartMobileStudio: JavaScript Ajax / HTML5
– Featuring almost all framework abilities
• JSON marshalling, security, instance lifetime

Event-Driven mORMot
• Interface-based services callbacks
– Specify the callback as an interface parameter
• Native way of coding in object pascal
– Real-time push notifications over WebSockets
• Upgraded from a standard HTTP connection
– Peer To Peer communication
• No need of a centralized message bus / server

Event-Driven mORMot

Event-Driven mORMot
– Specify the callback as an interface parameter
ILongWorkCallback = interface(IInvokable)
...
end;
ILongWorkService = interface(IInvokable)
['{09FDFCEF-86E5-4077-80D8-661801A9224A}']
procedure StartWork(const workName: string;
const onFinish: ILongWorkCallback);
function TotalWorkCount: Integer;
end;

Event-Driven mORMot
• Once upgraded, communicates using frames
over a bi-directional socket connection
using application-level protocols
• Security, frames gathering, REST emulation

Event-Driven mORMot
• Once upgraded, communicates using frames
over a bi-directional socket connection
using application-level protocols

Event-Driven mORMot
– Security, Frames gathering, REST emulation
• TWebSocketProtocolBinary = SynLZ + AES-256
• Regular blocking methods expecting results
will be emulated like regular REST requests,
with REST security and parameters marshalling
• Non-blocking methods with no result
will be asynchronously sent,
optionally gathered as “jumbo frames”

Event-Driven mORMot
– Publish/Subscribe Pattern
also known as “Observer”
Publisher
Subscriber 1
Event
Subscriber 2
Event
Subscriber 3

Event-Driven mORMot
– Beware of “Race Conditions”
• Use critical sections (e.g. TSynLocker)
to protect your shared data on multi-threaded server
• You can gather all non-blocking callback process in a
background thread via TSQLRest.AsynchRedirect

Event-Driven mORMot
– Beware of “Dead Locks”
• If your callback triggers another method
which shares the same critical section
in another thread, you may block both threads
• You can gather all non-blocking callback process in a
background thread via TSQLRest.AsynchRedirect

Event-Driven mORMot
– Sample 31
• Long-Work Push Notification
also known as “Sagas”
• Publish/Subscribe Pattern
also known as “Observer”

• Credits
Martin Fowler
https://martinfowler.com
code and samples at
https://synopse.info

Event-Driven Systems
• Questions?

Ekon21 Microservices - Event Driven Design

More Related Content

What's hot (20)

Similar to Ekon21 Microservices - Event Driven Design (20)

More from Arnaud Bouchez (20)

Recently uploaded (20)

Ekon21 Microservices - Event Driven Design