Microservices, Node, Dapr and more - Part One (Fontys Hogeschool, Spring 2022)

Classificatie: vertrouwelijk
Microservices in
real life –
with Node &
Dapr.io
March 2022
Lucas Jellema
Architect & CTO AMIS | Conclusion
labs: https://github.com/lucasjellema/fontys-2022-microservices-kafka-dapr

Microservices in real life – with Node & Dapr.io
Founded in 1991 by students
from University of Twente –
Aircraft Maintenance Information
System (AMIS)
80 colleagues,
located in
Nieuwegein
the core of what we do:
working with Data.
partnering with peers
and companies in
several countries
Lucas Jellema (2002)
Cloud Solution Architect & CTO
lucas.jellema@amis.nl | technology.amis.nl | @lucasjellema | lucas-jellema

Data Engineering, Data Analytics
(& Data Science)
Data &
Application
Integration
web
applications
Internet of Things
cloud, DevOps, PaaS,
streaming, microservices,
Software Studio, database,
software engineering
Oracle, Microsoft Azure,
open source, Java, SQL,
NodeJS, Python,
React/Angular

and you are invited to come do an internship and explore job opportunities with us!

Classificatie: Public
Publiek
Moving On. Our Main Story today (and next week)
• Applications
• more specifically:
Microservices – the best form of applications we have come up with so far
• How they are designed
• How they interact
• How they can be implemented
• Using:
• Dapr.io – distributed application runtime
• Node – popular, full stack, omnipresent programming language
• Apache Kafka – distributed message broker, enterprise backbone

Publiek
Overview
• Part One (today)
• Microservices recap
• Dapr.io – personal assistant for applications & distributed application runtime
• Handson with Dapr.io
• Quick Intro to programming in Node[.js]
• Handson Microservice implementation with Node and Dapr.io
• Part Two (next week, 30 March)
• Asynchronous interactions through Apache Kafka
• Handson with Apache Kafka and Node
• Real world use cases and scenarios with Apache Kafka
• Handson Multi-microservice set up with Apache Kafka, Node and Dapr.io

Publiek
Assumed is
• A development environment with VS Code, Docker and the ability to install
• Knowledge of
• HTTP, REST, JSON
• Containers, Docker (and Kubernetes)
• Java or C#
• SQL and a database (MySQL or PostgreSQL or SQL Server)
• perhaps Message Broker/Event Queue (RabbitMQ?), Cache (Redis?)
• Cloud fundamentals
• Microservices concepts

Publiek
What is an Application?
• Collection of (business) functionality
• implemented with COTS, code, platform & infra
standard custom
Compute (Containers, VMs, bare
metal servers)
Storage Network
Database
IAM/
LDAP
Event / Message
Broker
Web
Server
API
Gateway
Data Center Power Cooling
Custom
hardware
Custom
platform
Operating System
Language Runtime
use as is
out of the box
(pre-)integrated
application
platform, to be
custom configured
best of breed
applications per
functional domain.
(custom
integration)
low code tools
for workflow,
automation,
integration en
frontends
high code technology
for bit & pixel perfect
software with tailored
fit (functionally & non-
functionally)

Publiek
What is a microservice?
• Why is a microservice?
• Problems with (BIG) applications
• complex
• inflexible to change (when roll out changes)
• long test, long build
• impossible to upgrade technology (everything is affected)
• hard to scale
• hard to migrate (to different technology)
• hard to learn (for new staff)

Publiek
Final Demo
• Microservice

Publiek
What is a microservice?
• Not TOO big (but not too small either)
• understandable by normal human beings – easy to on board new team members
• easy to change – allow for agile functional and non functional evolution
• functionally cohesive/logical collection of features => domain
• design time decoupled, clear standards based external interfaces: API & events
• low external impact of: change, refactor, move, migrate, scale, failure
• reliable, robust, resilient
• clear ownership (business sponsor; one IT team builds it, runs it, evolves it)
• contains everything needed to run [on a generic platform]
• application software, libraries/frameworks, platform configuration
• pipelines for CI and CD, tests, documentation, infra as code

Publiek
What is IT all about?
Application
Production Runtime

Publiek
Application
Production Runtime
Platform
Business
value

Publiek
Application
Platform
Production Runtime
Operations
Monitoring &
Management
Platform
Platform

Publiek
Production Runtime is the Result of Preparation Runtime
Application
Platform
Production Runtime
Operations
Monitoring &
Management
Application
Preparation Runtime
Platform
Development
CD
Agile Design,
Build, Test

Publiek
One team has Agile responsibility through full lifecyle
Application
Platform
Production Runtime
Operations
Monitoring &
Management
Application
Preparation Runtime
Platform
Development
CD
Agile Design,
Build, Test

Publiek
One team has Agile responsibility through full lifecyle
Application
Platform
Application
Platform

Publiek
A microservice is owned by one team through its full lifecyle
Microservice

Publiek
The Microservice
API
HTTP REST/JSON
µ

Publiek
Independent microservices and an integrated front end

Publiek
Independent microservices and an integrated front end
microservice / team

Publiek
The Microservices Platform
Runtime Platform
API
µ
API
µ
API
µ
API
µ

Publiek
Cloud Native Application Development- on Oracle Cloud | APAC OGB Tour 2021
power cooling
rack
space
physical real estate
IAM
Certificate
Management
Vulnerability
Scanning
Logging
Monitoring Auditing
Cloud Usage
Analysis
Resource
Manager
APIs
Cloud
Events
Notifications Cloud Guard
Tagging
Search
Serverless
Functions
Container
engine
API
Gateway
Load
Balancer
Container
Registry
Artifact
Repository
Job
Scheduling Build & Deploy
Pipelines
Code
Repository
Web Application
Firewall
CDN
Virtual
Machines
API
µ

Publiek
power cooling
rack
space
IAM
Certificate
Management
Vulnerability
Scanning
Logging
Monitoring Auditing
Cloud Usage
Analysis
Resource
Manager
APIs
Cloud
Events
Tagging
Search
SQL
Database Serverless
Functions
Container
engine
Vault
API
Gateway
Load
Balancer
Message/
Event Broker
Data
Lake
Container
Registry
Artifact
Repository
Job
Pipelines
NoSQL
Database
Data
Cache
Code
Repository
Web Application
Firewall
CDN
Virtual
Machines
File Storage
API
µ

Publiek
power cooling
rack
space
IAM
Certificate
Management
Vulnerability
Scanning
Logging
Monitoring Auditing
Cloud Usage
Analysis
Resource
Manager
APIs
Cloud
Events
Tagging
Search
SQL
Database Serverless
Functions
Container
engine
Vault
API
Gateway
Load
Balancer
Message/
Event Broker
Data
Lake
Container
Registry
Artifact
Repository
Job
Pipelines
NoSQL
Database
Data
Cache
Code
Repository
Web Application
Firewall
CDN
Virtual
Machines
File Storage
API
µ
Zipin

Publiek
What do most microservices do?
• Store and retrieve state
• Read runtime secrets and configuration values
• Publish “observability data”
• Subscribe to event topic and handle incoming events
• Publish events to event topic
• Interact with other microservices
• Invoke external services
• Run microservice specific business logic
This requires quite a bit of
repetitive “plumbing” tied to
specific technologies
– not meaningful effort
[from a business functionality
perspective]

Publiek
Dapr
Distributed Application Runtime
quick introduction

Publiek
Dapr
• Runtime
• Personal Assistant for Applications and Microservices
• Distributed Application Runtime
• Microservice architecture
• Decoupling

Publiek
Dapr.io
• Launched by Microsoft (2019)
• Application runtime framework –
supporting cloud native and serverless
• Every application gets a uniform Personal Assistant
that takes care of common tasks
• remember (take note and reproduce)
• keep log
• restrict access
• handle incoming
• route outgoing
• interact with
other PAs (and
their apps)
Application
Microservice or
Monolith
Personal
Assistant
(Dapr)
Personal
Assistant
(Dapr)
Personal
Assistant
(Dapr)
Personal
Assistant
(Dapr)
Personal
Assistant
(Dapr)
Personal
Assistant
(Dapr)

Publiek
Dapr.io
• Personal Assistant is Dapr Side Car
• companion process or side car container in Kubernetes Pod
My App Dapr API
HTTP/gRPC
Application
Dapr sidecar

Publiek
key value
myApp-weapon "DeathStar"
State management
My App
POST
http://localhost:3500/v1.0/state/corpdb
[{
"key": "weapon",
"value": "DeathStar"
}]
Redis Cache

Publiek
State management
My App
key value
Redis Cache
GET
http://localhost:3500/v1.0/state/corpdb/planet
"DeathStar"

Publiek
State management
My App
key value
Azure CosmosDB
GET
http://localhost:3500/v1.0/state/corpdb/planet
"DeathStar"

Publiek
Dapr state API
Save state
POST /v1.0/state/corpdb
Retrieve state
GET /v1.0/state/corpdb/mystate
Delete state
DELETE /v1.0/state/corpdb/mystate
Get bulk state
POST /v1.0/state/corpdb/bulk
Submit multiple state transactions
POST /v1.0/state/corpdb/transaction
corpdb-redis.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: corpdb
spec:
type: state.redis
version: v1
metadata:
- name: redisHost
value: redis-master.default.svc.cluster.local:6379
- name: redisPassword
secretKeyRef:
name: redis-secret
key: redis-password

Publiek
Dapr state API
Save state
POST /v1.0/state/corpdb
Retrieve state
GET /v1.0/state/corpdb/mystate
Delete state
DELETE /v1.0/state/corpdb/mystate
Get bulk state
POST /v1.0/state/corpdb/bulk
Submit multiple state transactions
POST /v1.0/state/corpdb/transaction
corpdb-cosmosdb.yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: corpdb
spec:
type: state.azure.cosmosdb
version: v1
metadata:
- name: url
value: corpdb.documents.azure.com
- name: masterKey
secretKeyRef:
name: master-key
key: cosmos-key
- name: database
value: orders
- name: collection
value: processed

Publiek
HTTP API gRPC API
Microservice building blocks
Service-
to-service
invocation
State
management
Publish
and
subscribe
Resource
bindings
and triggers
Actors Observability Secrets Extensible

Publiek
Standard APIs accessed over http/gRPC protocols from
user service code
Runs as local “side car library” dynamically loaded
at runtime for each service
HTTP API gRPC API
Any language and framework
Application code
Microservices written in
Any code or framework…
Service-
to-service
invocation
State
management
Publish
and
subscribe
Resource
bindings
and triggers

Publiek
HTTP API gRPC API
Any cloud or edge infrastructure
Application code
Microservices written in
Any code or framework…
Service-
to-service
invocation
State
management
Publish
and
subscribe
Resource
bindings
and triggers
Hosting infrastructure
On-Premises
Azure Arc

Publiek
Service invocation
Service A
My App
mDNS
Multicast DNS component for
service discovery
mTLS encryption
POST
http://localhost:3500/v1.0/invoke/servicea/method/neworder
{"data":"Hello World"}
POST
http://10.0.0.2:8000/neworder

Publiek
Publish and subscribe
Service B
My App Redis
Cache
Service A
POST
http://localhost:3500/v1.0/publish/orders/processed
POST
http://10.0.0.2:8000/orders
http://10.0.0.4:8000/factory/orders

Publiek
Input triggers
My App
Twitter
POST
http://10.0.0.2:8000/newtweet
{"data":“📢 We are excited
to announce the …"}

Publiek
Output bindings
My App
Twilio
POST
http://localhost:3500/v1.0/bindings/twilio
Hello World

Publiek
Secrets
My App
GET
http://localhost:3500/v1.0/secrets/vault/mysecret
"supersecret"
HashiCorp
Vault
key value
mysecret "supersecret"

Publiek
Observability
OpenTelemetry collector
Logging &
tracing extensions
My App Redis
Cache
Twitter
Service B
Service A

Publiek
Dapr hosting environments
• Get started with dapr init -k
• Fully managed Dapr control plane
• Deploys dashboard, placement, operator,
sentry, and injector pods
• Automatically inject Dapr sidecar into
all annotated pods
• Upgrade with dapr upgrade or Helm
• Get started with dapr init
• Easy setup with Docker images
• Sets up placement, Zipkin, Redis
• slim-init available without Docker
• Run any application with Dapr sidecar
using dapr run
Self-hosted

Publiek
Dapr in self-hosted Docker mode
Local dev machine or virtual machine
Zipkin
tracing
Zipkin
Redis
state store
Redis
My App
State Stores
PubSub
Brokers
Secret Stores
Bindings
& Triggers
Observability
Dapr Components
dapr run myapp
Use components
Launch application
Launch sidecar process
Set env variables
Save and retrieve state
Publish and subscribe to messages
Send distributed tracing

Publiek
Hands On Lab 1
In a Linux compatible environment with Docker CLI set up
• Download and Install Dapr.io
• with out of the box state management and pub/sub based on Redis and
telemetry observation by Zipkin
• Run Dapr Sidecar
• Explore Dapr State Management APIs
• from the command line (cURL HTTP calls)
• Run MySQL container and have Dapr manage state in a MySQL database
• Explore Dapr Telemetry collection and presentation in Zipkin

Publiek
Dapr in self-hosted Docker mode – add MySQL Database
Local dev machine or virtual machine
Zipkin
tracing
Zipkin
MySQL
database
Redis
state store
Redis
myotherapp
daprdapr run --app-id myotherapp --dapr-http-port 3510 --components-path .
Launch application
Launch sidecar process
Set env variables
Save and retrieve state
mysql-statestore.yaml

Introduction
Node

Publiek
General Purpose Programming Languages
• C & C++
• COBOL, Fortran, ALGOL, BASIC, Pascal
• SmallTalk
• Visual Basic
• Python
• PHP, Perl, Ruby
• Java
• JavaScript (ECMAScript, ES)
• C#
• Go
• Kotlin
• Rust

Publiek
Node == Server Side JavaScript
• Script: Not statically typed, not compiled – interpreted at runtime
• Node(JS) – open source, runtime based on Chrome V8 JavaScript engine
• initial release: 2009
• Full stack development – server side and client side can leverage JavaScript
• for example: MEAN stack (MongoDB, Express, Angular, NodeJS)
• Light weight, quick startup, support for high concurrency
• Skills widely available, many resources, tools and libraries
• Easy to get started, quite high productivity and fun factor
• Functions are primary citizens – functional programming
well supported
• Quite suitable for microservices –
including containerized and serverless
• Available everywhere – cross platform, all public clouds
• Note: TypeScript compiles to Node that is interpreted at runtime

Publiek
Introducing Node – Single Thread model
• Multi-thread languages – such as Java –
typically block when file system or network operations take place
• Parallel processing continues in parallel threads
• Threads consume resources and are only somewhat scalable
• JavaScript has a main (single thread) event loop and uses
asynchronous processing
• Non-blocking handling of file system
interactions, HTTP calls, network
operations and other blocking actions
• Scalable, very quick startup

Publiek
Introducing Node - NPM
• NPM = Node Package Manager
• To include libraries from the ecosystem
• Leverages npm registry (> 1.3M NPM Modules available)
• Acquired by Microsoft – March 2020
• To build and run Node applications
• similar to Maven & Gradle (Java)
or pip (Python)
• Yarn and Turbo are popular
alternatives

Publiek
Hello World in Node
• app.js
• with the code to execute
• (optional) package.json
• with NPM dependencies, build and run configuration
(compare Maven pom.xml)
• Create Node application:
• npm init (optional, you can just create a text file by hand instead)
• Run Node application
• node app.js
• npm start
• Add dependencies
• npm install <name of module> --save

Publiek
Functions
• Functions
• have a name
• take zero, one or multiple untyped input parameters
• return zero or one untyped result
• or an exception instead of a result

Publiek
Functions
• Function references
• can be assigned to a variable
• can be passed around
• can be invoked
• Note: function name without
parentheses is not a call to the
function but a reference
• print vs print()

Publiek
Functions
• Function references can be
closures – function + state
• closures contain values of
local context at the time
of instantiation
Hello 1
World 2

Publiek
Require – to import a module
• Node contains many out of the box modules
• for example: file, os, http, stream, crypto, net, url
• The NPM package manager adds more than 1M modules to make us of
• You can create your own (reusable) modules
• To access elements from a module, that module needs to be identified,
using: const <logical name> = require('module name')
• Module name can be reference to a local file, to a global module or to the
name of a module defined in package.json (and installed in the
application’s node-modules directory)

Publiek
Node – Asynchronous calls
• JavaScript is single threaded – no parallel threads can be started from your
code
• Any action in Node that requires network or file system interaction is
typically performed asynchronously:
• the command is started and handed to the
Node background processes;
the main thread continues with other work
• when the background processes have completed
the task, the results are returned to the invoker
(when the main thread has an opening)

Publiek
Node – Asynchronous HTTP calls
• Many modules are available
for HTTP communications
from Node
• The standard HTTP
module is not the most
convenient
• An HTTP request is
processed asynchronously in
all cases
• main program loop
proceeds while request is
made
asynchronous HTTP request
response

Publiek
Asynchronous –
waiting for one or more asynchronous results
• A function is declared to be asynchronous using the async keyword
• The result of an asynchronous function is an object of type Promise
• in order to get the actual value – the Promise must be “resolved”
• this can be waited for using the keyword await
• Alternatively – especially when multiple asynchronous results are waited for at the
same time – we can use Promise.then( <callback function>) or Promise.all(
<callback function>) to wait for Promise results to come available

Publiek
Hello World Web Application
• Node is very good at handling HTTP Requests
• It is a great technology to implement REST APIs and backends for Web and Mobile applications
• fast, scalable, easy to develop and to deploy
define the function that
handles each HTTP
request and prepares the
response

Publiek
Dapr and Node Applications
• HTTP (or gRPC) with Dapr SideCar
• easier yet: Node SDK for Dapr
• Leverage Dapr and its building blocks for
• simple state management
• simple publication of messages
• subscription to incoming messages
• retrieving secrets
• retrieving configuration data
• invoking external services (database, message broker, store, API)
• register as listener to inbound requests and messages for external
services
• decoupled interaction between microservices – implemented in
potentially very different languages
components.yaml
sidecar (the personal assistant)
Node SDK
for Dapr
Observability
Prometheus AppInsights Jaeger
Zipkin
Bindings
& Triggers
GCP
Storage
AWS
S3 Kafka
Azure
Storage Twilio
State
Stores
Firebase Cassandra
Redis
Azure
CosmosDB
AWS
DynamoDB
Secret
Stores
AWS
Secrets Manager
Azure
KeyVault
GCP
Secret Manager
HashiCorp
Vault
Kubernetes
Secret
PubSub
Brokers
Redis
AWS
SQS
Azure
Service Bus
RabbitMQ
GCP
Pub/Sub

Publiek
State Management with Dapr from Node application
Node SDK
for Dapr
State Store
components
set & get state
components.yaml
State
Stores
state store
interface

Publiek
State Management with Dapr from Node application
Node SDK
for Dapr
State Store
components
set & get state
components.yaml
State
Stores
state store
interface
connect to
sidecar
retrieve
state
setstate

Publiek
State Management with Dapr from Node application -
switch to MySQL state store with configuration change
Node SDK
for Dapr
State Store
components
set & get state
components.yaml
State
Stores
state store
interface
connect to
sidecar
retrieve
state
setstate

Publiek
State Management with Dapr from any application
- either through SDK or using HTTP or gRPC
SDK for
Dapr
State Store
components
set & get state
components.yaml
State
Stores
state store
interface

Publiek
Pub/Sub with Dapr from Node applications
Node SDK
for Dapr
Pub/Sub
components
publish message
components.yaml
PubSub
Brokers
pub/sub
interface
Node SDK
for Dapr
components.yaml
subscribe on topic
with handler function
message sent to handler

Publiek
Node SDK
for Dapr
Pub/Sub
components
publish message
components.yaml
PubSub
Brokers
Node SDK
for Dapr
components.yaml
subscribe on topic
with handler function
message sent to handler
pub/sub
interface
publish
message

Publiek
Node SDK
for Dapr
Pub/Sub
components
publish message
ponents.yaml
PubSub
Brokers
Node SDK
for Dapr
components.yaml
pub/sub
interface
subscribe
on topic
message sent
to handler

Publiek
Asynchronous Interaction between Microservices
Front App
http
State
Stores
PubSub
Brokers
Node App
get get & set
publish
subscribe
consume
name = John

Publiek
Telemetry Observation in Zipkin
greeter
http
State
Stores
PubSub
Brokers
name-processor
get get & set
publish
subscribe
consume
name = John
Telemetry
Data

Publiek
Telemetry Observation in Zipkin
Telemetry
Data
Front App
http
State
Stores
PubSub
Brokers
Node App
get get & set
publish
subscribe
consume
name = John

Publiek
Handson Lab 2
• Install Node (and npm) runtime
• Explore Node
• hello world
• functions
• asynchronous
• http server
• Explore Node and Dapr
• add state management to Node application – through Dapr sidecar
• implement pub/sub for Node applications – through Dapr sidecar
• realize asynchronous communication between Node based microservices
through Daprized pub/sub
• Explore Telemetry for asynchronous interactions

Publiek
State
Stores
app.js
get & set
Node SDK
for Dapr
components.yaml
dapr_host,
dapr_http_port
app_port
key value
John
Monty
3
2
name,
instance count
Default, out
of the box
?name=John
Handson Lab 2

Microservices in
real life –
with Node &
Dapr.io &
30 March 2022
Lucas Jellema
Architect & CTO AMIS | Conclusion
Next Week:
Apache Kafka

Microservices, Node, Dapr and more - Part One (Fontys Hogeschool, Spring 2022)

More Related Content

What's hot (20)

Similar to Microservices, Node, Dapr and more - Part One (Fontys Hogeschool, Spring 2022) (20)

More from Lucas Jellema (20)

Recently uploaded (20)

Microservices, Node, Dapr and more - Part One (Fontys Hogeschool, Spring 2022)

Editor's Notes