Apache Kafka in the Insurance Industry

The Rise of Data in Motion in the Insurance Industry
Use Cases, Architectures and Examples powered by Apache Kafka
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de

Event Streaming with Apache Kafka in the Insurance Industry – @KaiWaehner - www.kai-waehner.de
The world is changing.

This transformation is
happening everywhere

Stores become Software

Banks become Software

Taxis become Software
2
min

Cars become Software

Insurance becomes Software

What enables this
transformation?

Challenges in the Insurance Industry
• Challenging market environments
• Stagnating economy
• Regulatory pressure
• Changes in customer expectations
• Proprietary and monolithic legacy applications
• Emerging competition from innovative insurtechs
• Emerging competition from other verticals that add
insurance products

Cloud Machine
Learning
Mobile Event
Streaming
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

This is a fundamental paradigm shift...
12
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Real-time Data beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Insurance
Claim processing
Claim fraud detection
Omnichannel
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

What is Event Streaming?

‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Insurance
Alice filed an insurance claim Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka

An Event Streaming Platform is the
Underpinning of an Event-driven Architecture
16
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams of real time events
Stream processing
apps
Connectors
Connectors
Stream processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order

STREAM
PROCESSING
CONNECTORS
Example Architecture for Event Streaming
ksqlDB
KStreams
Real-time decision making for claim processing and fraud detection
Dashboard
Oracle
DB
Oracle
CDC
CONNECTOR
Salesforce CDC
CONNECTOR
Salesforce
Source / Sink
CONNECTOR
Fraud Detection App

The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

Real World Deployments

Generali – Kafka as Integration Platform
for legacy and modern technologies
https://www.jug.ch/events/slides/190918_Microservices_and_Kafka_on_OpenShift.pdf

Integration Platform
for legacy and modern technologies

Database Integration and Offloading
Brownfield instead of Greenfield
Database
change
Microservices
events
SaaS
data
Customer
experiences
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
DB Workload
DB Workload
DB Workload
Read
No DB Workload

Database
change
Microservices
events
SaaS
data
Customer
experiences
Legacy
App
Modern
App 1
Complex business logic
Push changes once
Write
Write
continuously
Read
continuously
Modern
App 2
Write
continuously
Read
continuously
DB Workload
DB Workload
DB Workload
Read
No DB Workload
Database Replacement
Brownfield instead of Greenfield

Centene – Integration and Data Processing at Scale in Real-Time
Healthcare Insurer acts as intermediary for both government-sponsored and privately insured health care programs
Largest Medicaid and Medicare Managed Care Provider in the US
24
https://www.confluent.io/online-talks/building-an-enterprise-eventing-framework-on-demand/

Swiss Mobiliar – Decoupling and Orchestration
Orchestrator application to track the state of a billing process
Kafka as database and Kafka Streams for data processing
Complex stateful aggregations across contracts and re-calculations
Continuous monitoring in real-time
Daily reporting
25
Swiss Streaming Meetup: https://www.youtube.com/watch?v=do-Jl9CHLHA

Humana – Real-Time Integration and Analytics
Interoperability platform to transition from Insurance Company with Elements of Health, to truly a Health Company with Elements of Insurance.
Consumer-centric, health plan agnostic, provider agnostic. Cloud resilient and elastic. Event-driven and real-time.
26
https://www.confluent.io/resources/kafka-summit-2020/levi-bailey-keynote-humana-improving-health-with-event-driven-architectures/

freeyou – Streaming Analytics
Real-time price adjustments in vehicle insurance
https://www.confluent.io/kafka-summit-san-francisco-2019/how-to-build-real-time-price-adjustments-in-vehicle-insurance-on-streams/

Tesla – Carmaker and Utility Company
Trillions of messages per day for IoT use cases
https://www.confluent.io/kafka-summit-san-francisco-2019/0-60-teslas-streaming-data-platform/
https://www.confluent.io/blog/stream-processing-iot-data-best-practices-and-techniques/

Tesla – Car Insurer
Elon Musk: “much better feedback loop” instead of being statistical like other insurers
Shift from human experts to automation driven
by big data and machine learning
Real-time telematics data from all its drivers’
behavior and the performance of its vehicle
technology (cameras, sensors, …)
Better risk estimation of accidents
and repair costs of vehicles
Evaluation of risk reduction through new
technologies (autopilot, stability control, anti-theft
systems, bullet-resistant steel)

What is Apache Kafka?

Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

Kafka Makes Your Business Real-time
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.

Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?

Enterprise Data Platform Requirements Are Shifting
1 3 4
2
Scalable for
Transactional Data
Transient Raw data
Built for
Historical Data
Built for Real-
Time Events
Scalable for
ALL data
Persistent +
Durable
Enriched
data
● Value: Trigger real-
time workflows (i.e.
real-time order
management)
● Value: Scale across
the enterprise (i.e.
customer 360)
● Value: Build
mission-critical
apps with zero data
loss (i.e. instant
payments)
● Value: Add context &
situational awareness
(i.e. ride sharing ETA)
35

Only Event Streaming Has All 4 Requirements
Messaging
Databases
Event Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
36
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration

Kafka is Complementary to other Middleware
in the Enterprise Architecture
Orders Customers
Payments
Stock
REST
JMS
ESB
REST
CRM
Mainframe
SOAP
…
Kafka
Kafka
Kafka
Kafka
SOAP
API Management
HTTP
MQ

Project Example: Mainframe Offloading

“… rescue data off of the mainframe, in a cloud native, microservice-
based fashion … [to] … significantly reduce the reads on the
mainframe, saving RBC fixed infrastructure costs (OPEX). RBC stayed
compliant with bank regulations and business logic, and is now able to
create new applications using the same event-based architecture.”
Mainframe Offloading
for massive cost-savings
https://www.confluent.io/customers/rbc/

Mainframe Integration, Offloading and Replacement with Apache Kafka – @KaiWaehner - www.kai-waehner.de
Strangler Design Pattern
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/
https://martinfowler.com/bliki/StranglerFigApplication.html
“The most important
reason to consider a
strangler fig
application over a cut-
over rewrite is reduced
risk.”
Martin Fowler

Year 0: Direct Communication between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)

Year 1: Kafka for Decoupling between Mainframe and App
Application
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd
Party CDC Tool

Year 2 to 4: New Projects and Applications
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project
(Elastic, Spark,
AWS Services, …)
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd
Party CDC Tool

Year 5: Mainframe Replacement
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project
(Elastic, Spark,
AWS Services, …)
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
(Modern Technology)
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14

Project Example: Fraud Detection

Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html

Streaming Analytics for
Claim Processing and Fraud Detection at Scale
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Payment App
Streaming Platform
Analytics Platform
Other Components
Real Time
Alerting
System
All
Data
Alert
Ingest
Data
Human
Intelligence
Analytics
Platform
Train Analytic
Model
Data
Processing
Analytic Model
Preprocess
Data
Consume
Data
Deploy
Analytic Model
www.kai-waehner.de | @KaiWaehner | Streaming Machine Learning in FinServ and Insurance Industries

Ingestion of IoT Data
49
Replication
MirrorMaker /
Confluent Replicator /
Cluster Linking
Kafka Connect
Analytics /
Machine
Learning
Cars
Cars
Cars
Cars
Payment
App

SELECT payment_id, smartphone_id, payment_details
FROM payment p
LEFT JOIN user_database u ON p.smartphone_id = u.smartphone_id
WHERE u.payment_type = 'Apple Pay';
Preprocessing with ksqlDB

Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed Commit
Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
51
Model X
(at a later time)

Confluent Tiered Storage for Kafka
52

Use Cases for Reprocessing Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• New consumer application
• Error-handling
• Compliance / regulatory processing
• Query and analyze existing events
• Schema changes in analytics platform
• Model training
Real-time Consumer
Consumer of Historical Data

User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
54
“CREATE STREAM FraudDetection AS
SELECT payment_id,
detectAnomaly(payment_values)
FROM payment_table;“

Event Streaming Is The Future Of Data
56
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Why Confluent?

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
58

Car Engine Car Self-driving Car
Confluent Completes Apache Kafka

Confluent... Complete. Cloud-native. Everywhere.
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
kai-waehner.de
confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Apache Kafka in the Insurance Industry

More Related Content

What's hot (20)

Similar to Apache Kafka in the Insurance Industry (20)

More from Kai Wähner (20)

Recently uploaded (20)

Apache Kafka in the Insurance Industry