Apache Kafka for Real-time Supply Chainin the Food and Retail Industry

The Rise of Data in Motion
in the Food and Retail Industry
Use Cases, Architectures and Real-World Examples powered by Apache Kafka
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
confluent.io
kai-waehner.de
@KaiWaehner

Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
The Food Industry
2

The world is changing.

Stores become Software

Payments become Software

@KaiWaehner - www.kai-waehner.de
Logistics becomes Software

Partnering becomes Software
2
m
in

The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data

The Food Industry
9
The Real-Time Food
and Retail Supply Chain

This transformation is
happening everywhere

Food and Retail

Resorts World Las Vegas + Grubhub
15

What enables this
transformation?

Real-time Data beats Slow Data.
Logistics
Real-time sensor
diagnostics
Delivery planning
ETA updates
Payment
Fraud detection
Risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Sales
Real-time
recommendations
Personalized
coupon feed
Pay by walking out

Chick-fil-A Restaurant IoT
• One of the largest American fast food
restaurant chains
• Chick-Fil-A IoT platform RIoT (Restaurant
IoT)
• Kubernetes cluster in each of their 2000+
restaurants for real-time analytics at the
edge without an internet connection
• Thousands of units of industrial kitchen
equipment, food safety sensors and
point-of-sale devices are sending millions
of streaming messages per second
18
https://www.linkedin.com/pulse/why-did-chicken-need-iot-imran-bashir/
https://www.infoq.com/presentations/chick-fil-a-k8-clusters/
https://medium.com/@cfatechblog/edge-computing-at-chick-fil-a-7d67242675e2

This is a fundamental paradigm shift...
19
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

What is Event Streaming?

‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Payment
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka

Data in Motion in the Food and Retail Industry
Your Business as Streams of Events, powered by Kafka
Inventory
Shipping
Reporting
Orders
Frontend
Retail Example

24
Increase Revenue
→ Customer Experience, Loyalty
Decrease Costs
→ Increase Operational Efficiency
Mitigate Risks
→ Regulatory Compliance
Reporting & Analytics
Stock inventory management
Legacy IT Replacement
(e.g. Middleware replacement)
Cyber Security
(incl. SIEM)
Fraud Prevention
(Anomalous credit card transactions)
Legacy IT Modernization
(e.g. Mainframe off-load / augmentation)
Example Retail Solutions for Event Streaming
Buyer journey - personalized offers
(Contextualized Next Best Action /
Targeted Offers)
Real-time Inventory Management
(across web, store, …)
Customer 360
(Call Center - Know Your Customer - KYC)
RT mobile applications
(clothing, food and beverage ordering, …)
Migration to the Cloud
(Hybrid on-prem / Cloud. Also Hybrid
Public Cloud vendors)
Microservices Architecture
Data Infrastructure layer
Business Application layer - the use cases
Data Pipelines Messaging
Microservice/
Event Sourcing Stream Processing Data Integration Streaming ETL Log Aggregation

Event Streaming
in the Food Industry?

Food Value Chain
IoT-Based and Data-Driven
Single source of truth
across the food value chain
(in the factories, and across regions)
Business critical operations
(tracking, calculations, alerts, …)
https://www.confluent.io/blog/creating-iot-based-data-driven-food-value-chain-with-confluent-cloud/

Migros
Switzerland's largest retail company, largest supermarket
chain and largest employer
Real-time transportation information visualization with MQTT
and Kafka
Supply chain optimization with a single streaming pipeline
(including replay whole day of events)
Forecast the truck arrival time; plan and reschedule truck tours
27
https://www.meetup.com/Messaging-Streaming-Switzerland/events/278115421

“Walmart is a $500 billion in revenue
company, so every second is worth
millions of dollars. Having Confluent
as our partner has been invaluable.
Kafka and Confluent are the
backbone of our digital
omnichannel transformation
and success at Walmart.”
VP of Walmart Cloud

Real-Time Inventory System
https://www.confluent.io/blog/walmart-real-time-inventory-management-using-kafka/
https://www.confluent.io/kafka-summit-san-francisco-2019/when-kafka-meets-the-scaling-and-reliability-needs-of-worlds-largest-retailer-a-walmart-story/
● Investment in Kafka and Confluent has helped topline
company growth
● 8,500 nodes processing 11 billion events per day
● Deliver an omnichannel experience so every
customer can shop the way they want to

Real-time Replenishment System
https://www.confluent.io/events/kafka-summit-americas-2021/speed-without-accidents-kafka-exemplified/
● Caters millions of its online and walk in customers
● Ensures optimal availability of needed assortment
and timely delivery on online fulfillment
● 4+ billion messages in 3 hours generate an order
plan for the entire network of Walmart stores with
great accuracy
● ~470M ordering decisions made daily
● Kafka as data hub and for real-time processing
● Spark for micro-batches

Simplifying Omnichannel Retail at Scale
31
https://kafka-summit.org/sessions/simplifying-omni-channel-retail-scale/

Domino’s Pizza
• Multinational pizza restaurant chain with ~17,000 stores
• Transformation from a traditional pizza company to an e-commerce driven organization
• Data-first approach and relentless customer focus
• Real-time operation views to franchise owners (order volume by channel, store efficiency metrics, …) across different
ordering channels
Benefits
Improve store operational real-time analytics
Support global expansion goals via legacy IT modernization
Implement more personalized marketing campaigns
Real-time single pane of glass
https://www.confluent.io/customers/dominos/

FREE NOW
Stateful stream processing with Confluent Cloud, Kafka Connect, Kafka Streams, Schema Registry
Cloud-native application elasticity and scalability leveraging Kafka and Kubernetes capabilities
Use cases: Dynamic pricing, fraud detection, real-time analytics for marketing campaigns, etc.
Various information about the trip, location and business performance
35

Fraud Detection @ Grab
GrabDefence SaaS service build with Confluent Cloud, Kafka Streams and ML for stateful stream processing
Billions of fraud and safety detections performed daily for millions of transactions (1.6% is lost in fraud in Southeast Asia)
Data science and engineering platform to search for anomalous and suspicious transactions and identifying
high-risk individuals
Example: An individual who pretends to be both the driver and passenger, and makes cashless payments to
get promotions

instacart
Grocery delivery and pick-up service in the United States
and Canada.
Order groceries from participating retailers with the
shopping being done by a personal shopper
Elastic scale and fast, agile internal adoption of real-time
data processing with Confluent Cloud
https://www.confluent.io/events/kafka-summit-americas-2021/driving-innovation-with-data-in-motion-at-instacart/
Covid Pandemic:
“Handle 10 years' worth
of growth in six week”

Context-specific Customer 360
39
Electrical retailer
Hyper-personalized online
retail experience, turning
each customer visit into a
one-on-one marketing
opportunity
Correlation of historical
customer data with real-
time digital signals
Maximize customer
satisfaction and revenue
growth, increased
customer conversions
https://www.confluent.io/customers/ao/

Live commerce with real-time data correlation
including integration of CRM, loyalty, inventory, chatbots, location-based services, etc.
40

Live Commerce with Data in Motion
Influencer Show
on Veggie Food
Payment
Shipping
Sales
Platform
Social
Platform

Live Commerce with Data in Motion
Interact with the customer during the show.
Recommend products that need to be sold.
Provide context-specific pricing.
All automated. In real-time. At scale.
Data Consolidation
Streaming ETL
(Kafka Connect + Kafka Streams)
Mobile App
(Swift on iOS)
(4) Receive
product recommendation
and context-specific price
Order Fulfillment &
Payment Service
Buy Now Pay Later (BNPL)
(C++)
Chatbot
External API
(REST API)
(1) Click on
food
(5) Order
product
(2) Q&A with chatbot
(5) Fulfill order
CRM / Loyalty
Salesforce
(Kafka Connect)
Source: channelize.io
Inventory
Live monitoring
(ksqlDB)
(3) Context-specific
pricing engine

@KaiWaehner - www.kai-waehner.de
Cross-Company Stream Data Exchange
Streaming Replication and API Management
Cluster Linking
Food
Supplier
Delivery
Service
Streaming integration
between companies
API Management
(REST et al) is not appropriate
for
streaming data
Restaurant

What is
Apache Kafka?

Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

Apache Kafka is a Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
46

Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB

Risk Management
with Stream Processing
51
British multinational groceries and general merchandise retailer
One of Europe's largest retailers
Risk-management platform built to detect and prevent attacks
https://www.confluent.io/resources/kafka-summit-2020
/risk-management-in-retail-with-stream-processing/

Central Nervous System
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.

Event Topics Storage Partitions
Events / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
250,000 500 25,000 25
100 5 300 3
Kafka Scales with Your Business.

Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?

Enterprise Data Platform Requirements Are Shifting
1 3 4
2
Scalable for
Transactional Data
Transient Raw data
Built for
Historical Data
Built for Real-
Time Events
Scalable for
ALL data
Persistent +
Durable
Enriched
data
● Value: Trigger real-
time workflows (i.e.
real-time order
management)
● Value: Scale across
the enterprise (i.e.
customer 360)
● Value: Build
mission-critical
apps with zero data
loss (i.e. instant
payments)
● Value: Add context &
situational awareness
(i.e. ride sharing ETA)
59

Only Event Streaming Has All 4 Requirements
Messaging
Databases
Event Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
60
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration

Kafka
Connect
Kafka Cluster
Data Warehouse
Integration
Domain-Driven Design and Decoupled Microservices
POS Interface Mobile Backend
Kafka Broker
in the Store
Kafka-native Client
(Java, Python C++, KSQL, etc.)
Data
Governance
Event Streaming Platform
Analytics Domain Store Domain Customer Domain
Security

Event Streaming in the Cloud
62
Focus on
Business Outcomes
• Real-time
• Serverless
• Elastic

Architectures for
Apache Kafka in Retail?

Real-time Inventory System
C2
Time
P
C3
C1
C2
C3
Asset / People Tracking
Real-Time Alerting
Geofencing for
Distance Enforcement
Continuous
Real-Time Aggregation
Plan Future Location
Requirements
Batch Analytics
on Historical Events

Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
for RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
68

‘My Porsche’
A digital service platform for customers, fans, and enthusiasts
70
https://medium.com/porschedev

Omnichannel Retail
Time
P
C3 C2
C1
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago

Omnichannel Retail
Time
P
C3 C2
C1
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions

CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Customer
data
Train
schedule
Payment
data
Loyalty
information
Hybrid Retail Architecture

Point of Sale
(POS) Loyalty
System
Local Inventory
Management
Payment Discount
Customer
data
Train
schedule
Payment
data
Loyalty
information
Global Inventory
Management
Event Streaming at the Edge
in the Smart Retail Store
Item Availability

Disconnected Edge
Time
P
C3 C2
C1
Context-specific
Advertisement
Real-time
(Milliseconds)
Location-based
Customer Action
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Payment Processing
Near Real-time
(Seconds)
Replication to Cloud
Batch
(Depending on Network Bandwidth)

Ship-Shore Highway – Swimming Retail Stores
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/

Event Streaming Is The Future Of Data
82
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Why Confluent?

The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
84

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
85

Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.

Kai Waehner
Field CTO
kai.waehner@confluent.io
confluent.io
kai-waehner.de
@KaiWaehner
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Apache Kafka for Real-time Supply Chainin the Food and Retail Industry

More Related Content

What's hot (20)

Similar to Apache Kafka for Real-time Supply Chainin the Food and Retail Industry (20)

More from Kai Wähner (20)

Recently uploaded (20)