Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022

Introducing KRaft: Kafka without
ZooKeeper
Colin McCabe
Principal Engineer
Confluent

About Me
2
I work on Apache Kafka at
Confluent.
Kafka Committer & PMC Member

3
● Introduction
● Demo
● Background
● KRaft Architecture
● Deploying KRaft
● Roadmap
● Q&A
Table of Contents

Removing ZooKeeper
5
Current
ZK-Based
Architecture

Removing ZooKeeper
6
Current
ZK-Based
Architecture
New KRaft-Based
Architecture

Apache ZooKeeper
10
● ZooKeeper is a service for maintaining
configuration, naming, and
distributed synchronization
● ZK is typically deployed in 3-node
clusters
● Traditionally, all Kafka clusters
needed an associated ZK cluster.

How Kafka Used ZooKeeper
11
As a metadata store for…
● Topics
● Partitions
● Configurations
● Quotas
● ACLs
And a way to coordinate
cluster membership and
elect a controller.
/brokers/ids/0
/brokers/ids/1
/brokers/topics/foo
/brokers/topics/foo/partitions/0/state
...

Kafka Clients and ZooKeeper
12
● Initially: admin tools and clients talked to ZK

Kafka Clients and ZooKeeper: Problems
13
● Initially: admin tools and clients talked to ZK
● Limited
security
● Complex
config
● Perf issues
● Limited
isolation

Isolating Kafka Clients from ZooKeeper
14
● Solution: admin tools and clients talk to brokers

Isolating Kafka Clients from ZooKeeper
15
● Solution: admin tools and clients talk to brokers
● ACLs
● Quotas
● Auditing

Brokers and ZooKeeper: Problems
16
● Limited metadata
scalability
● Long controller
failover times
● Complex
programming
paradigm
● Limited reqs/sec

KRaft Mode
17
● KRaft = Kafka on Raft
● Kafka stores its own
metadata in a Raft
quorum
● Leader = active
● Hot standbys
● 1 system

Controller Startup with ZK
20
● One broker wins the election
in ZooKeeper to be the
controller

Load full
metadata
21

22
● UpdateMetadataRequest
● LeaderAndIsrRequest
All
partitions

Problems with ZK-based Controller Startup
23
● Have to load all metadata synchronously on startup:
○ O(num_partitions), O(num_brokers)
○ Controller is unavailable during this time
■ Cold start: cluster unavailable.
■ Controller restart: admin ops and ISR changes unavailable
● Have to send all metadata to all brokers on startup
3 minutes

How KRaft Replaces ZooKeeper
24
● Instead of ZooKeeper, we have an internal topic
named __cluster_metadata
○ Replicated with KRaft
○ Single partition
○ The leader is the active controller
● KRaft: Raft for Kafka
○ The Raft protocol implemented in
Kafka
■ Records committed by a majority
of nodes
○ Self-managed quorum
■ Doesn’t rely on an external
system for leader election

Metadata Records
25
● Binary records containing
metadata
○ KRPC format
○ Auto-generated from protocol
schemas
○ Can also be translated into
JSON for human readability
● Two ways to evolve format
○ New record versions
○ Tagged fields
● Some records are deltas that
apply changes to existing state
{
"type": "REGISTER_BROKER_RECORD",
"version": 0,
"data": {
"brokerId": 1,
"incarnationId": "P3UFsWoNR-erL9PK98YLsA",
"brokerEpoch": 0,
"endPoints": [
{
"name": "PLAINTEXT",
"host": "localhost",
"port": 9092,
"securityProtocol": 0
}
],
"features": [],
"rack": null
}
}

Metadata as An Ordered Log
26
TopicRecord(name=foo, id=rtkInsMkQPiEBj6uz67rrQ)
PartitionRecord(id=rtkInsMkQPiEBj6uz67rrQ, index=0, …)
ConfigRecord(name=num.io.threads, value=...)
RegisterBrokerRecord(id=4, endpoints=…, …)
10434
10435
10436
10437
10438
10439
2
1

Controller Startup with KRaft
27
● The blue nodes are
designated KRaft controllers

28
● The KRaft controllers will elect
a single leader
● The leader will have all
previously committed records

29
● The newly elected KRaft controller
is ready immediately
● Brokers fetch only the metadata
they need
● New brokers and brokers that are
behind fetch snapshots

Controller Failover
30
ZK Mode KRaft Mode
• Win controller
election
• Load all topics and
partitions
• Send LeaderAndIsr
+ UpdateMetadata
to all brokers
• Win KRaft
election
• Start handling
requests from
brokers

Rolling Nodes
31
ZK Mode KRaft Mode
Restarted node
begins with no
metadata. Must wait
for full metadata
update RPCs from
controller.
Restarted node
consults its local
metadata cache,
transfers only what it
doesn’t have, based
on metadata offset.

Calling an Admin API with ZK
32
● External client connects to a
random broker

33
● Broker alters ZooKeeper
● Example: create znodes
for new topic

34
● Znode watch triggers
● Controller loads changes

35
● Controller pushes out changes to
brokers
● Incremental LeaderAndIsr
● Incremental UpdateMetadata

ZK-based Programming Model
36
Controller Thread
ZooKeeper
Create
/brokers/topics/foo
Request Handler
/brokers/topics
Changed
List /brokers/topics
Compute
changes
List /brokers/topics/foo

Problems
37
● What if the controller fails to send an update to a specific broker?
○ Metadata divergence
○ No easy way to know what we have and what we don’t have
● Difficult programming model
○ Multiple writers to ZooKeeper
○ Can’t assume your cache is up-to-date!
● ZK is the bottleneck for all admin operations
○ Often 1 admin operation leads to many ZK operations
○ Blocking

Calling an Admin API with KRaft
38
● External client connects to a
random broker

39
● Broker forwards the request to
the active controller for
processing

40
● Controller creates metadata
records and persists them to
__cluster_metadata

41
● Once the records have been
committed to the metadata
log, the active controller
returns the result to the
forwarding broker, which
returns it to the external client

42
● Brokers are continuously
fetching metadata from the
active controller.
● They become aware of the
admin changes by reading the
new metadata records.

KRaft-based Programming Model
43
Raft Quorum
Controller
CreateTopicsRequest
Request Handler
Write new records Last stable offset
advances
CreateTopicsResponse

KRaft-based Programming Model: Pipelining
44
Raft Quorum
Controller
CreateTopicsRequest
Request Handler
Write new records Last stable offset
advances
CreateTopicsResponse

Admin Operations in KRaft
45
● Pull-based metadata propagation model can recover from RPC send failures
easily
● Simpler programming model: single-writer
● Pipelining means that we can have multiple metadata operations in flight at once

Deploying KRaft
48
X 4
X 3
X 4
X 3

Deploying KRaft
49
X 4
X 3
X 3
X 3
● Combined mode
● Can run with 3 brokers and RF=3

New Tools
50
● “Double roll” process
for upgrading
inter.broker.protocol
● zookeeper-shell.sh

New Tools
51
● “Double roll” process
for upgrading
inter.broker.protocol
● zookeeper-shell.sh
● kafka-features.sh
with metadata flag
● metadata-shell.sh
● dump-log-segments.sh

kafka-dump-log
52
$ ./bin/kafka-dump-log.sh
--cluster-metadata-decoder
--files /tmp/logs/__cluster_metadata-0/00000000000000000000.log
Dumping /tmp/logs/__cluster_metadata-0/00000000000000000000.log
Starting offset: 0
baseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1
[...]
| offset: 0 CreateTime: 1650857270775 keySize: 4 valueSize: 19
sequence: -1 headerKeys: [] controlType: LEADER_CHANGE(2)
baseOffset: 1 lastOffset: 1 count: 1 baseSequence: -1 [...]
{"type":"REGISTER_BROKER_RECORD","version":0,"data":{"...

Interactive Metadata Shell
53
● Replaces zookeeper-shell in KRaft clusters
● Data sources
○ Snapshot
○ Running controller cluster
● Reads __cluster_metadata log entries into memory
● Constructs a “virtual filesystem” with the cluster’s information
● Commands available: ls, pwd, cd, find, etc.

kafka-metadata-shell
54
$ ./bin/kafka-metadata-shell.sh --snapshot
/tmp/logs/__cluster_metadata-0/00000000000000000000.log
Loading...
Starting...
[ Kafka Metadata Shell ]
>> ls
brokers configs local metadataQuorum topicIds topics
>> cat /brokers/1/registration
RegisterBrokerRecord(brokerId=1, incarnationId=tHo3Z8dYSuONV5hA82BVug,
brokerEpoch=0, endPoints=[BrokerEndpoint(name='PLAINTEXT',
host='localhost', port=9092, securityProtocol=0)], features=[],
rack=null, fenced=true)

Monitoring the Quorum
55
● Metrics
○ MetadataOffset
○ SnapshotLag
○ SnapshotSizeBytes
○ MetadataCommitRate
○ MetadataCommitLatency
○ Current Raft state (follower, leader,
candidate, observer)
■ Important for controller health
● DescribeQuorum RPC
○ Leader ID
○ Leader epoch
○ High water mark
○ Current voters
■ Controllers
○ Current observers
■ Brokers
■ Possible metadata shell
instances, etc.
○ Log end offset of all followers

Roadmap
57
● AK 2.8: first KRaft release (EA)
● AK 3.3: KRaft is production-ready
for new clusters
● KRaft deployed in Confluent Cloud

Roadmap
58
for new clusters
● AK 3.4: Upgrade from
ZK (EA)
● AK 3.5: ZK mode
deprecated

Roadmap
59
for new clusters
● AK 3.4: Upgrade from
ZK (EA)
● AK 3.5: ZK mode
deprecated
● AK 4.0: ZK mode
removed

Feature Gaps in Kafka 3.3
60
● Upgrade from ZK mode
● Dynamic configurations on the controller
● SCRAM support
● Delegation tokens
● JBOD support

Thank You!
Colin McCabe
cmccabe@apache.org

Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022

More Related Content

What's hot (20)

Similar to Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022 (20)

More from HostedbyConfluent (20)

Recently uploaded (20)

Introducing KRaft: Kafka Without Zookeeper With Colin McCabe | Current 2022