Apache pulsar - storage architecture
1 like957 views
Apache Pulsar is a flexible pub-sub messaging system backed by a durable log storage. It uses a segment-centric architecture where messages are stored in independent segments across multiple brokers and bookies for redundancy. This allows for strong durability, high throughput, and seamless expansion without data rebalancing. Pulsar brokers serve client requests and acquire ownership of topics, while bookies provide durable storage with replication for fault tolerance.
Apache pulsar - storage architecture
- 1. Matteo Merli fast, durable, flexible pub/sub messaging
- 2. Introduction 2 One sentence definition for Apache Pulsar: “Flexible pub-sub system backed by a durable log storage” • Easy to use API — Support both Queuing and Streaming • Strong storage guarantees — Durability, latency, scalability
- 3. Pulsar architecture basics 3 • Brokers — Serving nodes • Bookies (Apache BookKeeper) — Storage nodes • Each layer can be scaled independently • No data locality — Data for a single topic/partition is not tied to any particular node
- 5. Considerations • Stateful systems can become unbalanced when traffic changes • The system needs to be designed to allow for quick reaction, distributing the load across all nodes 5
- 6. Pulsar broker • Broker is the only point of interaction for clients • Brokers acquire ownership of group of topics and “serve” them • Broker has no durable state • There’s a service discovery mechanism for client to connect to right broker 6
- 8. Segment centric storage • Storage for a topic is an infinite “stream” of messages • Implemented as a sequence of segments • Each segment is a replicated log — BookKeeper “ledger” • Segments are rolled over based on time, size and after crashes 8
- 10. Broker failure recovery 10 • Topic is reassigned to an available broker based on load • Can reconstruct the previous state consistently • No data needs to be copied • Failover handled transparently by client library
- 11. Bookie failure recovery 11 • After a write failure, BookKeeper will immediately switch write to a new bookie, within the same segment. • As long as we have any 3 bookies in the cluster, we can continue to write
- 12. Bookie failure recovery 12 • In background, starts a many-to-many recovery process to regain the configured replication factor
- 14. Why should I care? “Segment centric” vs “Partition centric” 14
- 15. Comparison with Apache Kafka 15
- 16. Comparison with Apache Kafka • In Kafka, partitions are assigned to brokers “permanently” • A single partition is stored entirely in a single node • Retention is limited by a single node storage capacity • Failure recovery and capacity expansion require “rebalancing” • Rebalancing has a big impact over the system, affecting regular traffic 16
- 17. Recap Advantages of segment-centric architecture: • Unbounded log storage • Instant scaling without data rebalancing • Fast replica repair • High write and read availability via maximized data placement options 17
- 18. Q & A Thank You http://pulsar.incubator.apache.org 18