This is a good fit for streams carrying sensitive tenant data, audit logs, agent transcripts, or regulated records such as protected health information ¹. Your application supplies keys from your own infrastructure, while S2 handles request-time encryption behind the durable stream API.

Where this fits

There are three places encryption operations can happen — in your application, in S2 with keys you supply per request, or in S2 with keys governed by a cloud key management system (KMS).

Client-side record encryption is the right model when your hard requirement is that S2 never sees plaintext. You encrypt each record before append and decrypt after read. This gives you end-to-end control, but every producer and consumer must handle encryption, decryption, and encoding correctly. You also lose compression on the wire, since encrypted bytes don't compress.

KMS-integrated encryption (CMEK) gives you key governance through your cloud provider's KMS, IAM policies, and audit logs you already manage. However, it is less portable, since you are limited to that provider's KMS and the services that integrate with it.

Client-supplied encryption keys (CSEK) are the middle ground, in the same family as Google Cloud CSEK and Amazon S3 SSE-C. Encryption and decryption happen inside S2 at request time, so producers and consumers stay simple. Client↔S2 compression still works since it runs before encryption on writes and after decryption on reads. You decide where keys live and which workloads can use them, independent of any specific KMS.

The tradeoff is S2 sees plaintext and key material in memory while serving. If your compliance posture requires that the service operator never have access to plaintext, use client-side encryption instead.

How it works

Basins can now be configured with an encryption algorithm ². Every new stream created in that basin then requires an encryption key to append or read records.

Keys are sent as base64-encoded key material in the s2-encryption-key request header over TLS. The CLI and all SDKs support specifying it easily at the stream level.

In the s2.dev data plane, writes are encrypted at the edge service before being forwarded downstream. On reads, encrypted records are decrypted only after they return to the edge service, after any caching and coalescing.

The core encryption path lives in the open source s2 repo, so you can verify how keys are handled. s2-lite, the self-hostable form factor of S2, supports the same mechanism.

There is no meaningful impact to performance, as the supported ciphers are designed for high throughput and are hardware-accelerated in practice.

For operational guidance on key management, including envelope encryption and rotation, see the encryption docs.

Try it

Let's configure a basin to encrypt new streams:

$ s2 create-basin logs-prod \
  --create-stream-on-append \
  --stream-cipher aegis-256

You can also reconfigure an existing basin – only new streams are affected:

$ s2 reconfigure-basin logs-prod --stream-cipher aegis-256

Now, when a new stream is created in this basin, it will use the configured cipher. An encryption key must be provided with all data plane requests that read or write records.

For the CLI, generate a key and pass it explicitly, or use the S2_ENCRYPTION_KEY environment variable.

$ export S2_ENCRYPTION_KEY="$(openssl rand -base64 32)"
 
$ printf 'hello from an encrypted stream\n' | \
    s2 append s2://logs-prod/app/node-foo
 
$ s2 read s2://logs-prod/app/node-foo --seq-num 0 --count 1
hello from an encrypted stream

With the TypeScript SDK, the key is scoped to the stream handle:

import { AppendInput, AppendRecord, S2 } from '@s2-dev/streamstore';
 
const s2 = new S2({ accessToken: process.env.S2_ACCESS_TOKEN! });
const stream = s2.basin('logs-prod').stream('app/node-foo', {
  encryptionKey: process.env.S2_ENCRYPTION_KEY!,
});
 
await stream.append(
  AppendInput.create([AppendRecord.string({ body: 'hello' })])
);

Client-supplied encryption keys are available today on the s2.dev cloud service as well as s2-lite, at no additional cost.

Both are very impressive! And I wanted to take a stab at it by using the infrastructure designed for ordered, real-time, durable data streams, so I built a full video conferencing app on S2 streams.

You can try it here. Open it in two tabs or share the link with someone! The source code is on GitHub.

Architecture

S2 turns the humble log, the stream, into a first-class cloud storage primitive. Instead of storing entire objects, applications append and read records on named streams using its focused API.

Every record is durably sequenced at the stream tail. Consumers can read streams live as new records arrive or replay history from any earlier position. This allows a stream to act as both durable storage and reliable transport for ordered data.

Each room uses a small set of named streams:

rooms/{room}/media/{user}   -> video + audio + screen, interleaved
rooms/{room}/chat           -> persistent chat history
rooms/{room}/meta           -> join/leave + control events (like hand raises)

Audio and video are sent over a WebSocket connection to a Go server which makes them durable in S2 using an AppendSession and fans out to multiple readers over a ReadSession.

The key simplification here is that:

• live media is a stream read
• recording is a no-op, the stream is durable by design
• replay is another stream read
• MP4 export is another stream read

There is no separate recording pipeline, replay database, or post-processing step to assemble files!

Reading live media

The Go server writes media using an AppendSession, batching records in 5ms windows for low latency and high throughput. Each record body is the raw media payload, with the media type stored as an S2 record header.

For live viewing, each participant reads each remote media stream from the current tail, following new records as they arrive.

So the live path looks like this:

The same pattern is used for other features too:

• chat starts from SeqNum: 0, so new users get old messages first and then new ones
• meta tails live control events, so "hand raises" work like any other record
• replay finds past participants by reading join events from meta

Replay

Replay is not a special file format. The server just reads the room streams again!

/api/rooms/{room}/timeline
    ├─ read meta stream for participant history + join/leave events
    ├─ read first media record for start timestamp
    └─ CheckTail() for end timestamp
 
/ws?room=...&replay=true&from=T
    ├─ replay media/{alice} from T
    ├─ replay media/{bob}   from T
    ├─ replay meta          from T
    └─ replay chat          from T

Playback speed is derived from record timestamps, so the replay UI is mostly a thin layer over stream reads:

For MP4 export, the server reads each participant's media stream, pipes audio and video directly into ffmpeg for compositing, and streams the result to the browser with no intermediate files.

If you don't care about saving the video, you can just set the retention policy on the streams to be short, e.g. 5 seconds, instead of having a background job. Or to save it forever, you can set it to be infinite.

Thoughts

This might just become our go to meeting spot given how smooth it was. Every feature that I thought of could be simply mapped as a read or write on S2's durable streams. It is an unusual architecture for a video conferencing app, but it points to a broader idea that when streams are treated as a storage primitive rather than merely a messaging layer, many real-time applications become far simpler to build and operate.

We're also thrilled to share that we raised $3.85M in a seed round led by Accel, with participation from Y Combinator, a terrific group of angel investors like Theo Browne (t3.gg), Charles Zedlewski (Together AI), Paul Masurel (Quickwit), and more. This brings our total capital raised to $5.5M.

What we do

S2 is making real-time, serverless data infra for AI builders. We give users an API for durable streams, unlimited in number and storage.

The need for real-time infra is everywhere today: token streaming, live observability, agent communication. This has only reinforced our original thesis that streams need to be as simple, reliable, and scalable as object storage.

Our customers are now creating millions of durable streams and pushing terabytes of data, each week.

S2 is such a useful primitive for building realtime features. We use it to stream telemetry data so that users can see in realtime things like logs and metrics. It’s been rock solid and reliable—the kind of infra that just works™.

— Rafael Garcia, CTO, Kernel

S2 solved our problem with reliably streaming long-running AI sessions. Before, connection drops could cause lost data and broken streams. S2's bottomless storage means our customers can stream for hours and network hiccups don’t matter.

— Matt Aitken, CEO, Trigger.dev

The gap we fill

Companies have been stitching complex systems together to try to acommodate the demands of real-time AI applications.

LLMs return token streams. Sandboxed execution involves remote I/O streams. Agent sessions evolve as a sequence of events. How do you ensure real-time visibility with long-term history? What does distributed plumbing for multi-agent architectures look like?

With traditional infra like Kafka, NATS, or Redis Streams, you are forced to choose between cardinality of streams and durability.

What if one serverless resource could persist every agent action at the session-level — and make it instantly visible to any number of readers?

That's S2.

General availability

GA reflects our confidence in the maturity of the service, and our commitment to stay highly available, durable, and consistent. As a data infra company, this is existential for us.

So many aspects of our work have given us this confidence: we gained experience running production workloads for our customers; focused on clarifying, simplifying, and hardening our architecture; invested heavily in verification through simulation.

Where we are going

Reliability, scalability, performance, and security will always be a priority.

What we have in mind as we evolve the product, shaped by what we have heard from users —

• Features for agent builders: first-class support for forking, integrations
• Security: bring-your-own-key encryption, stateless access tokens
• Expansion: additional cloud regions, and customer VPCs
• Higher layers: queuing, large messages, and more to come

We are so excited to continue the work of making streams a cloud storage primitive. Thank you to all of our customers, testers, and open source adopters.

If you are new to S2, try it out with free credits! You can also connect with us on email or Discord.

All our investors

Accel, Adam Suskin, Alessandro Puppo, Aman Sidhant, Amitav Chakravartty, Andrew Stanton, Benjamin Bryant, Brian Kim, Chalmers Brown, Charles Zedlewski, Deep Kapur, Dennis Beatty, Eight Capital, Fredrik Björk, Grayscale Ventures, Hemanth Soni, James Wu, Jeff Ling, Jeremy Hindle, JJ Fliegelman, Kevin Li, Keyur Govande, Li Sabhaya Capital, Lyon Wong, Manju Rajashekhar, Materialized View Capital, Micah Wylde, Michael Shimeles, Mokhtar Bacha, Nikitha Suryadevara, Nimit Maru, Orange Collective, Paul Masurel, Pioneer Fund, Race Capital, Rafael Garcia, Ritual Capital, Rush Sadiwala, Satish Talluri, Shane Barratt, Spot VC, Theo Browne, Transpose Platform, Twenty Two Ventures, Uncorrelated Ventures, Victor Mota, Y Combinator

In practice, many of us have learned to compensate for this. For example, after dealing with repeated mistakes and oversights from Claude Code, I've fallen into a pattern where one Claude session writes most code and other independent Codex (or Claude) sessions review it.

The underlying goal is complete context separation. My reviewer is not constrained by the same intermediate assumptions that shaped the original output, it is unbiased by the context, even if not the training data. What we are doing informally is creating parallel reasoning paths, separating generation from critique to produce structural disagreement.

We are rediscovering in small, practical ways what has long existed in other disciplines. In science, it appears as blind peer review. In forecasting, it is the Delphi method. In risk analysis, it becomes competitive hypothesis testing. In law, as the adversarial system, where structured opposition precedes judgment.

These systems are all predicated on the idea that it is beneficial for participants to generate perspectives independently, and thus prevent cross-contamination, and only afterwards work toward consensus.

Independence requires separate contexts

Telling a model to "consider the opposing view" doesn't create independence. By the time that instruction runs, it's already committed to a framing. The attention mechanism within a single forward pass may reweigh information, but it's all happening inside a shared state. Independence requires distinct contexts with no shared memory or conversational history.

To make that separation concrete, the unit of reasoning really can't be a single model invocation. It should be some bounded execution context with its own internal memory. We can consider the agents who share a bounded context to be forming a cohort. A single cohort might contain multiple agents — say, one that researches and another that critiques — but the key is that while they share context with each other, that context is isolated from other cohorts.

Only after cohorts have independently reached some stopping point should reconciliation occur.

Enforcing isolation at the infrastructure layer

Capturing all context for a cohort on S2 streams allows us to enforce this independence at the infra layer.

In my setup, each cohort treats a stream as its shared, append-only journal. All agents within a cohort can append and read from their stream, allowing it to serve as both a message bus between agents, and as a longer-term context store. Streams can be created on demand, meaning cohort definitions and membership can also vary over the course of an experiment.

A record in a stream named group/growth is invisible to readers of group/risk. This makes isolation not an instruction but an intrinsic property of the system.

Within a cohort, agents maintain a persistent read session on their stream. When one agent appends a finding, it is pushed immediately to the others. Discussion unfolds as a live, ordered exchange grounded in specific prior records, enabling genuine multi-turn reasoning inside a cohort.

If a process crashes midway, the prior records remain. You can resume from the tail, or replay from the beginning to audit how a conclusion formed. Since S2 assigns timestamps to records, you can compare what different groups believed at a moment in time without merging their histories.

parallax

To explore this architecture in practice, I built a simple tool. parallax is a CLI that accepts a research question, instantiates multiple cohorts of agents (Claude, Codex, or any agent capable of emitting structured output), and assigns each cohort its own stream in S2.

Then it kinda just lets it rip.

It also handles convergence. A moderator agent reads across cohort streams during the run, deciding when to steer groups, spawn breakouts, or transition phases. Once the run concludes, parallax performs a synthesis pass across the selected streams and writes a final report.

As it turns out, using streams makes topology experimentation effectively plug and play. Rather than hard-coding coordination patterns into prompts or agents, you can experiment with new architectures simply by reconfiguring the stream graph that connects them. The architecture admits patterns such as:

Dynamic moderation: A moderating agent can build coordination strategy on the fly by rewiring the stream topology as the discussion evolves. Since streams are just append-only logs, you can create, fork, or merge them at any point — the coordination structure doesn't have to be decided upfront.

Applied epistemology: Run thought experiments across AI models and observe how opinions evolve across independently formed positions and structured convergence rounds. This can serve as an empirical lab for testing moral hypotheses.

Parallelized engineering workflows: Make distributed debugging, building, and researching tractable. Because S2 streams are durable and shared, agents running across different machines, operating systems, or worktrees can coordinate without any additional infrastructure, and even join an active run via parallax join. The stream is the coordination layer, with concurrency controls like fencing tokens and conditional appends baked in. You can also monitor those streams in real time and observe what each agent knows and when, which makes it possible to catch a group drifting off-course before the reasoning compounds into bad output.

Examples

Adversarial cohorts

parallax research \
  "What is S2's (s2.dev) competitive positioning in the \
streaming infrastructure market, and what adjacent \
opportunities should they pursue?"

The idea is that different personas, even on the same underlying model, can influence how reasoning unfolds. parallax assigns each cohort a distinct personality, so you get genuinely different angles rather than cosmetic variation.

Delphi forecasting

parallax research \
  "What percentage of production AI agent deployments will \
require durable stream infrastructure (not just ephemeral \
message passing) for coordination, memory, and auditability \
by 2028?" \
  --hint "delphi forecasting, 3 rounds. Include one codex \
panelist that analyzes existing open-source agent frameworks \
to check what infrastructure patterns they actually use today, \
grounding the forecast in code rather than speculation" \
  --groups 5

Five independent panels converged on 73% across three rounds starting from a wider spread. The Codex panelist focused on actual open-source agent frameworks and was the most bullish. It turns out agent frameworks already reach for queues and logs when things get stateful.

But anyway, we are betting on 100%, AI agents can sit this one out.

Conclusion

So what was the point of all this?

It was not just to introduce a tool or describe an architecture – but to also argue about how intelligence can be built and revive something older: the Socratic idea that understanding emerges from disagreement, not from a single uncontested source. Capability alone is not enough as structure also determines whether a result is convenient or credible. Distributed agents change the conditions under which conclusions are formed, as they can turn disagreement and parallel reasoning into a design choice.

If AI is going to shape complex engineering decisions, forecasts, or moral trade-offs, then how it arrives at those conclusions matters just as much as the conclusions themselves. Did the reasoning hold up when challenged from an independent context? Can you replay the full chain of thought? Streams give you that — every record is durable, ordered, and attributable. That's not a feature of the tool, it's the point of building on this kind of infrastructure.

With S2, we've tried to keep the core data plane API as simple as possible. This is deliberate – we're trying to be a new type of storage primitive, and a big part of that is being clear about the interface and therefore which guarantees you can and cannot expect from S2.

We feel that some functionality is best left to users. But I can see how this sounds a bit like learning how to draw an owl:

To help bridge the gap, I want to explore some patterns for dealing with complexities that may come up when building systems around S2.

S2's data plane

To start, let's first consider what S2 gives you. The central data structure is a stream: an append-only sequence of records. S2 makes streams durable.

If you have not seen S2 before, the concepts documentation walks through basins, streams, and records in more detail.

Records in S2 are immutable and receive a sequence number expressing their ordering within the stream. Streams can be bottomless – they can grow indefinitely in size – or can be trimmed explicitly or based on TTLs.

All appends to a stream are fully durable on object storage before acknowledgement, and writers can coordinate with concurrency controls.

What functionality is missing?

Schemas and typing

Records in S2 are a very minimalistic abstraction. A record has a sequence number, a timestamp, optional key-value headers, and a binary body. S2 is entirely agnostic about what you store in those headers and body.¹

const s2 = new S2({
  accessToken: process.env.S2_ACCESS_TOKEN!,
});
 
const stream = s2.basin('my-basin').stream('my-stream');
const session = await stream.appendSession();
 
// Append a single record.
const ack1 = await session.submit([AppendRecord.make('record 1')]);

This just means that you need to bring your own serialization and deserialization logic for appending to and reading from S2 streams. S2 only speaks bytes. There are many ways of doing this, as we will see.

Here, S2 and AppendRecord come from the TypeScript SDK (@s2-dev/streamstore).

Large messages

Depending on the type of data you are serializing, you may run into another limit: the maximum size of a record on S2 is 1MiB.

This is big enough to not be a concern for many use cases, but it adds complexity. Maybe you are storing messages from an AI assistant, for instance – typically these are tokens, but occasionally they might be entire files, or images, or audio clips, which could easily exceed 1MiB. What to do?

There are two general approaches for dealing with this:

You can store a pointer to the data. Instead of putting the image in an S2 record, you could store a URL to an S3 object or similar. The advantage to this is that your data can still be represented by a single message. The downside is that now there is another service in the mix, which also has to be accessible both by your stream writer and any readers.

Alternatively, you can implement a framing scheme, and essentially spread your data across multiple records. S2 can continue to be the single source of data for your application, but there's the complexity of implementing framing logic, which will also need to be resilient to failures.

Duplicates from retries

S2's SDKs provide transparent retry logic for transient failures. But, this gets tricky with datastores.

What happens if you retry an append which failed in an "indefinite" way – for example, due to a network timeout? Perhaps your original attempt did succeed, even though the call ultimately failed. Retrying this append would therefore result in record duplication on the stream.

For some applications, the possibility of duplication is acceptable. For others, it's not. This can be particularly treacherous if you are framing large messages over multiple records! What options do we have?

Well, you could simply not retry any failed appends. The SDKs specifically allow you to configure whether or not to retry appends. But then it shifts the burden onto the user for inspecting if their prior attempt succeeded or not.

You could use match_seq_num as a concurrency control. This is the S2 equivalent of a CAS operation. This works, though it again throws retries back to the writer.

Alternatively, you could inject data that allows readers to perform deduplication. This can be the simplest option, as it allows you to configure the SDK to retry as much as it wants. If duplicate records are stored on the stream, readers can disambiguate them by inspecting the data.

Patterns for dealing with these issues

We want a strongly typed S2 client. We want to be able to store large messages safely. And we want to avoid duplicate records while still allowing retries for the smoothest possible user experience.

There are many ways of accomplishing this. Here's one! We will start in reverse.

Dealing with duplicates

Duplicates are fine so long as any reader of our stream can identify and remove them. A simple and efficient way of doing this is simply for the stream writer to include an idempotency key.

A really simple key could just be a writer-assigned index, stored in a header.

From the perspective of the stream writer, this would look like:

{ "headers": [["_index", "0"]], "body": "Hello" }
{ "headers": [["_index", "1"]], "body": "world!" }

If a retry occurs, it would be possible for a reader to see a duplicate:

{ "headers": [["_index", "0"]], "body": "Hello" }
{ "headers": [["_index", "0"]], "body": "Hello" }
{ "headers": [["_index", "1"]], "body": "world!" }

In the reader code, we just need a filter. It can increment its internal counter every time it sees a new record with a higher index. If it ever sees a record with the same or lower index, it can assume it's a duplicate, and simply ignore it.

What if a writer crashes entirely and needs to append to the same stream though? Or if there are multiple writers? We need an extra bit of information to identify the writer, such as a UUID.

{ "headers": [["_writer", "x-B1At9xVppm"], ["_index", "0"]], "body": "Hello" }
{ "headers": [["_writer", "x-B1At9xVppm"], ["_index", "0"]], "body": "Hello" }
{ "headers": [["_writer", "x-B1At9xVppm"], ["_index", "1"]], "body": "world!" }
{ "headers": [["_writer", "uSEFRpXqIldw"], ["_index", "0"]], "body": "How cool." }

Using an idempotency key like this allows the reader to reconstruct the actual flow of records, even across retries and writer crashes: ["Hello", "world!", "How cool."]. With both _writer and _index present, the reader just keeps track of the highest _index it has seen for each _writer, and drops any record whose _index is not greater than that per-writer maximum.

In the TypeScript patterns library (@s2-dev/streamstore-patterns), this idea shows up as _dedupe_seq and _writer_id headers added to each record, and a DedupeFilter that drops duplicates based on that pair.

Storing large messages

With the dedupe strategy, readers can confidently reassemble the exact sequence of messages that were appended, even if the appender experienced failures and generated duplicate records.

Similarly, writers and readers can co-operate to store large messages over multiple records. This is "framing" generally. Again, many ways of doing this, but a simple one is to use a framing header.

When a stream writer appends a message larger than 1MiB, it can split it into multiple <= 1MiB records, and include headers for bookkeeping. In our example implementation, whenever we map a message into multiple records, we inject headers on the first record to capture the total number of records that make up the message, plus the size in bytes of the payload.

Readers can consume the stream, and start assembling a new message whenever they see these frame headers. The size hint can be used for allocating a buffer, and also for detecting truncation.

Truncation might happen if the stream writer crashes before it completes sending all of the records comprising a full framed message – it is fair to simply ignore any messages that can't be completely reassembled.

This is a very simple framing scheme, but it's a good starting point. If you plan to have multiple concurrent writers to the same stream, you'll need to add additional logic for parsing interleaved messages – but for single writer / multiple reader architectures, this works well.

Strong types

If our stream writer and readers are collaborating to dedupe any repeats, and to frame large messages, then it's pretty simple to add support for strong types. We just need some way for our type to be serialized to bytes (for the appender), and be deserialized from bytes (for the reader).

If you are working with largely textual data, this could just be JSON. You could also use binary formats like MessagePack or Protocol Buffers. Anything works!

Putting it together in TypeScript

To make this a bit more concrete, here is a small example that combines all three ideas — strong typing, framing for large messages, and deduplication — using the helpers in @s2-dev/streamstore-patterns.

While the examples here are in TypeScript, the patterns themselves are S2-level ideas. You can apply the same approach with any of the S2 SDKs or even directly against the REST API by managing framing headers and dedupe keys yourself.

First, we define a multimodal message type and a ChatMessage that wraps it. This is the message type we want stored in our stream.

import { S2 } from '@s2-dev/streamstore';
import { serialization } from '@s2-dev/streamstore-patterns';
import { encode, decode } from '@msgpack/msgpack';
 
type MultimodalMessage =
  | { type: 'text'; content: string }
  | { type: 'code'; language: string; snippet: string }
  | { type: 'image'; bytes: Uint8Array };
 
type ChatMessage = {
  userId: string;
  message: MultimodalMessage;
};

Now we can write messages to an S2 stream. The SerializingAppendSession will take care of serializing to bytes, splitting large payloads into frames, and tagging each record with a dedupe sequence so retries are safe:

async function writeMultimodalConversation() {
  const s2 = new S2({ accessToken: process.env.S2_ACCESS_TOKEN! });
  const stream = s2.basin('my-basin').stream('my-stream');
 
  const appendSession = new serialization.SerializingAppendSession<ChatMessage>(
    await stream.appendSession(),
    (msg) => encode(msg),
    { dedupeSeq: 0n }
  );
 
  await appendSession.submit({
    userId: 'alice',
    message: { type: 'text', content: 'hello world' },
  });
 
  await appendSession.submit({
    userId: 'alice',
    message: {
      type: 'image',
      // Imagine this is a large screenshot or image; the patterns
      // helpers will automatically frame it across multiple records.
      bytes: new Uint8Array(5 * 1024 * 1024),
    },
  });
}

On the read side, DeserializingReadSession will dedupe, reassemble frames into full payloads, and hand you back strongly-typed ChatMessage values that you can branch on:

async function readMultimodalConversation() {
  const s2 = new S2({ accessToken: process.env.S2_ACCESS_TOKEN! });
  const stream = s2.basin('my-basin').stream('my-stream');
 
  const readSession = new serialization.DeserializingReadSession<ChatMessage>(
    await stream.readSession({ tail_offset: 0, as: 'bytes' }),
    (bytes) => decode(bytes) as ChatMessage
  );
 
  for await (const msg of readSession) {
    switch (msg.message.type) {
      case 'text':
        console.log(`[${msg.userId}] ${msg.message.content}`);
        break;
      case 'code':
        console.log(
          `[${msg.userId}] (${msg.message.language} code) ${msg.message.snippet}`
        );
        break;
      case 'image':
        console.log(
          `[${msg.userId}] <image bytes=${msg.message.bytes.byteLength}>`
        );
        break;
    }
  }
}

You can derive your own patterns from here: for example, using different streams per conversation, or layering your own schema/versioning metadata on top of the framed messages.

Conclusion

You can find a SerializingAppendSession<Message> and DeserializingReadSession<Message> in TypeScript that implement the strategies discussed above in the @s2-dev/streamstore-patterns package.

We've just scratched the surface of what's possible to build on top of S2. If you have a use case that doesn't fit nicely into the pattern above, reach out and we would be happy to help see if S2 might still be a good fit.

A common approach to building such a collaborative "room" is to have a designated coordinator that every client talks to, paired with a background worker that periodically takes the in-memory live state and persists a checkpoint to storage. However, if the coordinator crashes, the latest checkpoint becomes the only recoverable state. This could result in losing a significant amount of work!

To fix this issue of lost state, we can introduce a durable log or a journal into the mix, which is more optimal for incremental writes. So, even in case of a crash the recovery process can catch-up from the backlog in the journal which was not yet persisted as a checkpoint. As an example, Figma took this approach to make their multiplayer editing more reliable.

In the open source world, Yjs is a foundational CRDT framework that solves the hard problem of merging edits without conflicts. What it does leave open is how those edits are transported, stored, and replayed — but thankfully with pluggable backends!

Many Yjs backends are available, covering the gamut from y-redis to y-durableobjects. And now, there is y-s2 — but, really, y S2?

Durable streams for multiplayer

With S2, we took the humble log — the stream — and turned it into a first-class cloud storage primitive. Instead of working with entire objects, you interact at the granularity of records using a simple API akin to object storage: Append, Read, and Trim on a named Stream inside a Basin. Every record is durably sequenced at the current "tail" of the stream, no matter how many writers are active.

This means an S2 stream can serve both as storage and reliable transport: you can not only follow live updates, but also replay history from any position.

These properties makes S2 a natural fit for collaborative systems — and pairing S2 with serverless functions like Cloudflare Workers makes it straightforward to separate client vs server-side concerns. This is the magical combination that allows y-s2 to be a serverless yet durable Yjs backend.

To see it in action, head over to this demo and start a sharable collaborative editor.

A client connects to a worker through a WebSocket connection, and in turn the worker establishes an SSE (Server-sent events) session to receive live updates from the S2 stream. Every worker invocation effectively acts like a thread bound to a client: it reads from the log to propagate updates downstream, and appends any new updates from that client back to the stream.

When a client connects, the worker performs a catch-up to restore the latest state:

1. Load the checkpoint: Fetch the most recent snapshot from object storage (R2 in this case), which includes the last processed sequence number in its metadata.
2. Replay recent updates: Read all records from the S2 stream between the checkpoint and the current tail, applying them to rebuild the current state.

At this point, the worker synchronizes the materialized document with the Yjs client, and switches to live mode: streaming document updates in real-time.

Further, each worker reactively attempts to create checkpoints after buffering updates in memory until a threshold is reached. Since every update is already durably written to the S2 stream, the buffer is only an efficiency hack — it saves the worker from re-reading those same records when composing the snapshot. But if every invocation races to checkpoint at the same time, we waste resources! Additionally, checkpointing needs to be coupled with trimming the prefix of the stream up to that point — otherwise, the stream will grow unbounded even after data has been safely persisted.

How do we ensure that only one worker at a time is allowed to write a checkpoint and trim the log, while the rest continue serving clients?

A distributed mutex?

Martin Kleppman discusses fencing in his excellent article, how to do distributed locking. As he posits, the store needs to be involved for this approach to be robust — and fencing is a native capability in S2.

A fencing token can be set on a stream by appending a special "command record". Fencing in S2 is cooperative: an append that does not specify a fencing token will still be allowed. However if an append does include a fencing token that does not match, this results in the request failing with a 412 Precondition Failed. We can leverage fencing in this way to give each worker invocation an opportunity to obtain a unique "lease", or a time-bounded license, to create a checkpoint.

Another enabler to the design is that a batch of records is always appended atomically to a stream, and explicitly trimming a stream is also a command record. This means resetting the fencing token and trimming the stream can be committed together when a checkpoint has been completed, ensuring state stays consistent.

To create a checkpoint, an invocation attempts to set a unique fencing token on the stream with a unique ID and deadline ("{uuidInBase64} {deadlineEpochSecAsStr}"). The worker always provides its knowledge of the current fencing token, and this way if another worker won the race, the attempt will fail.

If a snapshot takes exceptionally long, or a worker invocation fails to release the lease due to a failure, the deadline acts as an auto-expiration that allows other invocations to attempt to grab the lease again. While it is possible that multiple invocations end up attempting a snapshot in this case, only one will manage to commit.

Side quest: making workers easier to debug

In a distributed scenario like this where we have multiple invocations trying to coordinate over checkpoints, we wanted to see logs in two different ways:

1. Logs per invocation: Useful for debugging issues specific to one worker's behavior — like understanding why a particular invocation failed to acquire a lease, how it processed its buffered records, etc.

2. Interleaved logs for all invocations: To get a chronological, comprehensive view of logs from all worker invocations. This would help understand the system-wide coordination and timing between different workers — like seeing the sequence of lease acquisitions, which worker successfully set a fencing token, how race conditions play out in real-time, and the overall flow of the distributed checkpointing process across all active invocations.

It takes a while for worker logs to appear on the Cloudflare dashboard and even toggling into its "Live" mode, the logs lag and sometimes error out. When it does succeed, the total volume of events is limited to only 256 KB which is not enough for long-running sessions. Moreover, locally when using wrangler tail or wrangler dev, worker logs for WebSocket connections never seem to show up! The overall experience was not pleasant.

I found it much more ergonomic to use S2 as the log sink and tail logs in real-time to see how workers coordinate over checkpoints. For traceability, it was easy to name individual streams by their assigned Cloudflare Ray ID. S2 supports creating unlimited streams, so every invocation can have its own dedicated log for inspection.

And it is just as easy to interleave them on a shared stream:

$ s2 read s2://hello-world/logs/workers-shared

We are only scratching the surface of possibilities here. If you are curious about building with S2, experimenting with collaborative backends, or helping improve y-s2, join us on Discord!

A prior version of this post included an inaccurate cost comparison with Cloudflare Durable Objects that was not apples-to-apples.

Because it's important, we also need to test it! We can gain confidence that S2 is linearizable by taking an empirical validation approach, using a model checker like Knossos, or Porcupine.

Last week I wired up a system for validating this property using S2 logs collected from our deterministic simulator. This post is a summary of that work.

Context

We’ve written a bit about our approach to testing S2 for correctness using deterministic simulation testing ("DST"). In short: we use turmoil to run the components of our distributed system — which, in production, are separate processes on different network hosts — within one, single threaded, Tokio runtime. The core of S2 is already in Rust, and we make use of Tokio, so turmoil is perfect (though we do have to mock external dependencies, like FoundationDB and S3).

Our setup provides both a deterministic environment for testing our system, as well as levers for injecting faults. We can run randomized workloads on it — which we do both as a CI step, and in much larger volumes in a nightly suite — and monitor for configurations (specific inputs, types of network stress, etc) that trigger violations of important invariants.

When an assertion is failed, we can easily recreate the exact steps which led to it just by restarting the simulation using the same seed.

It's hard to overstate how revolutionary this has been for our ability to build S2! At this point I can't imagine building infra without a DST framework.

While we assert on a lot of different conditions in our simulations, some correctness properties aren't feasible to evaluate during test runs as Rust assertions, and require additional tooling to verify after the fact.

Linearizability is an example of one such property — it’s a strong consistency model for distributed systems. If you aren’t familiar with it, a lot of great material has been written about what it entails. This post, by the author of the Porcupine checker, is a great place to start.

I’ll borrow from Martin Kleppmann’s summary in DDIA:

In a linearizable system, as soon as one client successfully completes a write, all clients reading from the database must be able to see the value just written. Maintaining the illusion of a single copy of the data means guaranteeing that the value read is the most recent, up-to-date value, and doesn’t come from a stale cache or replica. In other words, linearizability is a recency guarantee.

What to test

If you're not familiar with S2 (hello then, in that case!), the core data structure is the stream, which is an append-only log.

The API is very simple: you can append batches of records to the "tail" of a stream, read sequenced records starting at any position in the stream, and check-tail to see what the next assigned sequence number will be on the stream (i.e., the value of the "tail").

S2 Stream Operations

For an S2 stream to be linearizable, it just means:

(We also make strong consistency guarantees about S2's concurrency control mechanisms, like fencing tokens, but we’ll get back to that later.)

Verifying linearizability

We have a sense of what linearizability in S2 entails — how would we actually verify it?

I decided to try out Porcupine. It evaluates whether a client-observed log of concurrent calls to a system can be assembled into a total ordering in a way that satisfies linearizability.

Porcupine doesn’t test a live system directly, but rather evaluates a model of the system using a log collected from the real thing.

So first we need to represent S2 as a Porcupine model, then we need a way to collect relevant access logs from S2.

Modelling S2 in Porcupine

Linearizability validators are often demonstrated on "register" objects — key-value pairs essentially.

If you squint, an S2 stream can also be viewed as a type of register, where:

• Key = the name of the stream
• Value = a tuple of (tail, last_written_record)

For simplicity, we can store a hash of the last record, instead of the content itself.

To flesh this out a bit, we can see how the state of an actual S2 stream, and our associated register model, change in response to some successful append calls:

Stream State:                                    | Register State:
┌──────────────────────────────────────────────┐ │
│              Empty Stream                    │ │ (tail=0, content=0)
│                                              │ │
└──────────────────────────────────────────────┘ │
                         │                       │
                         ▼ append([a,b,c])       │
┌──────────────────────────────────────────────┐ │
│    seq:0    seq:1    seq:2                   │ │ (tail=3, content=hash(c))
│      a        b        c                     │ │
└──────────────────────────────────────────────┘ │
                         │                       │
                         ▼ append([d,e])         │
┌──────────────────────────────────────────────┐ │
│    seq:0    seq:1    seq:2    seq:3    seq:4 │ │ (tail=5, content=hash(e))
│      a        b        c        d        e   │ │
└──────────────────────────────────────────────┘ │
 

Since batches are appended atomically, we never expect to see intermediate states like (tail=4, content=hash(d)) — either the entire batch ([d,e]) would commit, or none of it does.

Step function

A model requires an initial state — for us, that’s just (tail=0, content=0) — and a step function which describes how to transform the register state, given an observed call input and output.

The model's step function signature looks like: step(current_state, observed_input, observed_output) → (is_legal, new_state). In other words, we need to provide a pure function that maps our prior register state, and the observed call/response, into a response indicating if a new state can legally be constructed — and if so, providing that resulting state.

The logic for the function depends on the input type (i.e., the type of call being made):

Note that only append will actually alter the register! Both read and check-tail will simply return the original current_state unchanged, unless an inconsistency is detected.

Porcupine uses this model to essentially solve for a linearizable sequence of transformations to our state — and will let us know if it’s impossible to do so.

Definite and indefinite failures

The model sketch above assumes that calls observed from S2 always succeed, but of course that’s not the case.

With calls that don’t cause side-effects, namely read and check-tail, failures are pretty simple to handle — we can simply change our model to reflect that an output’s tail or content values are optional, and won’t be provided if the call fails.

Appends are trickier, since they mutate the register. In S2, if an append call fails, we don’t necessarily know if the records became durable or not — and therefore, in our model, we don't know if we should update the tail and hash in the register or not.

Some failures may be definite, of course. For example, a validation error from S2, which tells us concretely that the batch was rejected; for these, we know there should be no change to the register.

But there's a wide class of indefinite failures for which we can’t know, just by looking at the output, if they took effect.

Porcupine lets us deal with this ambiguity via the NondeterministicModel type. Step functions in these models return a set of all possible state transformations, meaning that an append which receives an indefinite failure can, in our model, be witnessed as either producing an unchanged original state, or a state in which the batch was made durable and added to the tail. Both options can legally be part of a linearizable history.

An updated step function for indefinite failures would then look like this:

A linearizable history may pass through either one of these possibilities, but no others.

Is the model too basic?

In S2, a read needs to be capable of returning all prior acknowledged writes, not just the latest one. People aren’t using S2 as a KV store after all. Yet the model I’ve sketched above treats reads simply as a mechanism to receive the last record on the stream.

Our model would be more robust if the register value reflected the entire state of the stream, and each read returned that state.

When I originally thought about this, I decided that would be overkill, since S2 already contains many assertions around checksumming, and continuity of returned sequence numbers. These assertions in our code would halt and fail the DST even before we could collect a set of logs for Porcupine. But it’s true, this model could be made stronger by capturing the entire stream in the register.¹

Collecting concurrent histories

Porcupine can deal with either timed, or relative orderings for start/end events. To that end, I made a basic Rust library that performs a randomized workload on a single stream.

Rust is significant because it allows us to run the code within our internal turmoil-based DST — but anything could be used to collect these histories.

My code simply specifies a set of concurrent clients (Tokio tasks), which randomly alternate between read, check-tail, and append calls.

Any time a call starts or ends, the client emits a log event. These logs look like this:

{
  "event": {
    "Start": {
      "Append": {
        "num_records": 3,
        "last_record_xxh3": 375918985
      }
    }
  },
  "client_id": 19,
  "op_id": 6603
}

This is followed, some time later, with the corresponding op_id's return:

{
  "event": {
    "Finish": {
      "AppendSuccess": {
        "tail": 1683
      }
    }
  },
  "client_id": 19,
  "op_id": 6603
}

Model outputs

Our porcupine model can then try to assemble a linearizable history from our log. This even yields a neat visualization, where each step also displays the specific next possible states.

Example of a Porcupine visualization. The line displays a valid linearizable sequencing of events through the observed histories of concurrent callers.

Deterministic simulation

At this point, we have the capability of collecting concurrent histories, and validating them for linearizability with a Porcupine model of S2. But, given this is an empirical test, we really want to be systematic in how we collect these histories.

For instance, it's important to validate that histories collected even when the system is under extreme stress are linearizable. Our DST allows us to do that, by giving us a platform for:

• Running parallel test invocations under different workload configurations
• Injecting network faults
• Causing component crashes or other loss of availability

And, thankfully, in a reproducible environment, where reusing the same seed lets us experience the same sequence of events.

Linearizability validation now happens as a step in our nightly suite.

Our sleek nightly test UI.

An interesting gotcha

When I first wired this up and actually started to run the Porcupine model in the context of our nightly suite, I discovered a number of trials which were reporting violations of linearizability. I started to see my professional life flashing across my eyes.

In reality, I was just discovering a flaw in my model, regarding how I was handling clients which experienced indefinite failures while appending — one which is probably familiar to followers of Jepsen content, but which I had failed to internalize and had to discover via the DST.

Recall that the concurrent clients used to collect logs for Porcupine will randomly switch between operations. In my initial setup, a client would continue making requests, even after experiencing an indefinite failure for an append (e.g., due to hitting a timeout). The client would simply report the indefinite failure, then move on to the next randomly selected operation — but this is unsafe!

Consider this sequence of events. For simplicity, assume there is only one concurrent client interacting with the system, and all operations are sequential:

This is totally possible to experience in my setup! In Step 3, it is true that the prior append had not become durable; but it is not true that it would not become so at a later moment. If we considered Step 2 to have ended with the indefinite failure, then this looks like a violation of linearizability.

How can this happen in the first place?

It’s possible that the prior indefinitely failed append was able to submit the batch to S2 before a network failure caused the client error — but S2 did receive it, it just hadn’t flushed it to object storage yet and thus made it durable.

Or alternatively, even if S2 had never received the batch at all, it’s possible that a prior append’s payload is caught in some network switch somewhere, indefinitely delayed, only to become unstuck in the future.

What to make of this

This isn’t actually a problem for linearizability of S2! The solution is just to consider any client that experiences an indefinite failure as essentially unsafe to continue using. In the history we feed to Porcupine, the end time of any indefinitely failed appends should be set to a moment after all other operations complete. This models the fact that these calls could still take effect on the register at any moment in the future. In a sense, the call can never be witnessed to have ended, because there is never a point at which we can definitively say it will no longer be possible to have an effect.

In a practical sense, if you are a user of S2, this might sound like a technicality: sure, it's linearizable — but if you experience a timeout, how do you actually proceed?

If you find yourself in this position, and need to know precisely if a failed append can or cannot become durable in the future, S2’s concurrency control mechanisms can achieve this.²

Additional steps

With pretty minimal tweaks to our model, we can also incorporate logic around our concurrency control mechanisms — i.e., that any append which specifies a match_seq_num (our version of a CAS essentially) must only succeed if the register’s tail matches the provided sequence number. We can also expand the register to include fencing tokens. Establishing a token via a fence record is a strongly consistent operation in S2, and by modeling that as part of the state we can be more confident that this is always the case.

In addition to our homegrown DST, we've also recently started experimenting with Antithesis's deterministic testing environment, and run this model as part of the workloads sent there. More about our experience with Antithesis soon!

If you're interested in the Porcupine model, check out the code on GitHub!

S2 delivers streams as a cloud storage primitive, which means no limit on the number of streams you can create and access, with a simple, serverless API and pricing model.

The dominant pattern in durable streaming today is to use sharded topics as a "firehose". This works fine for transporting, say, clickstream data into a data lake. But it is the wrong fidelity for a rising class of applications: agents.

Unlike a synchronous and stateless request-response cycle, sessions are often deeply stateful and long-running. For example, consider AI agents handling travel booking, customer support, or legal assistance; balancing learned preferences over multi-turn conversations.

Each session charts a distinct path with its own context, decisions, and outcomes. It sure quacks like a stream!

Landscape

Traditional streaming will have you choose between:

• Small number of durable topics (Kafka¹, Kinesis²) — multiplexing unrelated sessions through shared pipes
• Expensive ephemeral state (Redis³, NATS⁴) — sacrificing durability for granularity

You might then turn towards a more general database with a row-per-event model. For a streaming workload, OLTP databases (like Postgres) imply high costs ⁵, while OLAP databases (like ClickHouse) come with very high write latencies ⁶. Neither will let you tail a fine-grained event stream.

Most databases are designed for where you landed up — streams are the journey. And with agents, the specific journeys matter.

Why agents need granular streams

Memory

Write-ahead logs (WALs) are a fundamental technique for databases, and have a parallel in the practice of event sourcing for applications. It applies perfectly to building agents you can rely on.

The high-level idea is to use what is in effect an "agent WAL". All activity is modelled as events on a session-specific durable stream — so this includes user inputs, prompts, model responses, and tool calls.

This gives us a solid foundation for agent memory. A sketch:

Working memory

The session-specific stream serves as a replayable source of working memory for an agent. When an agent experiences any kind of interruption and needs to be reincarnated, the context can be completely restored quickly.

However, the perspective of working memory as merely being context for an AI model is narrow. A unified treatment includes resilience to crashes, i.e. can the agent truly remember and resume where it left things off? What if the latest we know, based on the stream, is that the agent noted its intent to call an API, but the result was not recorded?

The service being invoked must cooperate to facilitate idempotent retries, usually by allowing the client to provide a unique token. Then this idempotence token can be stored alongside the intent, and voilà, it is safe to retry and resolve the result.

If you have come across durable execution frameworks/runtimes, this is essentially what is going on, with varying degrees of intrusiveness. None will magically make a third-party API idempotent, and it is important to peek under the layers of abstraction to question if they are helping or obscuring.

Long-term memory

The working memory stream can be used to identify insights worth preserving. This may happen inline by instructing the primary model, or by replaying the stream into another model.

As long-term memories are extracted, they can be made available across sessions by writing them to an entity-level (per user, per project, etc.) stream. This naturally preserves the order in which those memories were made, which is meaningful as facts may be overridden.

While context window sizes continue to grow, they are not infinite, and model performance starts to degrade. A stream as a source of truth for long-term memories enables us to asynchronously and reliably build specialized indexes — lexical, vector, graph, or a combination — for efficient memory retrieval⁷ without loading the entire history.

Moreover, memories may be consolidated and reindexed as they grow in size, by using a versioning approach to stream and index resources. The previous versions serve as raw input for this compaction process — yet another time-tested pattern in data systems ⁸, that is being referred to as recursive summarization in this, er, context.

Auditability

Immutability is a cornerstone of AI applications that can be trusted with real-world consequences.

With agency in the mix, you need to be able to audit exactly what context went into going down a certain path. Why the $450 United flight was rejected for the $520 Delta option, how the user's "I need to arrive before 3pm" constraint eliminated 60% of options.

Streams are append-only, and the history is immutable by design. This unlocks a whole lot:

• trace a session for debugging
• detect patterns of abuse or manipulation, such as prompt injection attacks
• transparently share with your users how decisions affecting them were made
• comply with laws and regulations, such as non-discrimination
• construct evaluation datasets based on actual interactions

Isolation

Building secure multi-tenant agentic applications requires strict boundaries. We don't want context from one session bleeding into another due to accident or abuse — only by design, such as long-term memory for the same user.

Isolation is natural with granular streams. If you use S2, you can even enforce fine-grained access control.

Branching

When an agent reaches a critical decision point, you can:

1. Clone⁹ the stream at that point
2. Explore alternative paths in parallel
3. Compare outcomes
4. Merge the most promising path back into the main timeline

This enables sophisticated planning and "what-if" analysis without corrupting the primary flow. git checkout for agents!

In how we built our multi-agent research system, Anthropic note:

Asynchronous execution would enable additional parallelism: agents working concurrently and creating new subagents when needed. But this asynchronicity adds challenges in result coordination, state consistency, and error propagation across the subagents. As models can handle longer and more complex research tasks, we expect the performance gains will justify the complexity.

Granular, durable streams are a simplifying primitive to tackle these challenges.

Patterns over frameworks

We don't have a framework to sell you on. You can be as minimalist with these ideas as your application demands.

Event sourcing as a pattern has been around forever. It matches how state actually evolves: as a sequence of events over time. Agent sessions are no different, and AI models provide inference APIs expecting exactly this framing.

But here's what's different now: the infrastructure finally matches the pattern. You can create a durable stream for every session, every user journey, every parallel exploration — without worrying about extreme costs or operational complexity.

If you are excited to count on streams as a primitive, we would love to help! You can join our Discord, email us, or jump straight into the quickstart.

Avoiding delays by design

Traditional data integration approaches have relied on batch processing, where data pipelines periodically extract full or partial datasets and load them into downstream processors.

The apparent simplicity is a mirage:

• Bloated full dumps introduce significant overheads and latency as updates are only reflected with each batch cycle which may only be feasible hourly, daily, or even weekly.
• Periodic incremental loads with timestamp-based queries are certain to go wrong over time, as clocks are never perfectly synchronized.
• Real-time use cases that require reacting to specific state changes as they happen are not feasible, and this leads to managing application-level triggers.

Change Data Capture (CDC) flips this model by capturing changes as they occur in the database.

The idea is to leverage the database’s write-ahead logging, and produce a stream of logical changes. Instead of moving entire datasets, CDC streams deltas — insert, update, and delete operations — in real-time.

This is exactly what you need in a modern event-driven architecture.

        Batch Processing             │              Event-Driven
═════════════════════════════════════│════════════════════════════════════════════════
  Order Created                      │    Order Created ───▶ Stream ──▶ Services
  Payment Processed                  │    Payment Processed ─▶ Stream ──▶ Services
  Inventory Updated                  │    Inventory Updated ─▶ Stream ──▶ Services
  User Registered                    │    User Registered ──▶ Stream ──▶ Services
  Status Changed                     │    Status Changed ──▶ Stream ──▶ Services
       │                             │                            │
       │ (accumulate...)             │                            ▼
       ▼                             │                     ┌─────────────┐
  ┌─────────────┐                    │                     │  Real-time  │
  │ Process all │                    │                     └─────────────┘
  │  at once    │                    │
  └─────────────┘                    │
                                     │
Flow:                                │  Flow:
[──wait──][process][──wait──]        │  [→][→][→][→][→][→][→][→]
                                     │
Resources:                           │  Resources:
Idle ──── Spike ──── Idle            │  ~~~~~~ Steady ~~~~~~

Sequin just makes sense

In the current ecosystem of CDC tools, Debezium is quite established. However, it comes with significant operational overhead due to being built around Kafka Connect, a heavy-weight framework that has not kept up with the cloud-native times.

You may also know about Postgres triggers or LISTEN/NOTIFY. Both seem alluring, but come with gotchas:

• Postgres triggers can execute code in response to database changes, but they are constrained to PL/pgSQL and run synchronously within the transaction, impacting database performance.
• Similarly, LISTEN/NOTIFY provides a lightweight pub/sub mechanism, but offers only at-most-once delivery — if your consumer is offline or fails, notifications are lost forever, and there is no way to replay missed events.

Sequin has been developed as a modern, focused system that addresses these pain points. It eliminates the Kafka dependency and gives you the freedom to choose your ideal sink. It can run as a single Docker container that connects directly to a Postgres database, and stream changes to many destinations like SQS, HTTP endpoints, Redis – yes Kafka too – and now S2!

What makes Sequin particularly compelling is its focus on operational use cases - the scenarios where you need to trigger workflows, invalidate caches, send notifications, or keep services synchronized based on database changes. Unlike ETL tools like Fivetran or Airbyte that excel at analytical workloads but operate in batch intervals, Sequin delivers real-time streaming.

Making streams first-class

We have been building S2 as a true counterpart to object storage for fast data streams. S2 gives you streams on tap behind a very simple API, with no clusters to deal with whatsoever.

Each stream is totally ordered, and elastic to very high throughputs – 10-100x higher than most cloud streaming systems. So if you wanted to capture the precise order of all operations in a write-heavy table on a single stream, S2 can keep up!

On the other end of the spectrum from a high-throughput firehose, you can get as granular as you need to: streams are a first-class, unlimited resource in S2. We take responsibility for ensuring a high quality of service, such that a variety of streaming workloads can operate seamlessly.

Elastic throughput, bottomless storage, unlimited streams, and a serverless model where you only pay for usage – all make S2 an especially good fit for CDC from serverless Postgres databases, like Supabase, Neon, and Nile.

Trying it out

Let's look at how we can architect a simple and fast CDC pipeline from Postgres to S2 with Sequin.

Imagine you're managing an item inventory where you need to update prices and quantities, as new stock arrives or in response to changes in demand and supply. These updates need to be propagated to users in real-time, which can be a complex process as it may involve:

• Triggering notifications or alerts e.g. "Avocados are now $4.99!"
• Revalidating caches, as a product going out of stock may require invalidating various caches or updating search indexes, so users don’t end up trying to check out a product that is no longer available.
• Internal updates to adjust profit margins and pricing, which may be owned by other microservices.

You can also follow along the same example via Sequin's S2 quickstart.

Consider the following schema for our inventory:

create table products(
  id serial primary key,
  name varchar(100),
  price decimal(10, 2),
  inserted_at timestamp default now(),
  updated_at timestamp default now()
);

We will start by creating an S2 basin and generating an access token.

To create a basin, navigate to the basins tab in the S2 dashboard and click on Create Basin:

Enable the on append option in the Create Streams Automatically section so that you don’t have to explicitly create streams.

Now, create an access token by navigating to the Access tokens tab, and then clicking on Issue.

Use a fine-grained access token scoped to the exact basin or the stream you want to use as a sink for CDC, and restrict the allowed operations to Append only, or Append and CreateStream if you enabled automatic creation of streams.

Next, follow the Sequin Postgres guide on how to connect your database to Sequin. After connecting your Sequin dashboard, wait for the connection to be healthy, and then add a new sink by choosing S2.

Fill out the S2 configuration and click on Create Sink. Sequin will start sending CDC events to our S2 stream!

We can verify this by tailing our S2 stream to see updates sent to the products stream in the Sequin message format:

$ s2 read s2://sequin-quickstart/products
⦿ 3500 bytes (6 records in range 6..=11)
{"record":{"id":1,"inserted_at":"2025-06-11T01:19:22.402358","name":"Avocados (3 pack)","price":"5.99","updated_at":"2025-06-11T01:19:22.402358"},"metadata":{"consumer":{"id":"282713d8-76c4-42db-8e16-3e0df04cc5f6","name":"sequin-playground-s2-sink","annotations":null},"table_name":"products","commit_timestamp":"2025-06-17T19:19:51.955551Z","commit_lsn":null,"commit_idx":null,"transaction_annotations":null,"table_schema":"public","idempotency_key":"ODQ5NWNhMzEtYTM0MC00MDI5LThjODItMTcxYmRjMmY4YjI2OjE=","database_name":"sequin-playground"},"action":"read","changes":null}
{"record":{"id":2,"inserted_at":"2025-06-11T01:19:22.402358","name":"Flank Steak (1 lb)","price":"8.99","updated_at":"2025-06-11T01:19:22.402358"},"metadata":{"consumer":{"id":"282713d8-76c4-42db-8e16-3e0df04cc5f6","name":"sequin-playground-s2-sink","annotations":null},"table_name":"products","commit_timestamp":"2025-06-17T19:19:51.955968Z","commit_lsn":null,"commit_idx":null,"transaction_annotations":null,"table_schema":"public","idempotency_key":"ODQ5NWNhMzEtYTM0MC00MDI5LThjODItMTcxYmRjMmY4YjI2OjI=","database_name":"sequin-playground"},"action":"read","changes":null}
...

Your applications can now easily consume these changes using an S2 SDK or its REST API.

It is worth highlighting that Sequin is not a dumb pipe – it also allows you to easily filter, transform, and route events!

   Postgres Tables           S2 Streams
 
    ┌─────────┐   ┌──────┐
    │  users  │──▶│      │─── user_auth_events
    └─────────┘   │      │─── user_profile_updates
                  │      │
    ┌─────────┐   │SEQUIN│
    │products │──▶│      │─── inventory_updates
    └─────────┘   │      │─── price_alerts
                  │      │─── category_updates
    ┌─────────┐   │      │
    │ orders  │──▶│      │─── payment_notifications
    └─────────┘	  │      │─── order_processing
                  └──────┘

Your CDC pipeline awaits

Modern applications are expected to be reactive. CDC with the right tools makes change propagation from databases a fundamental building block, rather than an advanced capability reserved for companies with large infrastructure teams.

Now you can combine Sequin's simplified Postgres CDC approach with S2's serverless streaming experience, and ship real-time features faster.

Join us on Discord.

S2 provides a serverless stream, or log primitive, backed by object storage. You can think of it as the core of Kafka, just... liberated from all the infra associated with it.

We've been enjoying plugging S2 into anywhere streams appear, just to see what happens. Which leads us to:

What if we implemented a terminal on S2?

Terminals are basically streams, right?

Suppose I want a shell on a remote system. I'd probably run sshd on that system – sshd would serve the role of pseudoterminal (or PTY), by interacting with a local shell process – and sshd would then broker access to this shell for any clients that connect to the system, via the SSH protocol.

So what if we replaced sshd with our own pseudoterminal, which instead of running as a server daemon communicates entirely through S2?

As it turns out, this is pretty easy to hack together! Shoutout to Claude for vibing the frontend in particular.

How does it work?

For a pseudo-terminal, we just need to read inputs (keystrokes, window resize events, mouse clicks) from an S2 stream, delegate them to a shell process, and then stream the terminal output back onto another S2 stream.

In this demo, the client is simply a webpage with xterm.js, and we can coordinate with S2 streams directly from the browser over HTTP. I used the S2 TypeScript SDK, but plain old REST works too.

The PTY process is a small Rust binary, which interacts with S2 via (you guessed it) the Rust SDK, and makes use of the portable_pty crate.

See a video of the demo setup in action here. Or check out the repo and try it yourself.

This has to be incredibly slow, right?

Interactive latency in a setup like this is for sure higher than connecting directly to my server over SSH.

In my case, I'm running the PTY process on a VM in us-east-1, and I want to get a shell on it from my home in California, where I am running the frontend locally.

With an Express storage-class stream on S2, p50 end-to-end latencies are around 25-30ms, with p99 < 50ms – but this is for a client in the same region as S2. While we are in preview, S2 is only in AWS's us-east-1 – and since I live in California, I have to pay a pretty significant "speed-of-light tax" to get bytes to and from Northern Virginia. At least the weather's nice!

Let's consider what has to happen on every interaction (e.g. a single keystroke) for this S2 terminal:

1. Frontend appends the keystroke to an S2 stream (California -> us-east-1 + time to make a write durable within a region).
2. The Rust PTY, on the VM, needs to read the (now completely durable) keystroke (same region, on an already open streaming read session).
   • The PTY sends the keystroke to its follower process (a shell); that process may simply be echoing, e.g. if I'm just typing in a command prompt.
3. The PTY writes any output from the shell to an S2 stream (the VM running this PTY is in the same region as S2, so this is just time to make a write durable within a region).
4. The output needs to be read, by the frontend, from an S2 stream (us-east-1 -> California, on an already open streaming read session).

Here's what it looks like put together:

Flow of the s2.term setup

We can approximate the latency for steps 1 and 4 at the same time by running the S2 CLI's ping function, from my laptop in California, and looking at the end-to-end latency.

This ends up being around 124ms, p50.

Home (in California) to S2 in us-east-1 ping test.

And similarly, we can approximate latency for steps 2 and 3 by running that command from an EC2 VM in us-east-1.

This is around 31ms, p50:

Intra-region ping test.

This means each keystroke I make on the terminal frontend takes about 155 milliseconds to be reflected back to me, given my distance. This latency is dominated by the input and output stream appends. If I were just using ssh, the latency would be closer to 70 or 80ms.

... but 155ms is actually not terrible for a terminal? Definitely not the snappiest I've ever used, but probably not the worst either.

Why do this?

The honest answer is: I was bored last Friday and thought it would be fun to try out. And it was!

But there are some cool properties which emerge when you do this. For instance:

... the terminal is automatically multi-player

It's surprisingly good for pair programming.¹ You can have multiple people controlling a terminal without having to spin up a server, or grant SSH access (e.g. if sharing via tmux).

You also get fine-grained access-controls at a per-stream (or per-stream prefix) level. So you could easily choose to only grant read-only access (i.e., terminal viewers), or read-write but limit certain operations like trim.

Plus, you could even make use of concurrency primitives for further safety (e.g., ensure all terminal users are fully caught-up with the tail of the terminal before their keystroke can be appended).

These types of things become easy when your terminal is essentially a shared write-ahead log.

... all I/O is automatically saved

Every S2 write has to be regionally durable in object storage before it can be delivered to readers, and you can control how long you want to retain stream history for (or just trim it explicitly).

This has some neat implications – for instance, you can share a "replay" (Asciinema-style) of a pair programming session for someone who couldn't make it when it was going on live.

Or, maybe you want to broadcast content from a TUI dashboard running on a server, and allow people to scrub around and see values from earlier. Records are indexed by timestamp in addition to their sequence number, so you can easily hop around to points of interest in a terminal session, or replay at different speeds, like having a "rewind" button on your terminal. See a demo video of that here.

... no servers to configure

With sshd, you need to make sure users can connect to the daemon – meaning you might have to forward ports in a gateway, or otherwise deal with NAT traversal. With S2 serving as the streaming medium, both the client and the server just need to be able to connect to the S2 API (which can be done via REST, or one of our SDKs).

What would it cost?

It really depends on what you are doing in your terminal!

S2 is priced as a serverless commodity, along dimensions that will be familiar to anyone who uses object storage, making it easy to do some math based on expected usage patterns.

As a somewhat maximalist example, I wanted to see what it would cost if I turned btop – which is a dynamic TUI dashboard for monitoring system activity – running on a computer of mine into a shared dashboard by broadcasting that process over S2 (e.g., as I did in this video).

• btop produces around 800KiB of terminal output per minute
  • About 1.1GiB of data written in a day.
    • 0.066 for Express writes (as I opted for the lower-latency storage class)
    • 0.044 for storage for 24 hours
    • ~= 11 cents / day
  • Reads require us to estimate usage (how many people will be connected)
    • 3 users tailing continuously, all day long
      • $0.08 (per GiB over public internet) _ 1.1GiB _ 3 users
      • ~= 26 cents / day
  • Per-op costs
    • Total on the order of <1 cent per day.

So around 38 cents per day to share my btop and a day's worth of history with the rest of the office!

We have seen a lot of excitement around our object-storage-like API, and a recurring ask has been a parallel mechanism to pre-signed URLs.

If you are not familiar with it, pre-signed URLs are a neat way to grant temporary, controlled access to objects in your storage bucket. The URL embeds time-bounded authentication and authorization information, allowing anyone with the URL to perform a specific action (like PUT or GET) on a specific object until the URL expires.

You can see where this is going — S2 streams are already available at public endpoints over HTTP. The API simply needed to support more granular access control.

Now it does!

S2 supports an unlimited number of revokable access tokens that can be scoped to a set of resources and the operations that are allowed on those resources.

You can issue permanent access tokens for services, or time-bounded ones for ephemeral usage, e.g. for end users to access an S2 stream.

Just like streams, there is no limit on the number of access tokens. After all, a key use case is being able to grant access to streams for "edge" clients — be they browsers, apps, or agents.

You get the flexibility to scope resources by an exact name, or a name prefix. In fact, when scoping streams with a prefix, you can enable transparent namespacing with a knob telling S2 to auto-prefix.

To make this concrete, let's take it for a spin!

Usage example

First, a quick review of S2 concepts would be helpful.

We will assume we have a basin storing data from headless browser sessions, with streams like this:

$ s2 ls browser-instances-wtvr
s2://browser-instances-wtvr/session/g7wrzr9t/console 2025-06-03T18:05:19Z
s2://browser-instances-wtvr/session/g7wrzr9t/dom-events 2025-06-03T18:05:26Z
s2://browser-instances-wtvr/session/g7wrzr9t/telemetry 2025-06-03T18:05:31Z
s2://browser-instances-wtvr/session/rhc2q95x/console 2025-06-03T18:05:35Z
# ...

While we could issue a new access token with the CLI too, let's try with good old curl for a change. (Of course, you will need an existing access token, which was perhaps issued from the dashboard, or itself using the API.)

$ curl --request POST \
  --url https://aws.s2.dev/v1/access-tokens \
  --header "Authorization: Bearer ${S2_ACCESS_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{
  "id": "agent/g7wrzr9t",
  "scope": {
    "basins": { "exact": "browser-instances-wtvr" },
    "ops": ["list-streams", "append"],
    "streams": { "prefix": "session/g7wrzr9t/" }
  },
  "auto_prefix_streams": true,
  "expires_at": "2025-06-04T08:00:00Z"
}'
{"access_token":"YAAAAAAAAABoPzx+yjvI25QSuwCzq9nnFna3rZF2tSkqRnma"}

We can confirm we got that right in the dashboard:

Note the scoping above, and how if we hand this access token to our beloved agent g7wrzr9t, it can only observe streams under its prefix:

$ S2_ACCESS_TOKEN="YAAAAAAAAABoPzx+yjvI25QSuwCzq9nnFna3rZF2tSkqRnma" s2 ls browser-instances-wtvr
s2://browser-instances-wtvr/console 2025-06-03T18:05:19Z
s2://browser-instances-wtvr/dom-events 2025-06-03T18:05:26Z
s2://browser-instances-wtvr/telemetry 2025-06-03T18:05:31Z

... but it is completely oblivious to that naming scheme, the prefix has been stripped.

We did allow it to write to those streams ✅

$ curl --request POST \
  --url https://browser-instances-wtvr.b.s2.dev/v1/streams/console/records \
  --header 'Authorization: Bearer YAAAAAAAAABoPzx+yjvI25QSuwCzq9nnFna3rZF2tSkqRnma' \
  --header 'Content-Type: application/json' \
  --data '{ "records": [ { "body": "hello world" }, { "body": "nice to be online" } ] }'
{"start":{"seq_num":0,"timestamp":1748976301339},"end":{"seq_num":2,"timestamp":1748976301339},"tail":{"seq_num":2,"timestamp":1748976301339}}

... but not delete them 🚫

$ curl --request DELETE \
  --url https://browser-instances-wtvr.b.s2.dev/v1/streams/console \
  --header 'Authorization: Bearer YAAAAAAAAABoPzx+yjvI25QSuwCzq9nnFna3rZF2tSkqRnma' \
  --fail-with-body
curl: (22) The requested URL returned error: 403
{"message":"Operation not permitted"}

New possibilities

What would be the alternative to an edge client directly accessing a stream? The answer, as always in computing, is a layer of indirection. You would have to implement a proxy that takes care of authenticating and authorizing, and all data flows through.

For many use cases, this can now be obviated, and you can offload your streaming data plane to S2!

With horizontal building blocks like granular access control — and S2 itself, designed to deliver streams as a cloud storage primitive — the possibilities are endless. Nevertheless, let's consider some examples:

• Running jobs on behalf of your users, such as builds? Let their browser tail logs directly, without having to implement a streaming endpoint yourself. You can simply hand out a read-only access token to a stream that you write to from the build runner. Long-term storage with real-time tailing, converged.

• Multi-player applications like collaborative document editing — allow reads and writes against a shared stream, but forbid trimming. The stream can serve as an immutable ledger, and the backend can retain responsibility to checkpoint state and trim.

• Building an application powered by a sync engine — the tech used by Linear, Figma, and Notion to power amazing UX? You may want change logs that your clients get read-only access to, while accepting updates over a write-only stream.

• In the business of personalization? Ingest user events from your customers as they happen over a write-only stream, and provide them read-only access to user-specific recommendation streams.

• Exercising S2's MCP server, or developing an agentic app that reacts to real-time events? You probably want some guardrails for your AI with tight scoping.

What's next

S2 will keep solving for bringing streams online. We have some immediate followups on our roadmap:

• Massive read fanout: one stream being read by thousands of clients? We intend to be up to the challenge!

• Usage visibility and limits per access token: so you can grant untrusted code access without letting it go amok.

Reach out by email or our Discord if you want to find out more or share your requirements.

But what about good old physical time? How long it's been since Jan 1, 1970? (Hundreds of years from now, I reckon that will still be a pretty important reference point.)

In our internal tooling, we had been rolling with inserting a header to propagate timestamps, which felt like a reasonable pattern to recommend. But real-world applications care about real-world time, and it became clear we should make timestamping first-class.

Storing a timestamp on each record wouldn’t be too useful if you can’t also consume by it — for example, if you want to use an S2 stream as a long-term source of truth on location data like vehicle movements. With cheap enough stream storage and a querying pattern of linear scans from a point in time, it is appealing to not involve another indexed data store.

If we assume that we can count on monotonicity — time marching inexorably forwards with each record — S2 wouldn’t even need a separate secondary index!

Searching for the right place to start reading by timestamp can be framed very similarly to reading from a sequence number: the backend is essentially navigating a distributed skip list over metadata storage, object storage, and cache of recent writes.

However, in a distributed system, time depends on who you ask. Are we going to go with what the client says or the service? They have different clocks, and network delays between them.

Further, what if you are primarily concerned not with the time of writing to the stream, but as an attribute of when events actually happened — like a fitness tracker squirting data points when it comes back online?

Other systems

Let’s take a look at what some other systems do:

• Kinesis assigns an ApproximateArrivalTimestamp to records, and you can use this as a starting position for reads. It does not have any special knowledge of client-specified timestamps – those can be made part of the record if needed.
• Pulsar tracks broker-assigned publishTime and optionally client-specified eventTime. You can only 'seek' by the former – event time does not get indexed.
• Kafka allows for either a client-specified (CreateTime) or broker-assigned (LogAppendTime) timestamp per record. If the client does not provide a timestamp when using CreateTime, it gets silently replaced with the LogAppendTime. Record timestamps are also used in retention, so certain knobs to clamp the timestamp can be a good idea if using CreateTime.

Time in S2

Coming back to what we have landed on for S2, in the simple case, you just let the service assign monotonic record timestamps of the arrival time in milliseconds since our favorite epoch.

How does the backend ensure monotonicity? We force it! It is tracking the highest timestamp per stream, and uses that if the new timestamp is somehow (cough distributed systems cough) smaller.

inputs:       42, 44, 42, 50, 50, 48, 55, ..
adjusted:     42, 44, 44, 50, 50, 50, 55, ..

Unlike the sequence numbers assigned by S2, timestamps are allowed to be identical between consecutive records.

A key decision point was whether to allow client-specified timestamps. We decided some essential complexity here was worthwhile, because event time is far too useful when you consider scenarios like offline devices or backfills.

A timestamping.mode knob made sense to introduce in stream configuration, so users can know with confidence whether client or arrival time is used:

• client-prefer – (the default) use client-specified timestamp if present, otherwise the arrival time
• client-require – require clients to specify a timestamp that will be used
• arrival – use arrival time regardless of whether or not the client specifies a timestamp

By default the arrival time acts as a cap to prevent out-of-whack values messing with the stream’s notion of time. You can enable timestamping.uncapped and get full fidelity within the 64-bit range of possibilities, just remember that there is no going back: the automatic adjustment to ensure monotonicity applies to client-specified timestamps too!

Append acknowledgments contain the first and last timestamps for the batch, so you can know if an adjustment was made. If a use case demands, we can add a way to opt-out of this behavior and reject such an append instead.

Given the monotonicity constraint, we end up with not-quite-an-arbitrary index — but whether you stick with arrival time or propagate your own timestamps, the service stays cost-effective and lets you efficiently read records along this additional time axis.

$ s2 read s2://perso-ingest-mnbk/user/foo --ago 1h --format json --count 3
{"seq_num":8,"timestamp":1747225273458,"body":"search \"hats\""}
{"seq_num":9,"timestamp":1747225343334,"body":"view-listing 42"}
{"seq_num":10,"timestamp":1747225380872,"body":"add-to-cart 42"}
 
$ s2 read s2://perso-ingest-mnbk/user/foo --timestamp 1747225340000 --format json --count 3
{"seq_num":9,"timestamp":1747225343334,"body":"view-listing 42"}
{"seq_num":10,"timestamp":1747225380872,"body":"add-to-cart 42"}

We hope our approach will serve users well! Feedback very welcome.

We knew DST was the way to go in building S2, inspired by the prior art of FoundationDB and TigerBeetle. But how could we practically implement it for our Rust codebase that uses Tokio as a runtime? My goal in this post is to share some of what we learned along the way.

Unit tests can be made plenty deterministic. What’s different here is we are trying to exercise a lot of different scenarios (like a property test) and with a whole-of-system mindset (like an end-to-end integration test).

The construction should be such that with each run, you randomly chart a path through the huge state space. Randomness and determinism may sound contradictory, but each choice flows from a single point – the random number generator (RNG) seed.

The test subject

DST works best when you not only check invariants as a client, but also more granularly in the mainline code – a practice in defense-in-depth. Databases in particular should be liberal with assertions; far better to loudly crash when ACIDic properties are involved.

Incidentally, while some languages may treat assertions as something to be optimized away for production, I think Rust picked the right defaults here – it is a rare expensive check that we allow getting compiled away with debug_assert!.

But of course, we don’t want panics in production! To supercharge the long journey of building a robust distributed data system, the core of the actual test is synthesizing chaotic interactions over an extended period of simulated time. We run our DSTs on every PR, commit, and in thousands of nightly trials.

Beyond internal checks, the system must also maintain externally observable invariants that are best verified from a client's perspective as the simulation progresses, such as durability after a crash and API semantics around concurrent operations.

Making it deterministic

What must be tamed to create a truly deterministic simulation?

• Execution – Single-threaded to eliminate scheduler noise
• Entropy – All RNGs should have a known seed
• Time – No physical clocks
• I/O – No dependency on any external IO with components not part of the simulation, and subject to all the same constraints

That’s a lot of variables to control for. Happily, we were able to leverage a lot of existing open source work done by the community!

Tokio does have first-class support for running with a single-threaded scheduler. Internally, its clock is abstracted, and can run “paused” for testing, where time only advances on calls to sleep(). By using tokio::time::Instant instead of std::time::Instant, you can ensure any measurement of elapsed time is aligned with this clock. The runtime also has an internal RNG used in making scheduling decisions such as picking a branch for tokio::select! – but this can be seeded.

Where does this leave us with regard to IO? Here we adopted the Turmoil project, which presumes Tokio as a runtime.

Turmoil is a framework for testing distributed systems. It provides deterministic execution by running multiple concurrent hosts within a single thread. It introduces “hardship” into the system via changes in the simulated network. The network can be controlled manually or with a seeded rng.

Simulated networking is precisely what we needed, between logical 'hosts' in the same physical process. The ability to inject issues like latency and crashed processes is immensely useful for teasing out behavior under the unhappy paths.

Each of our networked services runs as one or more hosts. Turmoil provides counterparts to Tokio’s TcpListener / TcpStream which we use when in DST mode, behind a compile-time feature.

There are also external dependencies to consider, like metadata and object storage. The practice of coding to minimal interfaces allowed us to easily substitute the implementation. We have in-memory emulators that can be accessed over the simulated turmoil network, and also run as hosts in the simulation.

We managed to get simulations running – but as we dug deeper, they were not completely deterministic. We experienced failures in CI that we could not reproduce on our Macs, and in some cases even between runs on the same platform.

What were we failing to control for? Looking at TRACE-level logs, all kinds of differences stood out. Stuff like timestamps in HTTP packets 🤦.

Part of what makes Rust a productive language is the ecosystem of high-quality libraries. But every single one of them represents a possible source of multi-threading, reliance on an RNG, current system time, or external IO. Controlling the world is hard – and hence the space for a startup like Antithesis to make this easy. However, it felt like we were closing in, and we weren’t ready to fold.

It was easy enough to tell no threads were being spun up, all work happened on the main thread. We knew there were no unexpected network calls, all communications were strictly over the simulated network. But for time and randomness – turmoil’s approach was not comprehensive enough to address what arbitrary dependencies may be doing!

We also realized subtleties like Rust’s HashMaps being randomized for DOS prevention. We could perhaps make sure our own app used a seeded RandomState for each hash map, but our dependencies..?

Blending Turmoil and MadSim

This is where we started poking around MadSim – the Magical Deterministic Simulator for distributed systems in Rust – used in RisingWave. What’s the magic?

We liked the overall ergonomics of a turmoil-based DST, but a bit of madness seemed like the missing ingredient – libc symbol overrides to control time and entropy. You can checkout the MadSim-derived crate we just pushed to Github, mad-turmoil.

• The rand module overrides getrandom, getentropy, and (Mac-only) CCRandomGenerateBytes. The new implementations utilize an RNG we statically initialize with set_rng().
• The time module overrides clock_gettime using turmoil’s clock. This is scoped using a SimClocksGuard to avoid issues when the process is tearing down.

Takeaways

So, are we deterministic yet? YES! To avoid repeating the scars of non-determinism, we also added a “meta test” in CI that reruns the same seed, and compares TRACE-level logs. Down to the last bytes on the wire, we have conformity. We can take a failing seed from CI, and easily reproduce it on our Macs.

Was the effort worth it? It goes without saying, distributed data systems are complex and production usage will always bring surprises. But there is a tremendous sense of relief in knowing we can mature our system in simulated time, and catch a lot of problems early.

We have a running doc of all kinds of gnarly issues our DSTs have helped us find, and the tally stands at 17 that were notable. Everything from nuances of our external APIs and internal protocols, to plain old deadlocks. Many of these would make for interesting stories, so watch out for future posts!

POV: You have a shared space with your housemates and want to know if it's empty, but in the least privacy-invasive way.

The Recipe:

• AMG8833 an infrared thermal imaging sensor – 8x8 array of infrared sensors, totaling 64 pixels measuring temperatures ranging from 0°C to 80°C (32°F to 176°F) ±2.5°C (4.5°F) up to 7 meters (23 feet) away at a max frame rate of 10 Hz. Cheap, and best for our use case as it covers a large area with a relatively high accuracy.
• Raspberry Pi 4B or just any low-cost, single-board computer.
• An S2 account as our secret sauce! Follow along here.

For schematics, or to purchase and assemble the electronic components, I recommend following this MakerPortal article, or this Adafruit page.

Once you have everything set up, it should look something like this:

The S2 API will allow us to model our raw sensor output as a Stream, which is just an unbounded sequence of records – in other words, our data in motion! The number of streams you can create is unlimited, all namespaced in a globally unique Basin.

Let's use the S2 CLI to create a basin called monitors, with a stream named amg8833.

$ s2 create-basin monitors
✓ Basin created
$ s2 create-stream s2://monitors/amg8833
✓ Stream created

For building our backend I will use the S2 Python SDK and the adafruit-circuitpython-amg88xx library. Python will help us reduce friction and iterate faster.

Each frame from the sensor reshaped into a 2D 8x8 array can be written as a record at the tail of the stream. Since we want to keep pushing data as we read from the sensor, we will use the streaming AppendSession, which gives us acknowledgments in the same order that records were sent.

async def read_amg8833(sensor) -> AsyncIterable[AppendInput]:
    while True:
        try:
            loop = asyncio.get_running_loop()
            pixels = await loop.run_in_executor(None, lambda: sensor.pixels)
            pixels = np.array(pixels)
            body = {"grid": pixels.tolist()}
            yield AppendInput(records=[Record(body=json.dumps(body).encode("utf-8"))])
        except Exception as e:
            print(f"Sensor error: {str(e)}")
            await asyncio.sleep(5.0)

async def producer(sensor):
    async with S2(auth_token=AUTH_TOKEN) as s2:
        stream = s2["monitors"]["amg8833"]
        async for output in stream.append_session(sensor_data_gen(sensor)):
            print(f"appended {output.end_seq_num - output.start_seq_num} records")
 
 
if __name__ == "__main__":
    sensor = initialize_sensor()
    asyncio.run(producer(sensor))
 

The S2 REST API supports SSE through which it can push real-time updates to a client over a persistent HTTP connection. SSE is perfectly encapsulated by the S2 Typescript SDK generated using Speakeasy. This will power our little Next.js app, where I can view the "live footage" and know if someone is in the shared space 👀.

const basinUrl = 'https://monitors.b.s2.dev/v1alpha';
const s2 = new S2({
  bearerAuth: AUTH_TOKEN,
});
 
interface SensorData {
  occupied: boolean;
  grid: number[][];
}
 
export default function AMG8833() {
  const [occupied, setOccupied] = useState(false);
  const [temperatureGrid, setTemperatureGrid] = useState<number[][]>([]);
  useEffect(() => {
    const fetchStream = async () => {
      try {
        const tail = await s2.stream.checkTail(
          { stream: 'amg8833' },
          { serverURL: basinUrl }
        );
 
        const result = await s2.stream.read(
          {
            stream: 'amg8833',
            startSeqNum: tail.checkTailResponse!!.nextSeqNum,
          },
          {
            serverURL: basinUrl,
            acceptHeaderOverride: ReadAcceptEnum.textEventStream,
          }
        );
 
        const stream = result.readResponse;
        if (!(stream instanceof EventStream)) return;
 
        for await (const event of stream) {
          const outputData = (event as ReadResponseOutput).data;
          if (!outputData) continue;
 
          const batch = outputData as Batch;
          if (!batch.batch?.records) continue;
 
          for (const record of batch.batch.records) {
            try {
              const sensorData = JSON.parse(record.body) as SensorData;
              setOccupied(sensorData.occupied);
              setTemperatureGrid(sensorData.grid);
            } catch (parseError) {
              console.error('Error parsing sensor data:', parseError);
            }
          }
        }
      } catch (error) {
        console.error('Error fetching stream:', error);
      }
    };
    fetchStream();
  }, []);
}

I prompted v0 to help me plot this temperature grid as real pixels converting temperature to hsl, and I was able to just ship it!

I added an extra field to the records (see the small bar at the top of the thermal image) to determine whether someone is really in the frame or not by doing some naïve connected-component labeling to notify me in a text message.

binary_mask = pixels > 28
structure = generate_binary_structure(2, 2)
labeled_array, num_features = label(binary_mask, structure=structure)
 
connections = sum(
    [np.count_nonzero(labeled_array == i) > 4 for i in range(1, num_features + 1)]
)
occupied = bool(connections > 0)
occupied = {"occupied": occupied, "grid": pixels.tolist()}

A sweet spot temperature threshold was 28-29°C, and positioning the sensor at a height around 7-8 feet in a cooler area tends to work quite accurately!

The obvious next step that comes to my mind is to train a machine learning model, using supervised learning, for determining the number of people in the frame. Here, S2 stands out since we can not only consume data at the time it is published by the device, but have it available as a durable stream of records to train my model.

So how much does this project cost us in a month? S2 is free for now, but we can refer to the intended pricing and figure out a monthly estimate.

Each record is ~470 bytes and at 10 Hz over the course of a month this would be ~12 GiB of data.

• Initial operations to CreateBasin and CreateStream are fractions-of-a-cent.
• AppendSession and ReadSession have a per-minute cost of $0.0000001, adding up to $0.00864.
• If we also keep record retention at a month, at $0.04/GiB-month that will cost us $0.48.
• Data transfer into S2 depends on the storage class. We'll go for the faster Express at $0.06 per GiB, which works out to $0.72.
• Data transfer out from S2 depends on where we are accessing it from, with the most cost-efficient being same cloud region. We'll be accessing the stream from home so at $0.08 per GiB for internet egress – assuming we kept the footage up at all times! – this works out to $0.96.

All together, around $2 a month. Not having to worry about high costs, and pushing that side project with a simple API for your data pipeline? Win!

S2 is currently in preview. Come join us on Discord and get hacking!

Overview

S2 is fundamentally a building block. We believe it is an excellent fit for a variety of use cases that require fast, scalable, and cost-effective access to sequenced data: change data capture, feature transformation, observability events, and click streams, to name just a few. A lot of the time, you really just want an object-storage-like interface for durable streams of records.

We are continuing to iterate on the core capabilities of S2, such as supporting record timestamps in addition to sequence numbers (on its way), and key-based compaction (roadmap). We also have our sights on offering higher-level layers for compatibility with protocols like Kafka that offer features like consumer groups and queue semantics, via software that runs on top of S2.

As we embark on building these additional layers, we are making the core S2 service available for anyone to use as a foundation for their own data-intensive systems. This post explores one concrete example of what that can look like, using a beloved "hello world" distributed data system: a replicated key-value store.

The S2 API is deliberately simple; however, it is by no means limited to basic use cases. S2 is designed to provide a first-class implementation of the shared log abstraction, a powerful tool for distributed systems engineers that has gained traction in recent years.

This post is structured in three parts:

1. An intro to the concept of shared logs.
2. Exploring how S2's stream API can be used to design a multi-primary replicated KV store with strong consistency properties. (This part won't be language specific at all.)
3. Digging into a sample implementation of the system which uses S2's Rust SDK.

Part 1: The shared log abstraction

Write-ahead logs (or "WAL"s) have been a cornerstone of database durability for decades. Traditionally, WALs are stored on non-volatile storage that is co-located with compute, and are used to provide crash recovery for database nodes. As soon as a mutation has been persisted in the WAL, it can be considered durable — its effects can be deterministically recovered after a crash — and the database is free to asynchronously complete more expensive actions, like rewriting a page in a B-tree.

In the distributed context, maintaining copies of a database for scalability, availability, and performance is notoriously tricky but well-researched. There are many types of consistency guarantees that a system may target, but an underlying challenge in a replicated setup is making nodes agree about the content of the database at a given moment. Strong consistency models promise that even a highly distributed database appears as if we are communicating with one that were only running on a single machine.

This is where consensus protocols like Paxos or Raft enter the picture to power state machine replication (or "SMR"). Generally, the database implementor has to take on a lot of the heavy-lifting involved in SMR, which has a whole spectrum of design possibilities.

Suppose that instead of residing on a block storage device attached to a single machine, the WAL were distributed — able to be written to and read from multiple nodes on different hosts concurrently, and independently durable across node failures.

If we had something like this, we could use the WAL itself, "disaggregated" from local compute, as the mechanism for distributing state across our replicas. Our core database code could then be much more focused on indexing and querying semantics.

This is the concept of the shared log, which Mahesh Balakrishnan explores in depth in his 2024 paper, Taming Consensus in the Wild:

As an analogy, think of the SMR platform as a filesystem and the shared log as a block device. In much the same way that a block device makes it easier to build a filesystem without worrying about hardware internals (e.g., HDD vs. SSD), a shared log helps us write an SMR layer without reasoning about the internals of the consensus protocol. The SMR layer is then free to focus on the complexity of materialization, snapshot management, query scalability, single-node failure atomicity, etc., in much the same way that a filesystem can focus on file-grain multiplexing, directories, crash consistency, etc.

This idea is at the heart of the Delos system at Meta, used for powering control plane services.

More recently, Amazon has shared how they built MemoryDB, a strongly-consistent replicated Redis, using this same technique:

MemoryDB offloads durability concerns to a separate low-latency, durable transaction log service, allowing us to scale performance, availability, and durability independently from the in-memory execution engine.

Access to a system that provides sufficient durability, performance, and a suitable API for using as a shared log service is, alas, pretty hard to come by unless you happen to find yourself at Meta or Amazon.

We want S2 to be the public implementation of exactly this type of service, which has to date been a super-power mostly just for super-scalers.

Part 2: Designing a replicated KV store

Let's see how a key-value database can be implemented using S2 as a shared log. Our example will not have nearly as sophisticated of an API as Redis — to start with, we'll just support get/put/delete.

Goals

Some things we would like to see in our KV store:

• HTTP interface over a get/put/delete REST API: it should be easy to interact with the system using curl.
• Durability: specifically, in multiple availability zones of a cloud region. Writes modifying the state of the store cannot be lost once acknowledged.
• Horizontal scalability: we aim to distribute our requests across an arbitrary number of replicas, for improved performance and availability.
• Multi-primary: each of our nodes should be capable of servicing both reads and writes. S2 does support concurrency control mechanisms¹ which simplify the robust implementation of leader/follower schemes, but we will dive into that in a future post.
• Strongly consistent, linearizable reads and writes: we want every read or write to reflect all prior writes that have completed before it (and also in the order in which those writes occurred). For clients that don't require strong consistency, we can provide better performance with an opt-in "eventual consistency" mode.

Non-goals

Some aspects we will consider acceptable limitations, but all of which make for good "exercises left for the reader" 😄:

• Returning prior values of a key from put and delete.
• Supporting values larger than 1 MiB, which is the limit on an S2 record's size.
• Snapshotting to allow for restoring state via a combination of a snapshot and, only for recent writes, the log.
• Sharding or any sort of data partitioning.

Connecting to S2 concepts

To build this system, we need a shared log, stored durably, where each entry in that log provides some payload representing a change to our KV store. These log entries will need to be totally ordered, so that there is no ambiguity about the sequence of events, and we can deterministically reconstruct state from the log.

Maybe you can already see where this is going... this is exactly what we get with an S2 stream! We are going to use "log entry" and "record" interchangeably for the rest of this post.

AppendSession for log writes

S2 streams can be written to via unary or streaming RPCs. Since our KV store is always going to need access to its log, each node can maintain a streaming AppendSession for its lifetime.

Only two of the routes in our KV store's API actually "write": put which supplies a new value, and delete which removes the existing value. Any time a node receives a request for one of these, it will also need to prepare a log entry corresponding to those actions, and send that as an append over the session.

The actual format of this entry can be virtually any lossless representation of the key it applies to and the value being supplied (in the case of put).

Once we have encoded our entry and sent it to S2, we have to wait to receive a corresponding acknowledgement that it has become durably sequenced within the log before we can in turn send an acknowledgment to the KV store requestor.

The AppendSession RPC is full-duplex. We can send records to it and concurrently also receive acknowledgements (or an error) from it. The acknowledgements will always arrive in the same order as the inputs.

Sequence diagram for a put to the KV store

S2 append latency is in the critical path of put and delete calls on our store, so we can never return faster than the "round-trip time" to append a record and receive its acknowledgment. Accordingly, these write-triggering calls on our KV store will typically be in the low tens of milliseconds (50 ms at p99) — assuming our KV store is also running in the same cloud region as S2,² and that we elect to use S2's Express storage class. (We do plan to offer a faster storage class in the future for single-digit millisecond latencies.)

ReadSession and CheckTail for log reads

We have not covered the actual "materialized state" of our KV store yet.

If we were not going for horizontal scalability, and could guarantee that there would only ever be one node running, then serving reads would simply be a matter of consulting with the node's internal state stored in an in-memory dictionary of some sort. Writes could update that state directly, after receiving an acknowledgment from S2. Get requests would also be incredibly fast, as there would be no external service to communicate with.³

In a multi-primary setup, however, any of the KV store replicas might be concurrently writing to the shared log, so updating the state directly in response to an append wouldn't work. We could be missing any writes that occurred on other nodes, and replicas would quickly get out of sync.

Instead, we can have each node keep a tailing ReadSession open for its lifetime. This is a half-duplex RPC that will let us read all entries on the shared log, in the same order in which they were appended, with minimal latency overhead. If all nodes then apply those entries to their local copies of the materialized state, they will end up as identical replicas of our KV store, and any of them can service get requests by consulting their local copy.

Almost, anyway. ReadSessions do tail updates to the log in real time, but to provide strong consistency we also have to ensure that the node which is servicing the get is fully caught up with the log.

Suppose a get request comes in to one of our nodes just after a put completed (either on the same node or a different one), but our read session hasn't quite caught up with it yet. If we didn't do any extra coordination, and simply served the get using the current materialized state, we could return a stale value, and our system wouldn't actually be linearizable.

This is where CheckTail comes in. This operation gives us the current sequence number representing the tail of the stream (i.e., the sequence number that would be assigned to the next record appended). When a get arrives, our KV store just needs to check the current tail of its log, and compare it to the last sequence number which has been reflected in the local state. If the applied local state lags from the tail, we simply need to wait until we've caught up to it over the ReadSession.

Sequence diagram for a strongly consistent get to the KV store

Similar to put and delete, which are bounded below by the time it takes to append and receive an S2 acknowlegement, any get on our KV store is bounded by the time it takes for check_tail to return. Unlike appends, this latency is not a function of the stream's storage class, and should be quite fast. Nevertheless, clients that can tolerate eventual consistency reads (potentially stale values) may elect to forego the check_tail operation for better latency.

Wrapping it up

If we do all of this, we will end up with a KV store that can satisfy our lofty goals. We only have to deal with three fundamental operations on a stream: append, read, and check_tail. This is the power of decoupled storage and compute!

Part 3: Implementing the KV store using the Rust SDK

Now we'll walk through an actual implementation of this system that is built around the S2 Rust SDK.

If Rust is not your jam, SDKs for other languages are on their way — starting with Go, Python, and Java — as well as a REST API.

Setup

Data formats

Keys in our system can simply be Strings. For values, we'd like to support several common datatypes, for convenience.

A reasonable approach to this is to create a new Value enum which wraps other types, e.g.:

enum Value {
    Str(String),
    UInt(u64),
    List(Vec<Value>)
    // ...
}

Our materialized state can be stored in just about any map datastructure. In the demo, we use a BTreeMap<String, Value>.

Similarly, log entries will need to represent Put and Delete actions:

#[derive(Clone, Debug, Deserialize, Serialize)]
enum LogEntry {
    Put { key: String, value: Value },
    Delete { key: String },
}

S2 streams expect to receive data as batches of AppendRecords. These consist of a binary body, as well an optional set of headers. Since we're working with bytes for the body, we have complete control over how to encode our log entries as records. The only real constraint is that, since no record can exceed 1MiB on S2, our Values must also be small enough to be serialized within that budget.

For convenience and debug-ability, we can simply encode our log entries as JSON bytes using the serde library, so that we get a nice human-readable represetation when reading the log via the CLI. In a production system, we would probably want to adopt an efficient binary format.

s2 read "s2://${MY_BASIN}/${MY_STREAM}"

... might for example produce something like:

{"Put":{"key":"hello","value":{"Str":"world"}}}
{"Put":{"key":"s2","value":{"Set":[{"Str":"is really cool"},{"UInt":1337}]}}}
{"Delete":{"key":"hello"}}
{"Put":{"key":"map-sample","value":{"Map":{"k1":{"Str":"hello"}}}}}

HTTP server

We can use the axum library for our actual HTTP server, and set up routes corresponding to our KV store's get, put, and delete API by using the RESTful GET, PUT, and DELETE HTTP methods respectively. The README in the demo repo has some sample curl invocations for testing these routes out with actual values.

For additional debug context, we'll also have each route return the shared log range reflected by the corresponding action. Any consumer of this specified portion of the log would see the value that was returned in the response, or in the case of put, the value which was provided in the request. The range values are simply S2 stream sequence numbers.

For example, the acknowledgment from this put:

$ curl \
    --silent -H 'Content-Type: application/json' \
    -X PUT \
    -d '{"key": "hello", "value": {"Str": "world"}}' \
    "localhost:4001/api"

... might look like this:

{ "Ok": { "end": 272 } }

A subsequent get for that key might return this:

{ "Ok": [{ "end": 272 }, { "Str": "world" }] }

... which would signify that no additional put or delete occurred anywhere in the KV store in the meantime, as the reflected log region is still (..272).

Let's dive into how the actual KV store functions will work.

Main event loop

Earlier, we touched on the main components of our system, and generally how the KV store API will interact with S2's stream API.

There are many ways to structure an actual implementation of this system. In the demo, the main "engine" of our KV store is modelled as an event loop, which is a common pattern for I/O bound workloads.

The idea is that we can spawn a Tokio task which will be responsible for executing this loop, and therefore responding to our various input and output streams, and managing the internal materialized state. This task will run for the lifetime of our node. By giving that task ownership of the materialized state, as well as of any I/O streams, we can avoid use of locks entirely.

Internally, our get, put, and delete functions will all rely on this task to accomplish what they need to.

We can model an event loop task like this as a regular async Rust function, which will get spawned during startup of our node. This is the orchestrate function in the demo.

This function mostly consists of one large loop, the body of which will await a few different async functions. We can use the tokio::select! macro, which is a handy way to await multiple futures simultaneously. That way, we can handle whichever future returns Ready first, and only run the code in the branch that was associated with it. The futures which we don't end up handling will get dropped, but we can re-await futures from the same underlying streams and channels on the next iteration of the orchestrate loop.

Here's what the skeleton of our event loop function will look like:

async fn orchestrate(
    // ...
) -> Result<(), KVError> {
    // ...
    loop {
        tokio::select! {
            Some(cmd) = command_rx.recv() => {
                // ... commands about new `put`/`delete`/`get` requests
            }
            Some(ack) = append_acknowledgments.next() => {
                // ... S2 record batch acknowledgments
            }
            Some(record) = tailing_reader.next() => {
                // ... S2 log entry records
            }
            else => { break; }
        }
    }
    // ...
}

The only catch with using an event loop is that, since we only have a single task responsible for all I/O, we have to be careful to make sure that we correctly offload work. We want the loop to be spending most of its time in the select! await, where the task is able to respond to whichever type of message appears next. We don't want the actual event loop task to be stuck awaiting a single function call within one of the branches — it will need to stay open to quickly deal with whatever occurs next, dispatch it, and move on.

For instance, a put request will take some time for our KV store to service; each put will require a write to the S2 log, and will need to then wait to see if that write was acknowledged by S2. We can't hold up our event loop while we're waiting for that acknowledgment, or it would block other concurrent callers to our system. A typical way to deal with this is by using some sort of async "callback" mechanism (more on that in a bit).

Defining command messages

What are these actual flows of information that orchestrate will be responsible for, and what actions it will take in response to each?

The first branch in our select! block handles commands. Since orchestrate is a task, not a struct, we can't have it perform actions by calling a function directly on it. Instead, we'll communicate with it via message passing. That way, commands are simply modelled as another I/O stream to react to in the main event loop.

The command_rx in the code snippet above is simply the receiving end of an unbounded channel. This is a "multi-producer / single-consumer", or MPSC, channel — so the receiver held by orchestrate is guaranteed to be the only one. We can have an unlimited amount of sender handles, on the other hand, which comes in handy. In other words, callers can easily clone and send new commands into our channel, and all of the messages will flow to the main loop.

What actually needs to be communicated to our task? Really, any action we want it to perform. In practice, this will be servicing requests that come from our KV store users directly — so put,delete, and get. The commands themselves can be an enum, allowing us to pattern match on the different variants within the select! branch.

We won't quite just have different commands for put/delete/get respectively. Recall that both put and delete can be handled almost identically. We can consolidate both of them as a single WriteLog command, where the sender constructs a log representing either a Put or Delete action.

On the other hand, get commands actually end up being processed in two different ways. If the caller wants strong, linearizable consistency, we need to make sure that when we execute the get that the local materialized state has caught up with the tail of the S2 log. If the caller is fine with eventual consistency, we can execute that get immediately, regardless of what the tail is.

Our commands end up looking like this:

type SequenceNumber = u64;
type SequencedValue = (RangeTo<SequenceNumber>, Option<Value>);
type WriteSender = oneshot::Sender<Result<RangeTo<SequenceNumber>, KVError>>;
type ReadSender = oneshot::Sender<Result<SequencedValue, KVError>>;
 
enum OrchestratorCommand {
    WriteLog {
        log: LogEntry,
        response_tx: WriteSender,
    },
    ReadStrongConsistency {
        key: String,
        reflect_applied_state: RangeTo<SequenceNumber>,
        response_tx: ReadSender,
    },
    ReadEventualConsistency {
        key: String,
        response_tx: ReadSender,
    },
}

You may be wondering what these response_tx fields are. Those are how we can call back to the original sender of the command (i.e., the actual put, get, and delete methods on the HTTP server).

Since commands are being communicated to the orchestrate task via an MPSC channel, and not a function call, there is no obvious way for the senders of those commands to be notified when the requested asynchronous work is completed.

A common pattern in these situations is to send a oneshot channel. These are very similar to MPSC channels, but used for receiving a single value — so particularly useful as an async notification mechanism. In our case, the commands contain a sender for the oneshot, which orchestrate will either use immediately or hold on to, and the receiver end is then retained by the original issuer of the command.

For instance, the put command (what actually ends up getting called by the HTTP server's PUT route) looks like this:

/// Put a new key/value to the store.
async fn put(
    &self,
    key: String,
    value: Value
) -> Result<RangeTo<SequenceNumber>, KVError> {
    // Create the oneshot for receiving a response.
    let (response_tx, response_rx) = oneshot::channel();
 
    // Send a `WriteLog` command corresponding to
    // the `Put` key and value, and providing the
    // sender end of the oneshot.
    self.orchestrator_cmd_tx
        .send(OrchestratorCommand::WriteLog {
            log: LogEntry::Put { key, value },
            response_tx,
        })
        .map_err(|_| KVError::OrchestratorTaskFailure)?;
 
    // Await a response on the receiver.
    response_rx
        .await
        .map_err(|_| KVError::OrchestratorTaskFailure)?
}

Executing commands

We've seen how commands flow into our orchestrate task, and how responses can be communicated back using oneshots.

What does orchestrate actually do in response to the different commands?

Write commands

For WriteLog commands (from put or delete requests), it will simply package the log into a record batch, and emit it to the active AppendSession via a channel that supplies (using tokio's ReceiverStream) an async stream of AppendInput values.

Some(cmd) = command_rx.recv() => {
    match cmd {
        OrchestratorCommand::WriteLog { log, response_tx } => {
            write_queue.push_back(response_tx);
            append_tx.send(AppendInput {
                records: AppendRecordBatch::try_from_iter(
                    [AppendRecord::new(bytes::Bytes::from(log))?]
                ).map_err(|_| KVError::Weird("unable to construct batch"))?,
                ..Default::default()
            }).map_err(|_| KVError::Weird("s2 append_session rx dropped"))?;
        }
    }
    // ...
}

Notice that we don't just send the record — we also push the oneshot sender received as part of the command into a write_queue. We can't resolve the originating request until we've received an acknowledgment from S2 that the corresponding log is durable, so we need to hold on to the oneshot sender for now.

Eventually consistent read commands

Commands for performing eventually consistent reads are pretty easy! Since the caller has elected for non-linearizable reads — in other words, they can tolerate staleness, and don't care if the read doesn't necessarily reflect all prior writes on the shared log — we can simply check the materialized state and return the result immediately:

Some(cmd) = command_rx.recv() => {
    match cmd {
        // ...
        OrchestratorCommand::ReadEventualConsistency {
            key,
            response_tx
        } => {
            // Use the oneshot to communicate the read result immediately.
            _ = response_tx
                .send(Ok((local_state.applied_state, local_state.storage.get(&key).cloned())))
                .inspect_err(|_| debug!("read rx dropped"));
        },
    }
}

Strongly consistent read commands

Strong reads are only slightly more involved. These commands will specify a log prefix that must have been applied to the materialized state before we can return a value. (If you're wondering where that prefix is obtained, we'll get to that later, when we discuss where check_tail gets called.)

If the materialized state already happens to have caught up to that point, we can immediately return the value. If not, we need to use some sort of queue — similar to what we do with writes — and hold on to the oneshot (as well as the key), until we catch up to the desired state.

Some(cmd) = command_rx.recv() => {
    match cmd {
        // ...
        OrchestratorCommand::ReadStrongConsistency {
            key,
            reflect_applied_state,
            response_tx
        } => {
            if reflect_applied_state.end <= local_state.applied_state.end {
                // Applied state is already caught up with the tail.
                _ = response_tx
                    .send(Ok((local_state.applied_state, local_state.storage.get(&key).cloned())))
                    .inspect_err(|_| debug!("read rx dropped"));
            } else {
                // Not yet caught up. Defer until we do.
                pending_responses.submit(PendingResponse::new(
                    reflect_applied_state, ResponseContext::Read{ key, response_tx }
                ))
            }
        },
        // ...
    }
}
 

Handling S2 I/O

At this point, we've seen how all of our different OrchestratorCommands are reacted to. Some reads get handled immediately, others are pending on some future log entries being applied, and are stashed until then. Writes, similarly, are all stashed into a queue where they will await acknowledgments from S2.

In both cases, we are awaiting some additional information from our durability layer, S2. Let's see how that works.

S2 append acknowledgements

We send batches of records (individual log entries) to S2 via an MPSC channel (the append_tx sender, as discussed earlier), but also receive acknowledgments back from S2 as an async stream of AppendOutput messages.

Within an AppendSession, acknowledgments are guaranteed to be received in the same order that we sent appends, which is why a FIFO queue is a good way to keep track of our pending, or "inflight", log writes.

Whenever a new acknowledgement arrives, we simply pop_front from our queue and obtain the oneshot for communicating back to the original put or delete call, and send a message indicating that the write has succeeded.

Some(ack) = append_acknowledgments.next() => {
    let response_tx = write_queue.pop_front().expect("queue entry");
    _ = response_tx
        .send(Ok(..ack?.end_seq_num))
        .inspect_err(|_| debug!("write ack rx dropped"));
}

S2 log entries

Our orchestrate task is also always listening for log entries via a ReadSession. These could correspond to writes that were proposed earlier by the same node — or they might have been written by one of the other nodes; it's a shared log after all!

Whenever a log entry is received, orchestrate simply has to apply it to the materialized state (the in-memory version of the map), and update the tracker of the currently applied contiguous prefix of log entries. Whenever it updates its state, it can also look for any pending strong get requests, and see if they can now be resolved (and the requested value can be returned from the now up-to-date state).

Some(record) = tailing_reader.next() => {
    let record = record?;
 
    // Deserialize the json log.
    let log_entry = serde_json::from_slice(record.body.as_ref())
        .map_err(|e| KVError::JsonError(e.to_string()))?;
 
    // Insert or delete the key from the local storage map.
    match log_entry {
        LogEntry::Put { key, value } => local_state.storage.insert(key, value),
        LogEntry::Delete { key } => local_state.storage.remove(&key),
    };
 
    // Update the applied state prefix. Our materialized state reflects
    // this contiguous range of the log.
    local_state.applied_state = ..record.seq_num + 1;
 
    // Find any pending `get` requests that may have been waiting for
    // the `applied_state` to catch up.
    for PendingResponse {
        end_seq_num: _,
        response_context
    } in pending_responses.drain_applied_responses(local_state.applied_state) {
        match response_context {
            ResponseContext::Read{ key, response_tx } => {
                _ = response_tx.send(Ok((local_state.applied_state, local_state.storage.get(&key).cloned())))
                    .inspect_err(|_| debug!("read rx dropped"));
            }
        }
    }
}

That's about it for orchestrate!

CheckTail operations for strong reads

Earlier, when we were looking at how OrchestratorCommand::ReadStrongConsistency commands get processed, we saw that all strong read requests need to specify a reflect_applied_state parameter. This value indicates the value of the S2 log's tail at the time at which the get was processed, and we can only achieve linearizability if we ensure that the materialized state has reflected all logs up to that position when we retrieve and return a value.

It is expected, therefore, that the get handler in our KV store call check_tail itself, and provide the value it received in the command to orchestrate. In theory, orchestrate could also be responsible for performing the check_tail op — it's just one additional I/O task after all.

Bus-stand optimization

In practice, it's a bit neater to have a separate, dedicated event loop task for check_tail purposes. This is mainly to take advantage of what Mahesh Balakrishnan refers to as the "bus-stand" optimization (from §3.2 in this paper mentioned earlier).

Since every single strong read serviced by our KV store needs to obtain the current tail, by default that means one check_tail operation per read request. If we're willing to slightly delay reads, we can instead group these tail requests into batches, where — similar to catching a bus at a bus stand — passengers wait until the next "departure" to the underlying check_tail call, and a single response can satisfy that input for several read requests at once. This allows us to trade off additional latency for strong reads with a ceiling on the maximum number of check_tail operations performed per second.

Similar to orchestrate, we can communicate with a dedicated check_tail task over an MPSC command channel, and receive responses via a oneshot. The bus stand optimized version of check_tail looks like this:

/// Get the current stream's tail.
///
/// Wait up to `max_wait` before the actual `check_tail` invocation occurs, allowing
/// other callers to also be served by the same underlying S2 tail op.
async fn bus_stand_check_tail(
    &self,
    max_wait: Duration,
) -> Result<RangeTo<SequenceNumber>, KVError> {
    let (response_tx, response_rx) = oneshot::channel();
 
    self.bus_tx
        .send(BusRider {
            max_wait,
            response_tx,
        })
        .map_err(|_| KVError::BusStandTaskFailure)?;
 
    response_rx
        .await
        .map_err(|_| KVError::BusStandTaskFailure)?
}

(The actual implementation of the bus stand task, which performs the batching, can be seen here in the demo).

Our new bus_stand_check_tail function can be called by get and used to obtain the tail before sending a read command to the orchestrator:

/// Get the value of a key.
async fn get(
    &self,
    read_consistency: ReadConsistency,
    key: String,
) -> Result<(RangeTo<SequenceNumber>, Option<Value>), KVError> {
    // Oneshot for receiving the value.
    let (response_tx, response_rx) = oneshot::channel();
 
    let cmd = match read_consistency {
        ReadConsistency::Eventual => {
            OrchestratorCommand::ReadEventualConsistency { key, response_tx }
        }
        ReadConsistency::Strong => OrchestratorCommand::ReadStrongConsistency {
            key,
            reflect_applied_state: self.bus_stand_check_tail(BUS_RIDER_MAX_WAIT).await?,
            response_tx,
        },
    };
 
    self.orchestrator_cmd_tx
        .send(cmd)
        .map_err(|_| KVError::OrchestratorTaskFailure)?;
 
    response_rx
        .await
        .map_err(|_| KVError::OrchestratorTaskFailure)?
}

Conclusion

We covered a lot in this post, and managed to build a simple multi-primary, strongly consistent database! I hope it gave you a glimpse into some of the ways in which S2 can be used at the foundation of real distributed data systems. We can't wait to see what people end up building.

The actual implementation of the system discussed here is only ~600 lines, and worth checking out if you made it this far!

Elevate the beating heart of data systems.

Our team has worked a lot on reliable real-time ingest, where The Log is foundational. We loved the serverless experience of object storage, but it simply did not exist for streaming data.

We believe the humble log – the stream – deserves to be a cloud storage primitive.

With S2, we are previewing just that: S2 is the Stream Store, our interpretation of streaming for the cloud era.

What if streams had the primacy of objects?

Object storage has been nothing short of revolutionary. S3 broke ground in 2006 with simple storage operations on named objects – and 18 years later, S3 Express One Zone even allows appends. But ultimately, object storage is all about blobs and byte ranges. It is best for data at rest. Our vision of stream storage is predicated on the idea that the demands of data in motion need a fresh perspective.

With S2, you are elevated to the natural granularity of records. Writes to an S2 stream are appended at the tail, and even if multiple writers are acting at a time, S2 will durably sequence all records. S2 takes care of serving your reads efficiently, whether you need to start streaming from seconds ago or years. Streams can also be tailed in real-time, which is not possible with a blob in S3.

Just like buckets are a namespace for objects, basins play that role for streams in S2. Basins and streams lean into the scale of the cloud – there is no limit on how many you can have, or how long data can be retained.

Want to model streams per user? Do it, this isn't Kafka. There are no cluster limitations to wrangle, and no infrastructure to tune.

$ s2 ls s2://copilot-rag-ingest
user-foo/cool-project
user-foo/another-project
user-bar/fork-of-cool-project
# ... ∞

This stream interface brought us closer to our vision, but we also wanted to liberate the superpower of offloading durability which databases like MemoryDB and Neon leverage. Decoupling compute and storage is safest when the storage service cooperates.

So we added a verb to check the tail of the stream with strong consistency, and support for concurrency control when writing. You can be a pessimist wielding a fencing token or optimistically supply the sequence number you expect assigned – no judgment which side of the fence you find yourself on.

Serverless – at what cost?

S2 is architected around the infinite scale and unrelenting durability of object storage. That does not necessitate a slow or expensive offering – quite the opposite! We bridge the abstraction gap with a multi-tenant service so that you can have a truly serverless API for streaming data.

Durability is not negotiable for us in the undeniable cloud storage triad. We allow users to navigate their latency vs cost tradeoff on a per stream basis, with storage classes. We are starting out with two:

1. Standard, backed by S3 Standard in AWS. S3 Standard has a counterpart in all public cloud providers, so we will be able to ship it in all cloud regions as we grow.

2. Express, backed by a quorum of three S3 Express One Zone buckets in AWS. Azure has had a regional counterpart for years, and it is in the cards at GCP, so we are optimistic about wider availability.

Our Standard storage class provides end-to-end p99 latencies of under 500 milliseconds. With Express you can expect under 50 milliseconds – in the realm of disk-based cloud streaming systems! S2 on the other hand is completely diskless, and all writes will be safe in S3 with regional durability before being acknowledged.

These latencies are supported at throughputs of hundreds of megabytes per second, per stream. The overhead of reading recently written data is negligible in S2 because of in-memory caching. Lagging readers can be particularly thirsty for throughput, and S2 serves them directly from object storage without a cap. We are initially throttling writes at 125 MiBps and reads against recent writes at 500 MiBps, per stream.

The service is free during our preview period to optimize for feedback. We want to be transparent about our intended pricing, and you will find that S2 comes in meaningfully cheaper than the norms of cloud streaming systems. This will be particularly stark in comparison to "serverless" offerings, which attract tiny ceilings on the number of streams and the throughputs you can push through them – at a very high premium.

There are no fixed costs in S2 like instances or cluster units. When we say serverless, we mean it!

What's next for S2

S2 stands on a foundation of battle-tested cloud infrastructure, and our own Rust codebase gets put through the wringer with deterministic simulation testing. That said, the system is young, and there will be kinks. We are working hard to mature towards general availability and an SLA you can count on in production.

We are now shipping a gRPC API, Rust SDK, and a shiny CLI – and we are going to get cracking on a REST API. Do tell us which language SDKs you would be most interested in.

To give you a bigger picture sense of our direction:

• Kafka protocol compatibility. This will be an open source layer, and we will integrate certain features like key-based compaction directly in S2.

• Multi-region basins. Once we expand into more cloud regions, we see a path towards basins that can span regions and even clouds, for the highest standard of availability.

• Under 5 millisecond latencies. We are just getting started with the architectural flexibility of storage classes, and another 10x improvement over Express is achievable.

Can you replace Kafka or Kinesis with S2 today? If you find yourself reaching for their “low-level” APIs, S2 is likely to be a fit, and even address your requirements more directly.

If you expect a lot more from the cloud than current norms for streaming data – like not being limited on how many streams you can have, 10-100x higher ordered throughput, and concurrency control – S2 is the missing piece.

We are beyond excited for all the innovative data systems S2 enables, and invite you to build with us!