We are excited to announce the release of the MongoDB Vercel AI SDK integration, which provides AI applications and agents built with Vercel’s SDK with a persistent memory store in MongoDB Atlas. It’s now open source and generally available on npm.

This integration adds MongoDB Atlas as the memory store in the existing Vercel AI SDK code. Once installed, the package connects Vercel AI SDK applications to MongoDB Atlas and automatically stores the application’s memories in documents. This allows the application to retain memories about conversations, user facts, and past events that persist across sessions, users, and restarts.

What is the Vercel AI SDK?

The Vercel AI SDK is a TypeScript toolkit for building AI applications. It handles calling models, streaming responses, using tools, and running agents that work through multi-step tasks. The SDK doesn’t include a database to persist memories between runs. Without memory, an AI application starts every conversation with no knowledge of the last one. It doesn’t know who the user is, what they asked yesterday, or what it learned along the way. It doesn’t have any context.

Today, developers have to build their own tooling to store memories in a layer. That usually means designing a database schema, writing jobs to clear out old data, and setting up vector search so the application can find relevant past information by semantic meaning instead of just keywords. It’s a lot of work to build an AI application that can remember context.

What types of memory does an AI application need?

An AI application needs five kinds of memory, each kept for a different length of time and retrieved differently. This integration automatically categorizes each type of memory and stores it in MongoDB Atlas; here are short definitions of the five memory types:

• Session: The conversation an application or agent has in each session. This helps the application follow what the user just said.
• Semantic: Facts about people, entities, and preferences, so users don’t have to repeat themselves.
• Procedural: Workflows and instructions, so the application carries out tasks the way you’ve defined them.
• Episodic: Key events and outcomes, so the application can recall what happened before. Kept for one year of inactivity by default.
• Scratchpad: Temporary notes for work in progress. They expire after an hour and can be promoted to episodic memory if they turn out to matter.

Semantic and procedural memories are versioned, so history is preserved when a fact changes. The semantic, procedural, and episodic tiers use MongoDB Vector Search, which finds stored memories by semantic meaning rather than by exact keywords.

This integration automatically creates the vector indexes and detects embedding dimensions at startup. Developers can use any AI SDK embedding model to create vector indexes, including Voyage AI, OpenAI, Cohere, or Google.

How should memory age?

Each category of memory has its own retention policy, so short-lived notes expire quickly, and long-term facts are stored for longer. The MongoDB Vercel AI SDK integration ships with defaults for every category, which developers can customize if they desire.

Effective memory retention requires a balance. Keeping memories indefinitely can decrease retrieval performance and increase costs, while deleting them too early can cause applications to forget important context. This integration includes features to help developers easily set the right balance:

• Retention modes: Choose none (never expire), ttl (expire after a set time), ttl+importance (expire after a set time unless the memory is marked important), or dynamic, where you define your own forgetting time period.
• Model-driven forgetting: With memory_forget, the model can delete a specific memory when a user says “forget that.” Removal usually happens within about 60 seconds.

Because the memory is stored in MongoDB Atlas, application operational data, its memory, and the vector search that retrieves it run on one platform. Developers can secure, monitor, and scale a single database without bolting on a separate vector store or memory service alongside it.

What changes when you run the MongoDB Vercel AI SDK Integration in production?

This production release gives developers greater control over their data and flexibility in how they use the integration:

• Reliable session history: By default, the model decides when to save and read conversation turns, so it can occasionally skip one. For production, you can use the AI SDK hooks so that the application’s own code loads and saves the transcript at every turn. Every user, assistant, and tool message is saved exactly once.
• Separate data per customer: Developers can add a filter field, such as tenant_id, to the application’s vector indexes to limit each search to a single customer’s memories.
• Application names and layout: Developers can rename collections and indexes to match the existing application schema or turn off indexing for unused categories.
• Open source: The package is licensed under Apache 2.0 and works with any AI SDK model.

Get started today

Developers can add memory to an existing Vercel AI SDK project in three steps. This integration works with AI SDK v6 and v7 and requires a MongoDB Atlas cluster.

1. Install the package and its peer dependencies: npm install @mongodb-developer/vercel-ai-memory ai mongodd
2. Create a memory instance with an Atlas connection string and an embedding model.
3. Pass the memory tools to the application, scoped to a user and session.

const mongodbMemory = createMongoDBMemory({
  uri: process.env.MONGODB_URI!,
  embedder: openai.embedding('text-embedding-3-small'),
})

const agent = new ToolLoopAgent({
  model: openai('gpt-4.1'),
  tools: mongodbMemory({ userId: 'alice', sessionId: 'sess-001' }),
})

From there, the model decides when to save and when to search for memories.

Ready to add memory to your application?

Start with the Vercel AI SDK memory documentation. For more details, see the following resources:

• @mongodb-developer/vercel-ai-memory on npm: the package
• GitHub repository: source code and examples

Introducing the MongoDB Vercel AI SDK Integration: Persistent Memory for AI Applications was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’re excited to announce the MongoDB Google ADK Integration, which provides developers with the building blocks they need to create durable memory for agent state.

As agents move from prototypes to real-world applications, they need secure, dependable access to data and the ability to retain context over a long time. However, developers still need to build their own tooling for a data store that provides durable agent memory for agent state.

This integration gives developers a cohesive, ADK-native experience for using MongoDB as the data platform for agents.

What is the Google ADK?

The Google Agent Development Kit (ADK) is an open source agent development framework that lets you build, debug, and deploy reliable AI agents at enterprise scale. It is available in Python, TypeScript, Go, Java, and Kotlin.

Google ADK offers developers an extensive set of tools for creating, deploying, assessing, and scaling agents, enabling developers to build anything from personal AI assistants to mission-critical business workflows.

How does the MongoDB Google ADK Integration help?

Developers can prototype an agent that searches a product catalog or knowledge base, then evolve it into a production experience that preserves user context across conversations and recalls relevant prior interactions. With MongoDB as the data layer, developers can keep the data their agents search over, the conversation state that needs to survive a restart or a scaled-out instance, and the memories agents build over time, all in the same place as the rest of their application data. That means no separate database to stand up for retrieval, no pipeline syncing app data into a different memory index, and no added latency pulling context from another system before the agent can respond.

ADK supplies the agent framework and tool interface; this integration provides the data and persistence layer beneath it, so developers can easily take an agent from prototype to production. These agents can maintain conversation and session memory independent of any single machine, and carry a user’s history across sessions.

Providing a unified agent architecture

At the core of this integration is Google ADK 2.0, which serves as the code-first orchestration layer for multi-agent systems. ADK 2.0 provides the deployment framework, while MongoDB provides session and memory storage for agents.

Integrating directly with the Gemini Enterprise Agent Platform, ADK 2.0 eliminates the need for custom routing logic by providing standardized tool-calling interfaces, native telemetry, and enterprise access controls out of the box. Developers can use Gemini 4 Argon for multi-step reasoning and Gemini 3.8 Flash for low-latency execution.

How do Sessions and Memory storage work?

Great agent experiences improve with context. This integration introduces MongoDbSessionService to persist sessions, events, and application-, user-, and session-scoped state to MongoDB.

This means developers can build agents that retain the context needed for longer-running and multi-turn interactions. The session service includes optimistic concurrency control to help applications safely manage concurrent updates, while configurable collections make it easier to fit the service into existing data architectures.

The following Python code shows how you can add MongoDbSessionService to your agent:

from google.adk.integrations.mongodb import MongoDbSessionService
from google.adk.runners import Runner


session_service = MongoDbSessionService(
    connection_string="mongodb+srv://...",
    database_name="my_app",
)

runner = Runner(
    agent=agent,
    app_name="my_app",
    session_service=session_service,
)

session = await session_service.create_session(
    app_name="my_app", user_id="user_123"
)

async for event in runner.run_async(
    user_id="user_123",
    session_id=session.id,
    new_message=types.Content(role="user", parts=[types.Part(text="Hello!")]),
):
    if event.content and event.content.parts:
        print(event.content.parts[0].text)

For agents that need memory beyond a single session, MongoDBMemoryService ingests session events into MongoDB collections and recalls relevant memories through keyword matching. Memory ingestion is idempotent, which prevents duplicates when a session is added again.

from google.adk.integrations.mongodb import MongoDbMemoryService
from google.adk.runners import Runner


memory_service = MongoDbMemoryService(
    connection_string="mongodb+srv://...",
    database_name="my_app",
)

runner = Runner(
    agent=agent,
    app_name="my_app",
    memory_service=memory_service,
)

await memory_service.add_session_to_memory(session)

# Recall the most relevant memories for a query (ranked and capped).
memories = await memory_service.search_memory(
    app_name="my_app", user_id="user_123", query="preferred shipping address"
)

Together, these services help agents become more useful over time without requiring developers to stitch together separate systems for retrieval, state, and memory.

How does MongoDB Search help agents with data retrieval?

Having memory for agent state is essential, but agents also need fast access to the operational data they search over, the same product catalogs, documents, and records your application already queries. This is where MongoDB’s built-in search capabilities come in. MongoDB Vector Search leverages Voyage AI’s embedding models, which are the same engine backing MemoryService’s semantic retrieval in this integration.

Developers can also perform hybrid search by combining vector search with full-text search using the $rankFusion aggregation stage.

This gives agents access to the right information and context at the right time, delivering better responses in production. Whether you are building product discovery, knowledge assistants, support experiences, or domain-specific copilots, agents can automatically leverage semantic and hybrid retrieval to ground their responses in your data.

The application owns the database connection, database name, and search configuration. The agent gets focused tools to call, without exposing those operational details in its context.

The following Python code shows how to add MongoDB Search to your agents:

from google.adk.agents import Agent
from google.adk.integrations.mongodb import MongoDbToolset

search_tools = MongoDbToolset(
    connection_string="mongodb+srv://...",
    database_name="products_db",
)

agent = Agent(
    model="gemini-2.5-flash",
    name="product_search_agent",
    instruction=(
        "Search the products collection with your MongoDB tools. "
        "Embed the user's query before calling the search tools."
    ),
    tools=[search_tools],
)

Using MongoDB as the operational data store, the memory layer for agent state, and the search engine over both, developers get one platform instead of three, on whatever deployment they run it on. There’s no separate vector database to stand up, no search engine to sync against a second copy of the data, and no added latency from hopping between systems before the agent can respond.

Which deployment versions does this integration support?

The integration supports MongoDB Atlas and self-managed MongoDB 8.0+ deployments for vector and hybrid search. Developers can provide a connection string or bring an existing PyMongo client, and configure search behavior through MongoDbToolSettings, including index names, embedding fields, limits, and candidate counts.

That flexibility makes it possible to align the integration with existing data models, deployment choices, and performance requirements.

Get started today!

Install the MongoDB Google ADK Integration by running the command below:

pip install google-adk[mongodb]

You can learn more about the integration in the documentation.

Introducing the MongoDB Google ADK Integration: Durable Memory for Agent State was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Claude Agent SDK is a package that lets you embed Claude Code’s own agent loop directly into your application: web search, file search, file editing, MCP server connections, and more, all without having to define those tools or manage the agent loop yourself. The SDK also automatically persists session transcripts to the local file system by default. Claude calls it the Agent SessionStore.

Today, we are excited to announce the MongoDB Claude SDK Session Adapter, available in Python (with TypeScript coming in December 2026).

What is the Claude Agent SessionStore?

But first, let’s understand what a SessionStore actually is. It is the extension point in the Claude Agent SDK for saving an agent’s session outside the machine where it ran. Configure one, and the SDK mirrors every conversation turn and action as the agent works, so the session can pick back up exactly where it left off, on any instance that has access to that store.

That matters most once an agent stops running on a single, persistent machine. Say a team deploys their Claude agent as a Lambda function, callable as an internal service by teams across the company. Every invocation is a disposable instance; there’s no local disk to store anything between calls. Without a SessionStore, each request is stateless by default; the agent has no memory of what came before. With a configured SessionStore, any invocation can pick up a session’s full history, regardless of which instance handled the last turn.

That is the gap the MongoDB Session Adapter closes.

This is especially useful for enterprises that have multiple teams developing agents across many machines, who need session storage that survives host failures, scales as the agent fleet increases, and enforces the same access controls and retention rules as the rest of their data estate, not a bolt-on system built solely for agent memory.

The MongoDB Claude SDK Session Adapter

The MongoDB Claude SDK Session Adapter gives Claude agents durable session storage: a record of the conversation and its action history that lets an agent pick back up exactly where it left off, whether that’s on the same laptop, a different server, or any server in a distributed deployment.

The adapter helps cover three crucial areas:

• Cross-environment continuity: Whether a different backend instance picks up a conversation after a restart or a user resumes on a different device, session state doesn’t live on the machine that started it. Because every step is mirrored to MongoDB as it happens, any instance in your deployment can resume exactly where the last one left off.
• Resilience to failure: If a host crashes or a container gets rescheduled mid-conversation, nothing is lost. Entries are written to MongoDB as they occur, not batched or cached locally, so whichever instance picks the session back up sees the same state as the failed one, up to the last completed step.
• Retention and auditability: Many teams need to retain agent transcripts to meet internal data-retention or audit requirements. Instead of standing up a separate system just for that, session data lives in the MongoDB cluster you’re already operating, under the same access controls and backup policies as the rest of your data.

How it works

This adapter implements the Claude Agent SDK’s session storage interface directly, so you can mirror session transcripts to MongoDB in a few lines in ClaudeAgentOptions:

from pymongo import AsyncMongoClient
from claude_agent_sdk import ClaudeAgentOptions
from claude_agent_sdk_mongodb_session_store import MongoDBSessionStore

client = AsyncMongoClient(MONGODB_URI)
store = MongoDBSessionStore(client=client)
await store.create_schema()

options = ClaudeAgentOptions(
  session_store=store,
  # rest of your agent's configuration... 
)

That’s it. Every session started with these options is stored in MongoDB and can be resumed across machines. Each agent writes locally; MongoDB makes the conversation portable between app instances.

Get started today!

Simply run the following command to start with session storage for your Claude agents:

pip install claude-agent-sdk-mongodb-session-store

Linked resources:

• Integrate MongoDB with the Claude Agent SDK
• MongoDB Session Adapter repo on Github
• Agent SDK overview — Claude Code Docs

Introducing the MongoDB Claude SDK Session Adapter: Durable Session Storage for Agents was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.

When an application uses Amazon Bedrock for foundation models and MongoDB Atlas for its data, the split I keep is strict: Bedrock owns the model, Atlas owns the data. Bedrock gets two kinds of stateless calls text in and a vector out, or a prompt with context in and tokens out. Everything that has to be stored, filtered, secured, or migrated stays in Atlas.

This article explains how that split works, where the two sides meet, and what it looks like in a small Spring Boot application built with Spring AI. The sample answers questions about internal technical documentation, and the full code is on GitHub.

A short note on the two services, for readers who know only one of them. Amazon Bedrock is the AWS-managed service for foundation models: it serves models from Amazon (Titan, Nova), Anthropic, Meta, Mistral AI, Cohere, and other providers behind a single set of APIs, with no infrastructure to run and on-demand pricing based on tokens processed. It also leaves the choice of where inference runs to the application: a model can be called in a single AWS Region, through a geographic inference profile that keeps processing within a single geography, such as the EU, or through a global profile that may route to any supported commercial Region. MongoDB Atlas is MongoDB’s managed database service on AWS, Azure, and Google Cloud. Its Vector Search feature stores vector indexes alongside the documents in a collection, and the application queries them using the $vectorSearch aggregation stage, including filters on regular document fields. In this design, one serves models, while the other keeps the data and never calls a model, which is what makes the split below natural.

The principle: one owner per piece of state

A retrieval-augmented generation (RAG) system produces a lot of state: source documents, the chunks cut from them, an embedding for each chunk, metadata, and the rules that decide who may read what. Each of these needs one owner, meaning one system that decides when it’s created, when it changes, and when it’s removed.

An embedding is a derived copy of a chunk of text. I treat it like any other derived data: it belongs with its source, because freshness, deletion, and access control all follow the source. If a page is withdrawn, its vectors must go with it. If a page belongs to one team, its vectors belong to that team too. Atlas holds the source, so Atlas holds the derived copy.

Bedrock’s inference APIs fit the other side of the line without friction. InvokeModel and Converse keep nothing between calls, so they can own exactly what a model provider should own: which model runs, which version, how much capacity, and the API that hides the differences between providers.

The seam between the two sides is narrow. Text crosses it on the way in; vectors and generated text cross it on the way out. Documents at rest, the filter logic and the team boundary never cross it.

The seam in Spring AI

Spring AI maps onto this split almost one-to-one. EmbeddingModel and ChatModel are the Bedrock side, VectorStore is the Atlas side, and QuestionAnswerAdvisor is the code that carries a request across the seam. The sample uses Spring AI 2.0.1 on Spring Boot 4.1 and Java 21, with these dependencies:

<!-- Bedrock side: the models -->
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-bedrock</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-bedrock-converse</artifactId>
</dependency>

<!-- Atlas side: the data -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-vector-store-mongodb-atlas</artifactId>
</dependency>

<!-- Retrieval in Atlas, generation in Bedrock -->
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-vector-store-advisor</artifactId>
</dependency>

On Spring Boot 4, the Spring Data MongoDB auto-configuration lives in its own module, so spring-boot-starter-data-mongodb has to be declared explicitly; without it, there’s no MongoTemplate for the Atlas vector store to use.

The configuration file shows the split more clearly than any diagram, because it has exactly two blocks:

spring:
  # Atlas owns the data: documents, chunks, vectors, metadata, index
  mongodb:
    uri: ${MONGODB_URI}
    database: ${MONGODB_DATABASE:docs}
  ai:
    vectorstore:
      mongodb:
        collection-name: doc_chunks
        index-name: doc_chunks_vector_index
        path-name: embedding
        metadata-fields-to-filter: team,product,version
        initialize-schema: ${DOCS_INIT_SCHEMA:true}

    # Bedrock owns the model: embeddings and generation, nothing stored
    model:
      embedding: bedrock-titan
      chat: bedrock-converse
    bedrock:
      aws:
        region: ${AWS_REGION:eu-central-1}
      titan:
        embedding:
          model: amazon.titan-embed-text-v2:0
          input-type: text
      converse:
        chat:
          model: ${BEDROCK_CHAT_MODEL:eu.anthropic.claude-haiku-4-5-20251001-v1:0}
          temperature: 0.2
          max-tokens: 800

Two settings in the Bedrock block deserve attention. Spring AI’s Titan integration defaults to the multimodal amazon.titan-embed-image-v1 model with an IMAGE input type, so a text application has to set both the model and input-type: text. There are no AWS keys in the file either: Spring AI falls back to the default AWS credential provider chain, which resolves environment variables, a local profile, or an IAM role.

The model identifiers also encode where inference happens. Titan is called by its base model id in eu-central-1, so embeddings are computed in that Region. The chat model uses the eu. geographic inference profile, which lets Bedrock pick an EU Region for each request while keeping processing within the EU; CloudTrail records the Region that actually served it.

Where data residency matters, this choice belongs in the configuration review as much as the Atlas cluster’s Region does.

Ingestion: Bedrock computes, and Atlas keeps

Each documentation page includes three metadata fields: team (the owning team, which also defines the access boundary), product, and version. Ingestion decides everything about the page’s lifecycle on the data side and calls Bedrock only to turn text into vectors:

public int ingest(DocPage page) {
    // Re-ingesting a page replaces its chunks: the data owner decides what is current.
    vectorStore.delete(new FilterExpressionBuilder().eq("page_id", page.id()).build());

    var source = new Document(page.body(), Map.of(
            "page_id", page.id(),
            "title", page.title(),
            "team", page.team(),
            "product", page.product(),
            "version", page.version(),
            // Nothing else records which model produced the vector, so we do.
            "embedding_model", embeddingModelId));

    List<Document> chunks = splitter.apply(List.of(source));
    vectorStore.add(chunks); // one Bedrock embedding call per chunk, then one write to Atlas
    return chunks.size();
}

This is one chunk as it sits in Atlas after ingestion, with the vector truncated and the content shortened:

{
  "_id": "d7e6dd6e-1df1-4c86-b4c4-9b3d6d1f8e0a",
  "content": "The gateway retries idempotent upstream calls (GET, PUT, DELETE) up to 3 times with exponential backoff starting at 200 ms and capped at 2 seconds. ...",
  "metadata": {
    "page_id": "platform-gateway-retries-2x",
    "title": "Gateway 2.x: retry policy for upstream calls",
    "team": "platform",
    "product": "gateway",
    "version": "2.x",
    "embedding_model": "amazon.titan-embed-text-v2:0",
    "chunk_index": 0,
    "total_chunks": 1,
    "parent_document_id": "1b545dd0-e1aa-40e3-b992-62cce654e089"
  },
  "embedding": [-0.004906, 0.013096, 0.076597, -0.014013, "... 1024 values"],
  "_class": "org.springframework.ai.vectorstore.mongodb.atlas.MongoDBAtlasVectorStore$MongoDBDocument"
}

Three groups of fields share this document. The _id, content, metadata, and embedding shape come from Spring AI’s vector store; chunk_index, total_chunks, and parent_document_id come from the TokenTextSplitter, and _class comes from Spring Data.

The embedding_model field is mine. Bedrock doesn’t remember which model produced a vector, and Spring AI doesn’t record it either, so the data owner writes it down.

The index is where the two sides touch

There is exactly one place where a Bedrock decision gets written into Atlas, and it’s the vector index:

{
  "name": "doc_chunks_vector_index",
  "type": "vectorSearch",
  "definition": {
    "fields": [
      { "type": "vector", "path": "embedding", "numDimensions": 1024, "similarity": "cosine" },
      { "type": "filter", "path": "metadata.team" },
      { "type": "filter", "path": "metadata.product" },
      { "type": "filter", "path": "metadata.version" }
    ]
  }
}

With initialize-schema: true, Spring AI creates this index on startup. It reads numDimensions from embeddingModel.dimensions() and adds one filter field for each entry in metadata-fields-to-filter. The 1024 is Titan Text Embeddings V2’s default output size, and Spring AI’s Titan client doesn’t send a dimensions parameter, so this configuration produces 1024. The number belongs to the model, but it lives in the database. Any change of embedding model has to come through this file.

Query: Atlas filters and Bedrock answers

The query path builds the access filter on the data side and attaches it to the retrieval request:

public Answer ask(Question question) {
    var advisor = QuestionAnswerAdvisor.builder(vectorStore)
            .searchRequest(SearchRequest.builder()
                    .filterExpression(accessFilter(question))
                    .topK(4)
                    .similarityThreshold(0.5)
                    .build())
            .build();

    ChatResponse response = chatClient.prompt()
            .user(question.question())
            .advisors(advisor)
            .call()
            .chatResponse();
    // ... map the answer and the titles of the retrieved chunks
}

// Built with the typed builder, not by concatenating user input into a filter string.
static Filter.Expression accessFilter(Question q) {
    var b = new FilterExpressionBuilder();
    var team = b.eq("team", q.team());
    return (q.version() == null ? team : b.and(team, b.eq("version", q.version()))).build();
}

The advisor also accepts a filter at call time through its FILTER_EXPRESSION parameter, but that path parses a text expression. When part of the filter comes from a request, I prefer the typed FilterExpressionBuilder, which never concatenates user input into a string. Spring AI turns the expression into the filter of the $vectorSearch stage. For team = platform and version = 2.x, Atlas receives:

{$and:[{"metadata.team":{$eq:"platform"}},{"metadata.version":{$eq:"2.x"}}]}

The advisor includes only the text of each retrieved chunk in the prompt, while the metadata, the vectors, and any chunk that failed the filter remain in Atlas.

The sample data makes this visible. Two teams document a retry policy with different rules: the platform team describes the gateway, the payments team describes the ledger. Asked “What is the retry policy?” on behalf of platform and version 2.x, Claude Haiku 4.5 answers from the two gateway 2.x pages: three retries for idempotent calls; exponential backoff from 200 ms, capped at 2 seconds; POST retried only with an Idempotency-Key header. Asked the same question on behalf of payments, the answer from the ledger pages is: clients don’t retry on their own; the outbox worker retries at most five times, 30 seconds apart.

Then the payments team asks about the other team’s product, “How many times does the gateway retry upstream calls?”, and the answer is:

I can't answer that question based on the provided context. The documentation discusses retry behavior for the outbox worker (which retries at most 5 times) and ledger clients (which must not retry on their own), but it does not contain information about how many times the gateway retries upstream calls. 

The gateway pages live in the same collection and the same vector index as the ledger pages. The model still never saw them, because the filter ran in Atlas before any text left for Bedrock. This is the property I care about most in the split: the data owner decides what the model is allowed to read.

Where the two owners touch

The seam is narrow but not free. These are the points where I’ve learned to slow down.

• The model version is data. Changing the Bedrock embedding model, or only its dimensions, invalidates every stored vector. Because Atlas owns the vectors, the change becomes an ordinary data migration: add a second vector field, build a second index, re-embed in the background, switch the query, then drop the old field. The embedding_model field in each chunk tells the backfill which documents still need work, which is why I added it from the first day.
• The write path calls a remote model. In the sample, vectorStore.add calls Bedrock synchronously, one request per chunk, because Spring AI’s Titan client doesn’t batch. That keeps ingestion simple and gives read-your-writes on vectors. It also puts Bedrock latency and quotas on the write path, and Bedrock throttles embedding models by requests per minute. For a documentation corpus that changes a few times a day, I keep it synchronous. When the source is operational data that changes constantly, I move the embedding behind a change stream (or an Atlas Database Trigger that forwards to Amazon EventBridge), so a slow model call never blocks a write.
• The abstraction owns the document shape. VectorStore decides that chunks look like content, metadata, and embedding. That’s fine for a dedicated collection. When vectors need to live alongside existing operational documents, or the query has stages the abstraction doesn’t expose, I keep EmbeddingModel for the Bedrock call and write the aggregation myself. The sample’s NativeSearchService does exactly that; this is the pipeline it sends:

db.doc_chunks.aggregate([
  {
    $vectorSearch: {
      index: "doc_chunks_vector_index",
      path: "embedding",
      queryVector: [ /* 1024 floats from Titan V2 */ ],
      numCandidates: 100,   // 20 x limit, MongoDB's suggested starting point
      limit: 5,
      filter: { "metadata.team": "platform" }
    }
  },
  {
    $project: {
      _id: 0,
      title: "$metadata.title",
      version: "$metadata.version",
      content: 1,
      score: { $meta: "vectorSearchScore" }
    }
  }
])

• Searching for “retry policy” on the platform returns the gateway 2.x page first, with a score of 0.72, followed by the deprecated 1.x page. No payments page appears, for the same reason as before. Spring AI’s own store uses a fixed numCandidates of 200 unless configured otherwise; writing the pipeline by hand lets me tune it per query.
• Identity is split too. Bedrock authorizes AWS IAM and Atlas with database users, so the application holds two identities. In production, I give it an IAM role limited to invoking the two models it uses, and an Atlas user limited to the collection it reads and writes to.

The Split Pays Off

The split pays off when retrieval depends on data the application already owns: teams, tenants, versions, document status. When the corpus is a folder of documents with no operational state around them, Bedrock’s managed retrieval (Knowledge Bases, including the Bedrock Managed Knowledge Base that reached general availability in June 2026) removes work that this design asks the team to do, and I’d consider it seriously.

That work is real. Owning the data means owning chunking, the ingestion path, the index lifecycle, and the re-embedding migrations. A team without the capacity to run those will get more from a managed service than from a clean boundary.

The boundary itself can also move. Atlas Automated Embedding, generally available since August 2026, generates and refreshes embeddings inside Atlas with Voyage AI models. It puts the embedding model on the data side and leaves Bedrock with generation only, a different split from the one described here, and one worth evaluating on its own terms.

From zero to hero

1. Record the embedding model id in every chunk from the first ingestion, so a later model change can be planned as a migration.
2. Create the vector index explicitly from a versioned definition, with all filter fields, and set initialize-schema to false.
3. Derive the access filter from the caller’s identity rather than the request body, and build it using FilterExpressionBuilder.
4. Replace static or personal AWS credentials with an IAM role scoped to the models in use, and restrict the Atlas database user to the collection.
5. Size ingestion and backfills to the Bedrock requests-per-minute quota of the embedding model.
6. Move embedding off the write path, behind a change stream, once write volume or latency makes the synchronous call visible.

Conclusion

Bedrock is good at serving models, and Atlas is good at keeping data accurate, filtered, and up to date. Keeping each on its own side of a narrow, stateless seam means a model change stays a configuration change, a data change stays a database change, and the decision about what the model may read is made by the system that owns the data.

Rent the inference, own the state.

Bedrock Owns the Model, Atlas Owns the Data: Amazon Bedrock and MongoDB Atlas with Spring AI was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.

A hands-on experiment testing when knowledge graphs actually help, when agents do it better, and what you can build in a single database.

This article was written by Jannis Busch.

Why this experiment

In conversations with engineering teams building AI on their enterprise data, one pattern comes up constantly: teams get ambitious about knowledge graphs.

Graphs feel intuitive. Everything in business seems connected: customers, orders, accounts, parts. So the immediate instinct is often: “We need a graph database or GraphRAG to make our AI smart.” But building and maintaining a graph isn’t free. It requires upfront ontology design, entity extraction pipelines, and synchronizing operational data into a secondary structure.

Meanwhile, over the past two years, LLM reasoning and agent tool-calling have matured significantly. Because modern agents can selectively fetch only the records they need, multi-step reasoning has become practical and economically viable without having to maintain a separate knowledge graph. That raises a practical question: is building and maintaining a specialized graph pipeline still worth the overhead, or can modern agents solve the problem directly on your existing operational data?

Opinions on this are easy to find; measurements are not. So we built a repeatable experiment around two questions:

1. When does a graph actually improve answers?
2. What does each approach cost to build, run, and keep in sync?

If your goal is solving a real issue, e.g., building reliable Q&A on your data, and you want to know the most effective architecture to use, read on. If you just fancy playing with graph technology for fun, go right ahead. But if you want to know what architecture actually pays off, here is the data.

How we tested

We generated three synthetic datasets, each with a different structural shape, and wrote about 100 questions for each. The questions were fixed before any run. Every question has a known right answer computed by code, and every answer was evaluated programmatically by code, not judged by an LLM.

Four approaches answered the same questions using the same model (Claude Sonnet 5.5) and the same instructions. For every answer, we recorded whether it was correct, what it cost, and how long it took. The source documents and the extracted graph facts were created using Claude Opus 5.5, mirroring how real-world projects extract graph data from messy source material.

Everything ran locally on a laptop using MongoDB Community in Docker. The entire project cost about $85 in LLM calls across all reruns.

How to read the numbers: Unlike real-world enterprise repositories, this synthetic data is consistent: facts don’t contradict each other and nothing is outdated. Real systems will rarely hit 100% accuracy. Read these scores as best-case baselines: the point of the experiment is the difference between the approaches and what each one costs.

Some background

Understanding the fundamental concepts behind databases and graphs is extremely important to be able to assess their value to a certain use case. Hence, here is a brief rundown:

Databases, documents and graphs

A database keeps information so you can find it again by what you know about it: the account with ID A-1042, or every transfer over €9,000.

Some databases keep rows in tables, like a strict spreadsheet. Others, like MongoDB, keep documents: one record with everything that belongs to it, nested as needed. Both are good at the same thing: finding records by their properties.

A graph stores that same data differently. In tables or documents, a relationship is a shared value (like a phone number or foreign key), and you discover the connection by querying matching fields. In a graph, a connection is an explicit line you can follow.

In our running example, Alice and Bob share a phone number. In documents or tables, you have to notice that by searching for records with matching phone fields. In a graph, it is one dot with two lines. A graph database stores each record together with its relationships, so following one is a jump to the next record rather than a search through a table.

Three ways to follow a link

Following links is not unique to graph databases. MongoDB offers two ways:

• $lookup joins one collection to another, exactly one step.
• $graphLookup repeats that join, following links of one kind as far as they go and returning everything it reached.

A dedicated graph database treats relationships as the data itself and can return the path between two records directly, across every kind of link at once. All three can follow a link. They differ in how far they go and what they hand back. MongoDB covers the first two natively, which is often enough.

Four approaches

RAG and GraphRAG

RAG (Retrieval-Augmented Generation) means: before the model answers, find the pieces of text most similar to the question and hand them over. The model answers from what it was given. In our setup, RAG hands over the twelve most similar passages and asks once.

GraphRAG keeps all of that and adds one step: it looks up the things the question and text mention in a graph and adds what they are connected to. Our GraphRAG expands three steps from every account, part, or feature it recognizes, up to 600 links, and still answers in a single pass. It is RAG plus neighbors: more context, and more cost.

Agents

An agent is a model that works in steps. It gets a set of tools (search the text, filter a document, look up a record), decides which one to call, reads the result, and decides again, for up to twelve steps, until it can answer. RAG does not disappear: searching for similar text becomes one of the agent’s tools, next to precise lookups. RAG answers in one pass; an agent can take a second look (or a third, or a fourth…).

An agent with a graph is the same agent with extra tools to walk relationships: follow links from a record, find the shortest path between two records.

That gives four approaches, defined by two questions: does it answer in one pass or in steps, and does it use a graph?

What each approach costs

The bill per question is the small part. The higher cost is what you build, run, and keep in sync. We measured cost and time per question and counted the moving parts (measured here on the machine-builder dataset):

• RAG is the cheapest to build and run: embeddings and one prompt. When documents change, you simply re-embed them.
• GraphRAG costs the most per question because it pastes a whole neighborhood of nodes into every prompt. Building the graph took an upfront LLM extraction pass, and the graph has to be re-extracted whenever documents or the data model change.
• Agents read only what they ask for, so they cost little more per question than RAG, but they take longer and need well-described tools and extracted fields.
• The agent with a graph combines both: one extraction feeds the documents and the graph, but there are multiple copies of the data to keep in sync.

Three kinds of data

The same four approaches ran against three datasets with very different shapes. Which one looks like your data?

• A help center (islands): 52 help articles, release notes, and a pricing page. Almost every answer sits in one self-contained article.
• A machine builder (trees): Installed machines, hierarchical parts lists, work orders, and service bulletins. Questions often follow parts lists several levels deep.
• A fraud network (a web): Accounts, people, phones, devices, transfers, and analyst notes of a bank. The answer is often the relationship itself.

Results

Help center: when RAG is enough

Ask “where do I turn on guest access?” and the answer sits in one article; finding that article is the whole job.

RAG answered all 100 questions correctly for $0.40 per 100 questions. The other three approaches were just as accurate but cost about twice as much. With a real knowledge base, no approach will reach 100%, but the order is clear: if each answer lives in one document, search is enough, and a graph adds cost, not answers.

Machine builder: the middle ground

Parts lists are trees, work orders point at machines, and some facts exist only in technicians’ notes. On the 53 questions all four approaches were run on, RAG got 23 right.

The big jump came from working in steps: the Agent got 52. Adding the graph added one more, for 53. GraphRAG landed in between, at the highest price per question.

The examples show why. Counting with filters (“How many leak repairs on one machine model in 2023?”) is a document job: two steps with structured queries, versus ten to twelve steps and ten times the token cost through a graph. Following a relationship (“Which known causes are listed for the failure in this repair?”) is a graph job: three hops, complete. The structured Agent handles the vast majority of this workload efficiently without needing a full graph pipeline.

Fraud network: when a graph pays off

A bank’s fraud team works with highly interconnected accounts, transfers, and case notes. Here, the answer is the relationships: who shares a phone, where the money went.

RAG got 41 of 99 questions right. The Agent with a graph got 95, for about the same price as the agent without one ($1.37 against $1.35 per 100 questions).

The clearest difference is following money. Asked where the money from one account went, up to three transfers on, the agent without a graph looked up transfers one account at a time and lost track after four lookups. The agent with a graph made one lookup (“follow transfers three steps” via $graphLookop), and named all 14.

GraphRAG also ran into limits here: it pastes up to 600 links into one prompt, and five times the model simply ran out of room before answering (counted as wrong).

Counting transfers and spotting bursts within an hour remained document query work, where both agents were perfect.

The architectural bottom line: Do you need a separate graph database?

Graph databases have earned their place: when your data truly is a network, and your questions walk deep paths across millions of links, all day.

But if your goal is to build an effective, production-grade Q&A assistant over enterprise data, you rarely need to introduce a dedicated, standalone graph database. A common architectural trap is assuming that because you want to traverse relationships, you must build a secondary graph pipeline. Doing so introduces ETL hurdles, cache invalidation issues, and secondary hosting costs.

In this entire experiment, the Agent and the Agent with a graph both ran natively within a single MongoDB database:

• Vector Search handled semantic document retrieval.
• Rich Document Schemas allowed the agent to filter, aggregate, and count records rapidly.
• $graphLookup handled recursive graph traversals and multi-hop relationship queries natively.

Three questions to help you decide:

1. What does your data look like?

• Islands (each answer in one doc) → search (RAG) is enough.
• Trees and links (parts lists, orders) → an agent with good document tools.
• A web (everything linked to everything) → give the agent a graph.

1. What do your questions look like?

• Find, filter, count → documents and structured queries.
• Follow a few links → $graphLookup in MongoDB.
• Deep paths and busy hubs across millions of links, all day → a dedicated graph database, fed from MongoDB by change streams.

1. Always wanted to play with a graph database?

• Just go for it. A team that loves its tools builds them well. And sketching your business as a graph is worth it even if you never build one.

In short: If each answer sits in one document, plain RAG is enough. When questions follow relationships, an agent gets you most of the way, and native database traversals like $graphLookup close the gap. By combining modern LLM agents with a unified operational database, you can get the reasoning depth of a knowledge graph with none of the multi-database overhead.

About this experiment: All data is synthetic, and all names are made up. Questions were fixed before any run, answers were checked by code against known correct answers, and every scoring fix is logged. The code and every recorded answer are in the GraphRunner repository.

Further reading from MongoDB

• MongoDB as a graph database: graph basics, $graphLookup, and when to add a graph database.
• GraphRAG with MongoDB and LangChain: a tutorial that builds a knowledge graph and queries it with $graphLookup.
• GraphRAG on MongoDB Atlas: a blog post (April 2025) on when GraphRAG helps and what it costs.

When Does a Graph Actually Pay Off? was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.

The handful of Python functions that gave your first agent its tools — fetch a customer profile, query a database, search a knowledge base — won’t survive production. They run fine on a laptop and fall apart the moment they hit a distributed environment. And by the time they do, three other teams have quietly hand-rolled the same ones.

As AI adoption spreads across an organization, the decentralized approach turns into a compounding tax: with no centralized way to share, discover, or govern agent tools, every team rebuilds the same capabilities in isolation.

What began as rapid innovation quickly hardens into a messy web of duplicated effort, diverging implementations, and zero production visibility.

Why Ad Hoc Agent State Breaks in Production

If you look inside almost any enterprise agent prototype today, you’ll find variations of the same brittle patterns. State lives in a dictionary:

agent_state = {}  # In-memory state: fast to write, but dies instantly on a pod restart

def get_agent_memory(session_id):
    return agent_state.get(session_id, {"history": [], "context": {}})

def save_agent_memory(session_id, memory):
    agent_state[session_id] = memory

This code works perfectly on a local laptop during a demo. But the moment you deploy it to a distributed environment, it shatters. In-memory state vanishes during routine auto-scaling events. So you write it to disk:

import json
import os



def get_agent_memory(session_id):
    # No schema enforcement, zero concurrency control, and absolutely no versioning
    if os.path.exists(f"memory_{session_id}.json"):
        with open(f"memory_{session_id}.json", "r") as f:
            return json.load(f)
    return {"history": [], "context": {}}

def save_agent_memory(session_id, memory):
    with open(f"memory_{session_id}.json", "w") as f:
        json.dump(memory, f)

Concurrent file writes lead to data corruption.

Worse yet, this data architecture forces a massive coordination failure across the organization. Because there is no shared ecosystem, every team is forced to build its own ad hoc plumbing in isolation.

Agent Tooling Is an Infrastructure Problem, Not a Coding Problem

When this starts to fail, the engineering instinct is to write cleaner Python: wrap the scripts in better abstractions, add a lock for concurrency, build a wrapper class.

But cleaner Python code won’t fix a fundamentally broken data architecture. Think of a centralized tool registry and state layer as the agent-era equivalent of a package manager. You wouldn’t expect every team to maintain their own private fork of pip or write a custom network protocol just to share code — you use a centralized registry. Without one, you lose control over reuse, governance, and cost.

How Agent Tool Duplication Happens: Support vs. Sales

Consider Team A (Customer Support) and Team B (Sales Operations). Both are building AI assistants, and both realize their LLMs need access to internal documentation to be useful.

Operating in silos, they write their own implementations of a knowledge base search tool. Team A spins up a script that reads a local file index, while Team B deploys a standalone server instance. Both are trying to solve the exact same problem with fragmented code.

What Breaks When Agents Scale: Duplication, Cost, and Governance

As these applications move from proof-of-concept to hundreds of daily active users, three pressures force an architectural shift.

• Duplication: Team A’s support repo and Team B’s sales repo contain nearly identical code, running in separate silos. Both maintain separate pipelines to process the same internal manuals.
• Escalating Costs: Every siloed setup needs its own compute, database instances, and embedding pipelines. You pay to embed the same document multiple times. Without a shared cache or state layer, agents hit LLM APIs repeatedly for redundant context, bloating token consumption.
• Missing Governance: When an agent hallucinates or touches restricted data, tracing the cause across ad hoc Python files and local JSON logs is an operational nightmare. There’s no single place to audit what an agent accessed, no role-based access control (RBAC) over which agent can call which index, and no clean way to deprecate a data version when an upstream document changes.

What a Proper Agent Data Layer Looks Like: Four Core Pillars

To solve tool sprawl and state fragmentation, organizations must transition from ad hoc scripts to a dedicated Agent Data Layer. This architecture relies on four core pillars:

• Centralized Tool Registry: Tools defined as discoverable documents with versioning, ownership, and schema — queryable by agents at runtime instead of imported from a local module.
• Shared Memory and State Storage: Session histories are dynamic and unstructured; forcing them into rigid SQL tables creates friction. They belong in a flexible, document-native format.
• Co-located Vector Search and Operational Data: Splitting operational state into one store and embeddings into a separate vector “sidecar” adds latency and sync headaches. A proper layer unifies both.
• Observable by Default: Every state transition, tool execution, and vector query is written to an indexed, queryable store — turning logs into operational intelligence.

The Fix: Building an Agent Data Layer with MongoDB Atlas

The beauty of shifting to a robust data layer is that your core LLM logic never changes. The agentic loop, business logic, and embedding models stay untouched; only the persistence layer evolves. Here’s how to consolidate Team A and Team B onto MongoDB Atlas.

Step 1: Build a Centralized Agent Tool Registry in MongoDB

Instead of Team A and Team B each maintaining a copy of search_knowledge_base, the platform registers it once. The registry is just a collection: tools are documents with a schema the LLM can read, an owner, and a version.

    """
    tools_col.update_one(
        {"name": name, "version": version},
        {"$set": {
            "name": name,
            "description": description,   # what the LLM reads to decide when to call it
            "input_schema": input_schema, # JSON Schema the LLM fills in to call it
            "owner": owner,
            "version": version,
            "is_active": True,
        }},
        upsert=True,
    )

def discover_tools():
    """An agent asks the registry what it can do — instead of importing a local module."""
    return list(tools_col.find({"is_active": True}, {"_id": 0}))

Now there’s a single source of truth for what tools exist, who owns them, and which version is live. Deprecating a tool is a flag flip (is_active: False), not a code change shipped across two repos.

Step 2: Store Documents and Vector Embeddings Together

Instead of separate local JSON files and on-disk indices, unify everything into one globally accessible collection. Pass the vector arrays straight from your existing embedding model — the ingestion interface stays uniform across teams.

knowledge_col = db.knowledge_base

def ingest_document(title, content, embedding, team):
    """
    A single, unified ingestion function used across the org.
    Replaces per-team file writes with a single collection.
    """
    document = {
        "title": title,
        "content": content,
        "embedding": embedding,  # Generated by your existing, unchanged embedding model
        "metadata": {
            "team": team,
            "version": "1.0",
            "is_active": True
        }
    }
    return knowledge_col.insert_one(document)

Step 3: Replace FAISS with Atlas Vector Search and Pre-Filter for Multi-Tenant Isolation

The most common multi-tenant bug here is post-filtering. Running a standard aggregation $match after a vector search stage filters an already-truncated result set — so if the top candidates all belong to another team, the query returns an empty array even when relevant documents exist.

Instead, we use native pre-filtering inside the $vectorSearch operator, which scopes the nearest-neighbor search before candidates are ranked. Define the MongoDB Vector Search index with the filter field declared:

{
  "fields": [
    { "type": "vector", "path": "embedding", "numDimensions": 1536, "similarity": "cosine" },
    { "type": "filter", "path": "metadata.team" }
  ]
}

Then scope the search — and this is also where tenant isolation actually lives. The allowed_teams argument is resolved from the calling agent’s verified identity upstream; it is never a free parameter that the agent passes for itself. That distinction is the difference between data scoping and real access control.

def search_knowledge_base(query_embedding, allowed_teams):
    """
    Platform-owned search. `allowed_teams` is derived from the caller's authenticated identity — not chosen by the agent at call time. Pre-filtering enforces the boundary on the cluster, deterministically.
    """
    vector_search = {
        "index": "vector_index",
        "path": "embedding",
        "queryVector": query_embedding,
        "numCandidates": 100,
        "limit": 5,
        "filter": {"metadata.team": {"$in": allowed_teams}},
    }
    return list(knowledge_col.aggregate([{"$vectorSearch": vector_search}]))

Pre-filtering gives you correct results. Binding the filter to identity instead of to a request parameter is what makes it governance.

Step 4: Add Agent Observability and Audit Logging by Default

Once every tool interaction passes through a unified layer, logging and auditing stop being afterthoughts. Each call is recorded automatically, and cross-team metrics become a standard aggregation.

from datetime import datetime, timezone

logs_col = db.agent_logs

def log_tool_call(agent_id, team, tool_name, latency_ms, status):
    """Capture tool execution analytics directly in MongoDB."""
    logs_col.insert_one({
        "agent_id": agent_id,
        "team": team,
        "tool_name": tool_name,
        "timestamp": datetime.now(timezone.utc),
        "metrics": {"latency_ms": latency_ms, "status": status},
    })

def query_cross_team_usage():
       """Aggregate cross-team usage and latency instantly."""
    pipeline = [
        {
            "$group": {
                "_id": "$team",
                "avg_latency": {"$avg": "$metrics.latency_ms"},
                "total_calls": {"$sum": 1}
            }
        }
    ]
    return list(logs_col.aggregate(pipeline))

Before and After: Consolidating Agent Infrastructure on MongoDB

Moving agent infrastructure off decentralized scripts shrinks the operational footprint dramatically.

This goes beyond clean code. Moving infrastructure logic out of application code and into an organizational data layer establishes a repeatable framework for security, optimization, and collaboration. For a deeper look at how this combats operational drag across enterprise environments, read our op-ed on Fighting Tool Sprawl: The Case for AI Tool Registries.

How to Migrate from Ad Hoc Tooling to an Agent Data Layer

Transitioning away from ad hoc tools doesn’t require a risky, multi-month rewrite of your entire codebase. Because frameworks like LangChain, LlamaIndex, and LangGraph decouple orchestration from storage, the LLM logic doesn’t change — only the persistence layer does.

You can capture quick wins incrementally:

• Step 1: Replace local JSON file persistence with a single MongoDB collection.
• Step 2: Don’t try to “move” a FAISS index — there’s nothing portable to lift. Extract the vectors and their source records, insert them as documents, and build an Atlas Vector Search index over the collection. Native framework integrations handle most of the wiring.

Conclusion: Stop Rebuilding Agent Infrastructure in Every Team

Building ad hoc tooling in Python is a perfectly reasonable way to stand up a prototype and prove out a value proposition. But as adoption matures, letting every team maintain its own isolated, hand-rolled solutions becomes a coordination failure.

Consolidating agent state, memory, and vector requirements into MongoDB Atlas isn’t adding complexity — it’s a consolidation move. It removes the operational burden of managing disjointed infrastructure so teams can stop reinventing core data patterns and start shipping production-grade agentic experiences.

Ready to simplify your AI infrastructure and eliminate tool sprawl? Spin up a fully managed instance in minutes on the MongoDB Atlas free tier and start building a unified data layer for your enterprise agents today.

Replace Your Ad Hoc Python Tooling with a Proper Agent Data Layer was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.

A mandatory path is an agent architecture in which every action an agent takes is routed through a single point that can permit it, record it, and meter it. Permission and budget checks run before an action executes, and actual usage is reconciled once it completes. Model calls pass through that point; so do tool calls, memory and retrieval reads, and handoffs to sub-agents. No agent has a route to a model provider or an internal API that goes around it.

This post looks at what a single interception point gets you, how much of an agent’s work it has to see before it can answer the cost question, and what putting it in the request path costs in latency and availability. We’ll call the component that sits there the enforcement point. It runs inside the execution environment but outside any one agent’s loop. Tool calls and retrieval have costs of their own, but this post is about the inference they trigger.

Spending more tokens on a hard problem can be a sound engineering call, but some companies have used token volume itself as a measure of productivity. Telling the two apart requires attributing spend to a task, and neither the provider invoice nor an LLM proxy currently does that. The invoice arrives weeks after spending decisions are made, with no way to tell which task, agent, or user produced any of it. A proxy sees model calls, but misses the tool calls, retrieval reads, and handoffs between them. Per-task attribution, or stopping a run that’s over budget, takes an enforcement point that every agent action has to pass through and that works with whatever frameworks and providers teams already use.

Why do agent bills behave differently from API bills?

Agent spend scales with the number of steps a task takes. A single agent task does not produce one inference call. It produces a planning call, several tool calls, model calls to evaluate what the tools returned, retries when a tool fails, delegations to sub-agents, and a final synthesis pass (Prefactor walks through this multiplication).

Each model step is billed separately, often against different models at different prices. The tool calls between them decide how many model steps there will be: when a tool sends back something the agent can’t parse, the agent calls the model to work out what went wrong, then tries again.

While per-token prices have fallen steeply since 2023, enterprise AI budgets have grown (Zylos, April 2026). Some of that growth is adoption. The rest comes from the step count described above, and cheaper tokens don’t reduce the number of steps a task takes, whether that’s working through a hard problem or retrying a tool call that keeps failing.

Uber rolled out Claude Code to roughly 5,000 engineers in December 2025. Adoption rose from 32% of engineers in February to 84% classified as agentic coding users by March, with internal leaderboards ranking engineers by usage. The entire 2026 AI budget was exhausted by April, with reported spend averaging $150 to $250 per engineer per month and heavy users at $500 to $2,000 (Forbes, May 2026). COO Andrew Macdonald has since said the link between token spend and useful features shipped “is not there yet” (Fortune).

Uber responded with per-engineer tracking on a dashboard each employee can see, a cap of $1,500 per employee per tool per month, and an approval path for exceeding it (TechCrunch). All three measures are keyed to a single person. That works for a coding tool, where the engineer both asks and runs the agent, and stops working once an agent acts for someone else or spawns agents of its own.

When is spending more tokens the right engineering call?

Spending more tokens is the right call when a wrong answer costs more than the extra inference. Deep research, long-horizon coding, contract analysis, and multi-step planning improve with more reasoning passes, more context, and more attempts.

For that work, tokens are often the cheapest input available, and spending them deliberately where errors are expensive, sometimes called token maxxing, is a sound trade-off.

Joche Ojeda’s July 2026 post on token-maxxing makes the same argument. He also notes that fan-out has diminishing returns, so the skill is in deciding how much to spend.

The term has picked up a second meaning. In April 2026, Gergely Orosz documented token leaderboards and spend targets at companies including Meta and Salesforce, which rewarded consumption as a productivity signal. Some engineers responded by inflating usage with work that produced nothing. Uber had the same setup, with one more problem: the teams pushing adoption weren’t the teams holding the budget (Forbes).

Token count rises with loop depth. Loop depth rises when a task is hard, and it also rises when a tool returns something the loop doesn’t handle and the agent retries or re-plans until something stops it. A ranking by volume counts hard work and an unproductive, stuck loop the same way.

Orosz also reports that Shopify cuts off access automatically when an engineer’s daily spend spikes, restores it if the spike was intentional, and uses the cutoffs to catch runaway agents. He quotes Shopify’s head of engineering, who finds it “more interesting to not look at ‘who spent the most in overall token cost?’ but instead, ‘whose tokens cost the most?’” Measuring cost per successful task is one way to make that question concrete, and PowerCode Group works through an example of that calculation. Yet the question is only answerable if spend can be attributed to a specific task and a specific requester, and whether that’s possible depends on where credentials live.

Why can’t the platform team see agent spend?

Where a credential lives determines where accounting is possible. Agents often hold their own provider credentials and call providers directly. If an agent can reach a provider without passing through anything the platform team operates, no downstream report can reconstruct which task or requester produced the spend.

A common deployment shape for agents is several services, built by different teams and deployed independently. Each holds provider API keys in its own environment and calls the provider directly. The provider sees traffic per key, finance sees one organization-level invoice a month, and nothing in between sees the task.

Where spend is attributed per agent, Zylos’s May 2026 write-up on token attribution describes the mechanism as tagging: direct attribution for calls tagged at dispatch, with the tags injected at the gateway layer, and proportional allocation for shared-context overhead that can’t be tagged. But tagging only works for calls that pass through the gateway. A call from an agent holding its own key carries no tag.

Tags also stop at hop boundaries. The OpenTelemetry GenAI conventions define no project or cost-center attributes, so, as Zylos notes, multi-hop agent chains lose attribution context unless the team adds custom attributes and propagates them. Separate keys at least let the provider’s per-key view tell one agent from another.

A shared service account removes even that. To keep deployment simple, teams often copy one broadly scoped credential into every agent, and no one can say which agent used it or why (a dev.to post on agent identity describes this pattern). That defeats both cost attribution and authorization, since both need to know which agent acted and for whom, and the credential doesn’t record either.

What does an LLM proxy see, and where does its scope end?

An LLM proxy intercepts model API calls, and anything the agent doesn’t route through it is outside its scope. The category includes open-source gateways such as LiteLLM, Portkey, and Agent Router (formerly Envoy AI Gateway), AI extensions of API gateways such as Kong AI Gateway, and managed services from Cloudflare, Vercel, and Helicone. The open-source gateways and Kong now also proxy MCP tool calls.

What a proxy gives you:

• Cost tracking per key, and in several products per team or user
• Latency and token usage by provider and model
• Automatic failover across providers and models
• Response caching
• Budgets and rate limits per key or consumer

Where its scope ends:

• Tool calls that don’t go over MCP through the gateway
• Retrieval and memory reads
• Sub-agent handoffs
• Identity beyond the key or consumer

In this illustrative example, a support-triage agent resolves one ticket for a rep who has billing access but not payment-method access.

A model-only proxy sees all six model calls, split across the parent agent’s key and the sub-agent’s, but none of the five actions between them. Unless something tags the calls with a shared task ID, nothing ties them to the same ticket or records whose authority the run carried.

Exposing the CRM and the ticket system as MCP servers through the gateway would bring three of those five into view (steps 3, 5, and 10), but not the vector read or the spawn. The spawn at step 6 is the only reason steps 7 through 9 happened, but without a tag that survives the handoff, the proxy books those calls with no link to the ticket.

Those calls also can’t be capped as part of the ticket. A per-key budget applies to each key separately. A per-task cap, a circuit breaker, or a kill switch needs a component on the path that knows which task each action belongs to and can act while the run is still going.

How does cost attribution relate to agent authorization?

Cost attribution and authorization run on the same plumbing. A token issued when the task starts, then re-minted and audience-bound at each hop, carries the originating task and identity through the chain. That lets a downstream API enforce the requester’s permissions, and it lets the platform sum the task’s cost across hops.

In the above example, the rep can see account history but not payment methods, while the agent’s own credentials allow both. The CRM checks the agent’s credentials, so it returns both, and nothing in the request tells it the rep is the one asking. The edge gateway, which authenticates and admits the inbound request at the perimeter, knows who started the task but nothing about what the agent did next.

MongoDB’s The Three Questions Agent Security Has to Answer works this through in depth; its decision-level interception pattern is the mandatory path argued from the audit side. The primary specifications are RFC 8693 (token exchange), RFC 8707 (audience binding), and the MCP authorization specification, which forbids token passthrough.

In our experience, teams fund authorization and cost attribution as separate projects, and prioritize the one a regulator has asked about. Where the identity work is already funded, the cost work can be added to it, since both need the same task identity. The identity side has those RFCs, the OWASP Top 10 for Agentic Applications (which ranks identity and privilege abuse third), NIST’s AI Agent Standards Initiative, and shipping implementations such as Auth0 and AWS AgentCore Identity.

The cost side has far less to build on. The closest thing to a standard is the OpenTelemetry GenAI conventions, still in Development status, which record token usage per call. No convention yet carries a budget or a cost attribution across hops.

What does the mandatory path cost?

Putting every action through one mandatory path adds a network hop to each action and puts the enforcement point in the request path as an availability dependency. It also needs capacity planning for its throughput ceiling, and it’s another component to secure. Deployments become coupled, since every team’s agents depend on infrastructure another team ships.

Availability has a sharper cost for agents than for batch infrastructure. A batch pipeline can queue while a dependency recovers, but an agent often has a person waiting on it, like the support rep in the example, so an enforcement point outage is a user-facing outage.

The enforcement point also needs a failure policy. If it fails closed and blocks every action, its outage becomes an agent outage. If it fails open and lets actions through, they run unpermitted and unmetered, which reopens the gap the enforcement point exists to close. The choice belongs to each action type, since a ticket write-back and a read of past tickets carry different risks.

Coupling can also reach past deployments. If routing every action through one path means committing to one agent framework, model provider, or cloud, the team now has a lock-in problem. The enforcement point has to stay neutral about what runs on top of it.

All of this depends on agents having no route around the enforcement point. An agent with provider credentials and open outbound access can call the provider directly. A framework default (or a developer in a hurry) can leave that route open. A permission check inside the agent’s own loop, as the one MongoDB’s orchestration post describes, has the same limit: it sees only the calls that go through that loop. The enforcement point sits outside the loop, so it also sees calls made by code the loop doesn’t control.

The path is only mandatory if agents hold no provider credentials and the execution environment controls its own egress. Different teams often own those two pieces of work. MongoDB’s security post already puts agent execution behind a default-deny network for containment, and the same control makes the path mandatory for cost. Without both, the enforcement point is advisory, and its numbers are incomplete in a way nobody can detect from inside it, because a call that went around it left no record there.

How do you measure what your enforcement point misses?

Take one task in your own system and list every action it took: model calls, tool calls, retrieval and memory reads, sub-agent spawns, and the model calls inside those sub-agents. If the run is already instrumented with OpenTelemetry GenAI spans, the trace gives you most of that list.

Compare it against what your enforcement point recorded for the same run, and mark which actions it observed. Then price the model calls it didn’t record, including the ones inside sub-agents, and add the recorded model calls you can’t tie back to the task. That total is the part of the invoice you can’t attribute.

The provider invoice gives you a check from the other direction. Divide the spend the enforcement point attributed to tasks by the invoiced spend for the same keys and billing period. A ratio well below one means some calls are going around the enforcement point or arriving without a task tag. The ratio doesn’t say which, so the per-task trace is still how you find the missing actions. Treat the ratio as an estimate: the invoice arrives weeks after the work and reflects negotiated rates and discounts that the enforcement point’s price table may not.

The trace works inside one organization’s agents. Spend attribution across agent-to-agent handoffs has no settled convention. The IETF’s Transaction Tokens draft, which reached working-group consensus in July 2026, carries identity and authorization context through a call chain inside one trust domain, and its extension to agents is still an individual draft. When a task crosses into another organization’s agents, no convention yet carries its identity or its budget with it.

Frequently asked questions

What is a mandatory path for agents?

A mandatory path is an agent architecture in which every action an agent takes, from model calls to tool calls, memory reads, and sub-agent handoffs, goes through one enforcement point. Permission and budget checks run before the action executes, and usage is reconciled afterward. No agent has a route around it.

Why do agent bills behave differently from API bills?

Agent spend scales with the steps a task takes. One task can produce a planning call, several model calls to evaluate what the tools returned, retries after tool failures, and delegations to sub-agents. Each model step is billed separately, often against different models at different prices.

Why can’t token volume show whether an agent is working hard or stuck?

Token count rises with loop depth. Loop depth rises when a task is hard, and also when a tool returns something the loop doesn’t handle and the agent retries or re-plans. A ranking by volume counts doesn’t distinguish between these two scenarios. Separating them takes per-task attribution.

Why does it matter where an agent’s provider credentials live?

Where a credential lives determines where accounting is possible. An agent holding its own provider key can call the provider directly, so no downstream report can say which task or requester produced the call. Tagging only works for calls that pass through the gateway.

Can an LLM proxy attribute agent spend per task?

Only for model calls that pass through it, and only if something tags them with a shared task ID. In the worked example, a model-only proxy sees six model calls and none of the five actions between them. Proxying MCP tool calls brings three of the five into view.

Why doesn’t an edge gateway see what an agent does?

An edge gateway authenticates and admits the inbound request at the perimeter, so it knows who started the task. It knows nothing about what the agent does next, including its tool calls, memory reads, and sub-agent handoffs.

How are cost attribution and agent authorization related?

They run on the same plumbing. A token issued when the task starts, then re-minted and audience-bound at each hop, carries the originating task and identity through the chain. That lets a downstream API enforce the requester’s permissions and lets the platform sum the task’s cost across hops.

Is there a standard for carrying cost attribution across agent hops?

No. The OpenTelemetry GenAI conventions (still in Development status) record token usage per call but define no project or cost-center attributes. The IETF Transaction Tokens draft carries identity through a call chain inside one trust domain. No convention currently carries a budget or cost attribution across hops.

What makes the path mandatory in practice?

Agents must hold no provider credentials, and the execution environment must control its own egress. An agent with a provider key and open outbound access can call the provider directly, and that call leaves no record at the enforcement point, so its totals look complete when they aren’t.

What does a mandatory path cost?

A mandatory path adds a network hop to every action, and the enforcement point becomes an availability dependency with a throughput ceiling. It also needs a failure policy. If it fails closed, its outage becomes an agent outage. If it fails open, actions run unpermitted and unmetered.

How do I measure what my enforcement point misses?

Trace one task, list its actions, and compare them with what the enforcement point recorded. Then divide attributed spend by invoiced spend for the same period. A ratio well below one means calls bypass the enforcement point or arrive untagged. Treat the ratio as an estimate, since invoices reflect negotiated rates.

Why You Can’t Answer “What Are We Spending on Inference?” was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.

In October 2024, six weeks into my tenure at MongoDB, I found myself in a meeting with fellow server and support engineers, drawing straws. The data growth in a large MongoDB Atlas cluster had been accelerating. Now, that is great news: this is the problem you strive to have! However, our back-of-the-envelope estimate for storage expansion came to several days. The desired answer would have been several minutes. The question on the table was who would go and tell this to the customer.

I’ve thought about that meeting a lot. The database was behaving as designed. Like every traditional database, standard MongoDB Atlas is architected with coupled storage: each database node has its storage tightly bound to its compute.

It was this coupling that was in the way, because storage has gravity.

Data doesn’t want to move. The bigger it gets, the harder it pulls down. Anyone who has managed a large database knows the symptoms: scaling a cluster, resharding, rebalancing, restoring from backup, and adding a read replica. They are slow. They are slow because they all reduce to the same storage-shaped primitive: copy a very large amount of data from one place to another. Nothing about better software makes a terabyte weigh less.

And because compute and storage are coupled, regardless of whether you need more storage or more compute, you get more of both.

So we ask the obvious question: if the problem at hand is tight coupling, then would decoupling be the answer?

Turns out the answer is yes. And that answer is Atlas Infinite, a new deployment option within MongoDB Atlas, launched in public preview on September 29th at MongoDB’s Investor Day. This post examines it in detail: what we set out to make possible; the architecture Atlas runs today and the disaggregated one we launched; the life of a write and of a read through the new machinery; what an append-only history buys you; what happens when things break; and the one line we would not cross to get any of that.

Decoupling is how a database stops being infrastructure you manage and starts being a capability you consume. But that’s the marketing promise — here’s the machinery that makes it real.

A peek under the hood

How do you move storage out from under a highly optimized, lock-free, parallel storage engine with non-deterministic, speculative eviction? Carefully.

Commercial databases built on decoupled compute and storage have existed for at least a decade. So it is reasonable to ask, “Why did it take MongoDB this long, and what’s new here?” The honest answer is that the properties that make our storage engine fast are exactly the ones that make it hard to pull apart — and that we refused to change it in ways that would give up the performance or the security our customers already had.

Besides, our customers have always had horizontal scale-out. When a workload outgrew a machine, sharding was the natural succession: one that several disaggregated architectures in the market lacked until recently. That freedom bought us the room to get this right rather than ship it first.

The challenge

WiredTiger™ is one of the most sophisticated storage engines in production. It earns its speed by deferring and batching work. It is built around checkpoint-based consistency: rather than logging each page modification, blow by blow, it lets writes accumulate in memory. A page can absorb many mutations and get written to disk just once, at a checkpoint. This amortizes the cost of turning writes into a readable state, and it is increasingly where the field is heading, with modern engines converging on copy-on-write, multi-version, checkpointed designs precisely because of their efficiency. Something WiredTiger has been doing for years.

The decade-old mainstream disaggregation designs work by shipping a physical redo log that is obligated to record every change to a page, so the shared storage tier can reconstruct an arbitrary page version. Adopt that directly, and you’d have traded WiredTiger’s core efficiency for the newfound elasticity.

Furthermore, conventional disaggregation relies on the shared storage layer to read, compose, and rewrite your data in the clear. That is something we rejected. In Atlas Infinite, if your data is unencrypted cleartext, it is not shared. And if it is shared, it sure ain’t clear. We don’t hold cleartext in a shared service. Period.

The final wrinkle: because WiredTiger is lock-free, parallel, and evicts pages speculatively, two nodes holding identical logical data will not hold identical bytes on disk. Layout depends on timing, eviction order, and allocation state, so there is no canonical page image to ship and no single layout for copies to converge to. “Replicate the pages” has no well-defined target.

This is where being MongoDB helped. We are the one company with its hands on the whole stack: the storage engine, the database server, and the cloud that runs it. We didn’t have to bolt disaggregation on from the outside and work around the engine — we could reach into the engine. This is what allowed us to solve something sharper than the textbook problem: keep WiredTiger’s deferred, batched writes intact, keep the storage layer completely blind to customer data, and still deliver read replicas that lag the primary by milliseconds, not by checkpoint intervals. None of the existing recipes delivers all three at once.

The rest of this series is about what we had to invent to get there. We’ll take a glimpse at MongoDB Atlas Core’s architecture, followed by an in-depth look at the Atlas Infinite design, service decomposition, write and read paths, as well as how they inform our security, durability, availability, and performance: SDAP — the yardsticks by which we measure everything we do. Subsequent posts will expand further into the security and durability aspects in particular.

Let’s start with a brief look at Atlas Core.

MongoDB Atlas Core: The attached-storage architecture

A MongoDB cluster is a replica set: three nodes, each a complete, self-sufficient machine. Each node holds a full copy of the data on its own attached disk. Each stays in step with the others by replicating the oplog: our logical operation log, from a primary to its secondaries. If the primary fails, a secondary is elected to replace it. Every node is doing every job: durability, availability, and serving queries, all bound together in the same box. It is shared-nothing. Its simplicity affords a near absolute security and isolation model: it simply relies on virtual private cloud (VPC) and virtual machine (VM) guaranteed isolation. One tenant’s actions cannot bleed into another.

But because compute and storage live on one machine, anything you want to do to the data, you have to do to the machine. Scaling reads means cloning an entire node before it can answer a query. High availability means paying for full physical copies that mostly stand and wait. Growing a node means rebuilding it and warming a cold cache. The design is simple, robust, and cleanly isolated tenant-from-tenant, and it makes every storage-shaped operation an act of moving the whole machine.

Atlas Infinite: The disaggregated architecture

We took a design so simple — three nodes in a replica set — and replaced it with at least fourteen! Database nodes, log nodes, page materializers, and page serving nodes. That sounds like the wrong direction, until you notice who runs them: not you. The cluster goes from three machines doing every job to fourteen sharing the work — extra hands, not extra burden. Managing data is hard. But done right, managing a million times more data is not a million times harder. Disaggregation splits the architecture into two halves.

The compute layer runs the database duties: parsing queries, running transactions, and handling aggregations. The change is that it becomes ephemeral and without storage: it becomes weightless. It offloads its other duties (oplog replication, consensus, page copying, and backups) to the storage layer. In place of Atlas Core’s three full replicas, Atlas Infinite runs a primary database node, a standby node for high availability, and optionally, additional read replicas for read scaling. Just as with Atlas Core, the compute layer runs isolated in its own VPC.

The storage layer is made up of multi-tenant storage layer services, responsible for managing storage for that “million times more data.” The machinery here is sophisticated, as we’ll see. But none of it is yours to run. From the customer’s side of the glass, storage is simply managed by MongoDB.

This layer consists of three main services: the Log Service, the Page Service, and the Object Index Service, each able to scale, fail, and recover independently. The entire storage layer was purpose-built from the ground up in Rust.

The Log Service is the shortest path to durability. Every write becomes an append to a replicated log. The service does exactly one thing: reach consensus on the log. And a service that does exactly one thing can do it with verifiable correctness, higher performance, and a tighter tail latency than one juggling durability alongside the full duties of a database.

The Page Service is a pool of page servers holding the materialized pages in each zone, serving page reads. A new compute node reads from a cache that is already warm instead of cloning a disk of its own. Every log’s pages are distributed across many page servers in each zone, so a read spike is absorbed across many instead of hammering one. This is the moment the fleet can finally bring more than one node’s worth of muscle to bear on a single customer’s workload.

A Page Materializer reads entries being written to the log and distributes them to the page server group responsible for that log. In parallel, it does one more thing: it continuously spools the log onto object storage.

Object Index Service and Object Storage — S3, Google Cloud Storage, Microsoft Azure Blob — sits underneath as the durability floor: designed to provide eleven nines, continuously maintained, the bottom of the world. Object storage is organized and managed by an Object Index Service, and access is mediated by an Object Read Proxy, which can answer the same shape of read queries as a page server — except from object storage. This is what allows us to deliver continuous data protection, time travel, and fast backups and restores.

Through all this, the front door does not move. Same wire protocol, same drivers, same read and write semantics. The application cannot tell the difference. Everything that changed is underneath it.

Control plane and heat management

I won’t get into the details in this blog, but suffice it to say that the storage layer services are all provisioned, monitored, and heat-managed by control plane services that ensure automated fault tolerance and recovery as well as load spreading and tail-at-scale operations.

Scale anything independently

A quick reality check: if compute and storage are really separate, three operations that were expensive in the coupled world should become cheap without special-casing. Let’s have a look.

Adding a read replica is zero-copy. In MongoDB Atlas Core, a secondary needs to clone a full copy of the disk before it can answer a single query, which is where the wait comes from. In Atlas Infinite, the data was never on the node to begin with. A new compute node starts up, tails the committed oplog from the Log Service to catch up to the present, and reads its pages from the shared Page Service, whose cache is already warm. Its cost is bounded by the length of the oplog tail rather than the size of the database, so it is serving reads in seconds regardless of how large that database is.

Vertical scaling swaps the compute without touching storage. In Atlas Core, resizing a node means standing up a bigger machine and then waiting out a cold cache, because both the cache and the data live on the machine you are replacing. In Atlas Infinite, they don’t. A larger compute node launches, takes over as primary, and the old one steps aside. Nothing moves, and nothing has to re-warm, and the swap closes in under a minute.

Storage performance scales on its own axis. With attached storage in Atlas Core, IOPS are a property of the attached disk. In Atlas Infinite, IOPS scale with the storage pool rather than with any one disk. If a workload is storage-bound, you request more throughput or more IOPS, and the storage layer grants it, with no need to touch compute at all.

The common thread, and the real test of the design, is that each of these touches only the layer that has to change, and its cost scales with the size of that change rather than the size of the database.

A deeper dive into the write path

Let us take a closer look at the life of a single write. With consensus and replication delegated to the Log Service, and the separation of durability from readability, there is a lot happening on the write path.

A write arrives at the primary. The first thing that happens, as always, is that the primary appends an entry to the oplog describing what changed. That record goes to the Log Service, which replicates it across the three zones, and the write is acknowledged the instant a majority of the log replicas hold it durably.

As the storage engine settles its pages in the ordinary course of checkpointing, it emits a second, physical log we call the phylog. Where the oplog says what changed in logical terms, the phylog carries the physical manifestation of the change: page images, and far more often, page deltas.

The standby and read replicas are clients of the Log Service and tail the log. At the same time the page materializer — one in each zone — tails the log as well, and distributes the phylog records across the page servers.

The page servers receive the phylog records, organize images and deltas by page, and store them on fast, directly attached drives.

In parallel, the page materializer uploads both the oplog and the phylog to object storage.

Notice the shape of everything downstream of the primary: logs — append-structured, versioned, immutable. Hold on to that; it is about to pay off.

The read path and solving standby lag: Layered tables

The read path on the primary is … uninteresting, in that it is substantially the same as before. On the standby and read replicas, it gets interesting.

Checkpoint-based consistency in WiredTiger means that durable page state only moves forward at checkpoint boundaries. That’s bad news for a standby or read replica that reads only materialized pages: it would trail the primary by as much as a full checkpoint interval: multiple seconds. That is a deal-breaker for a read replica serving reads. We wanted replica lag measured in milliseconds.

We accomplish this by layering two tables. Underneath sits the stable table: the checkpoint-consistent, shared, materialized data, the same page state the primary authors as it checkpoints. On top of it sits an ingest table (more a table cloth, really), into which the standby applies the oplog as it arrives. The ingest table holds the recent tail of changes — the writes that have landed since the last checkpoint the standby adopted. As newer checkpoints arrive and the stable table catches up, the standby garbage-collects the now-redundant entries from its ingest table, keeping the tail bounded.

A read composes both. The stable table supplies everything settled as of the last checkpoint. The ingest table supplies the fresh changes on top. Together, they give the standby a current, consistent view without waiting for the primary to publish its next checkpoint. That is what turns standby lag from a checkpoint-interval problem into an oplog-replication-lag one, measured in milliseconds.

Branching logs and ancestry

At the end of the write path I asked you to hold on to one property: everything downstream of the primary is a log, append-structured and versioned, immutable. Here is where that pays off.

When history is an append-only log, the storage-shaped operations a database dreads most no longer require moving data at all. A snapshot, a clone, a branch, a rewind, a restore: in the coupled world, every one of these is a copy. Over a versioned log, they are metadata. You are naming a point in history that already exists. Constant time, zero copy, independent of how large the database is.

Start with restoring a backup. A restore creates a new log derived from an existing one, branching off at the chosen point in time and proceeding independently from there. The original keeps going untouched; the restored copy is its own independent timeline from that instant forward. Do this a few times, and the logs form a lineage. We call it log ancestry: an ancestry tree rooted at the original log, each restore a new branch hanging off the point it was taken from.

Restore is not special. It is just one use of a more general primitive: take any log, mark a point, and fork. A snapshot is a marker on the timeline, nothing copied, just a named instant you can return to. A clone, or a writable snapshot, is a fork in that timeline: a branch that shares all history up to the fork point and then diverges.

Atlas Infinite sets the foundation for a production database you can branch like a Git repository: to test a migration against real data and throw the branch away. A rewind to any moment inside a retention window. A full-fidelity copy in every engineer’s harness, costing almost nothing to create because it copies almost nothing.

When things break

Drama, they say, needs three things: a villain, a victim, and a hero. A failing database has the first two in abundance. The villain is failure itself, in all its usual garbs: a node that dies mid-write, a network that partitions, a disk that goes bad, an availability zone that drops off the map. The victims are the machines snared. With Atlas Infinite, there is no hero in operations. We designed the system so the situation never rises to the level of needing one. Nothing rescues the database, because nothing put it in peril.

If a database primary fails, the standby is hot and ready to take over. If the network partitions, the Log Service mediates consensus, and the storage layer control plane may even be able to heal the raft group. If a log server fails, a new one gets commissioned, while writes never stop. If a page server fails, there are others with staged copies of the data. If a disk fails, there are other copies. If an entire availability zone goes dark, the cluster keeps serving: the log commits on a majority spread across three zones, each zone has its own page servers, and object storage that sits beneath it all, is zone failure-proof as well. The durability and availability guarantees stem from data that is held in three independent forms at once: the replicated log, the materialized pages, and the object store. And since the storage layer is multitenant, most recovery parallelizes many-to-many: many machines pitch in to take over the work of the departed machine, and to help a new node get up to speed.

That non-event is the visible tip of a great deal of deliberately unglamorous work that happens behind the scenes. The fleet is carved into cells, spread across availability zones, so trouble in one cell is contained and cannot cascade into the next. The converse also holds: a cell is designed to insulate itself from most trouble happening outside it. We call it Anti-Vegas: what happens outside a cell, stays outside a cell.

A resource scheduler watches the heatmap continuously, a tenant or a shard drawing more than its share, a node running over the utilization threshold, resource pressures of any kind, and moves load before anyone becomes a noisy neighbor, or feels one. Storage volumes are checked for integrity and repaired, with nothing visible from above.

None of this is drama. Boring is the highest compliment you can pay infrastructure. It means the system runs predictably and safely through spikes and failures and the ordinary daily chaos of a very large fleet, and asks our customers to think about none of it.

The line we wouldn’t cross

Every disaggregated database has to answer one question about its shared storage layer: can that shared layer read your data? For most of them, the answer is yes: a natural consequence of what conventional disaggregation demands of storage: to materialize pages from a redo log.

The database ships a log of changes, and the storage fleet reconstructs pages by interpreting that log: find the page, apply the change, produce the bytes. This composition has a consequence that no amount of engineering can wish away. To interpret data, you must be able to read it. A storage tier that reconstructs your pages is, by definition, a storage tier that can see your pages.

The defenses built around that fact tend to be operational. Mandatory access controls, sacrificial nodes, scoped credentials, periodic audits. In other words, part of the security promise is “trust us.”

In contrast, we wanted to say: trust our algorithm, not us. In Atlas Infinite, the shared storage layer is not semantically aware — on purpose. Your data is encrypted on the compute node, at the database level, before a single byte is handed down. It is decrypted in exactly one place: the single-tenant compute node running inside your own isolated network, using keys that can be yours to hold, rotate, and revoke. The storage layer services cannot read your documents. They cannot see your collection names, your indexes, or where one tenant’s data ends and the next begins. An operator with root on the entire storage fleet would hold nothing but opaque bytes.

That is a different kind of boundary. We augmented the full network isolation of Atlas Core, with cryptographic segmentation. This is the reason we could take a system that had always been isolated by machine and network boundaries and put it on shared storage without compromising its trustworthiness.

Making an entire storage fleet operate, heal, compact, and garbage-collect data it cannot understand is genuinely hard, and it is where a lot of the real invention in this project lives. How we did it, and the tenant-isolation guardrails around it, is the subject of the next post in this series.

Epilogue

I studied databases at the University of Wisconsin–Madison, back when the field was still arguing about B-trees and data layouts. Then I wandered off for twenty-five years: high-performance computing, networks, security, and a long stretch building load balancers, whose entire job is to stand in front of machines that fail and make them, together, into something that doesn’t.

I look at MongoDB Atlas Infinite through that lens. We stood a service in front of the database’s storage and made it something that is designed not to fail, not to run out, and not to require trust. And the old question of how data should sit on your disk got a new answer: it shouldn’t sit on your disk at all.

Storage had gravity. Less so now.

Atlas Infinite is in public preview. This post is the first of three. Next, my colleagues will show how a storage fleet operates, heals, and compacts data it cannot understand. After that, we open up mongod itself: what it took to make a database engine born in the era of attached disks feel at home in a world without them.

If you’ve read this far, you’re who we built it for. Come kick the tires. Tell us how you experience it.

Get started with these new capabilities today for free at mongodb.com/atlas.

To learn more about Atlas Infinite and MongoDB 9.0, check out the Atlas Infinite documentation, version release page, or the MongoDB 9.0 documentation.

The Ground Beneath the Database was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most agent-memory posts are about the memory algorithm. This one is about the storage layer underneath it, because that’s where I’ve watched these systems actually fall over.

The pattern is familiar by now. An agent accumulates conversation history, you can’t replay all of it into every call, so you compress it into some smaller representation and retrieve the relevant parts on demand. Plenty of libraries do the compression well. What they mostly hand-wave is what happens next: the compressed state has to live somewhere, and that somewhere has to survive a crash, resist tampering, serve hybrid queries fast, and delete on request in a way you can prove to an auditor.

Those four requirements turn out to be a fairly demanding database spec. I spent a couple of days testing them against MongoDB using Semvec, a memory engine that keeps a fixed-size state vector plus bounded memory tiers, with an Atlas persistence layer on top. Everything below is measured on a laptop against mongodb/mongodb-atlas-local 8.3.8, and you can reproduce it.

Requirement 1: the write has to be atomic across three shapes of data

Every turn produces three writes that have to agree with each other:

If those three drift apart, the memory is corrupt in ways that are hard to detect. An event logged without its items means a fact silently vanishes from retrieval. A snapshot written from a state that doesn’t match the log means recovery restores the wrong thing.

This requirement quietly rules out most of the standard stack. If your log is in Postgres and your vectors are in a dedicated vector store, there is no transaction boundary around those three writes, so you reach for an outbox table or a saga and accept a window of inconsistency. In MongoDB, there are three collections in one database, and the commit is one multi-document transaction:

committer = TurnCommitter(client, db, scope, embedder=embedder, snapshot_every=10)
committer.run_turn(state, session="s1", turn=1, text="customer prefers index funds")

turn 04 committed | event_seq=4 | items +1 | dedup.is_update=False | snapshot ✓

Measured cost of that atomicity: p50 21.7 ms, p95 47.2 ms. For context, generating the embedding for the same turn took p50 81.8 ms on CPU. The transaction is not what you’ll be optimizing.

event_seq is monotonic per tenant and agent, so gaps are detectable rather than silent. That matters for the next requirement.

Requirement 2: You have to be able to rebuild state and prove it’s the same state

“Durable” usually means “we back it up.” For agent memory that’s too weak. If you restore a snapshot and the state is subtly different from what the agent had before, the agent’s behavior changes, and nothing tells you.

Because every turn is in an append-only log, recovery can replay the log from scratch and compare:

recovered: replayed_events=10 (mode=full replay, log is the source of truth)
engine checksum (recovered): a8d9d3b4ffa8adf3… == live ✓

Byte-identical, not approximately equal. That’s what lets you treat the snapshots collection as a disposable cache: if a snapshot is missing or suspect, throw it away and replay. The same property held across the LOCOMO benchmark, where checksums matched on 10 of 10 conversations and the run is set to abort on the first mismatch.

Two MongoDB details make this practical rather than theoretical. Snapshots are compare-and-swap versioned, so two processes racing to checkpoint the same session can’t silently overwrite each other. And integrity is checked twice on read, once against a SHA-256 of the blob and once against the engine’s own embedded checksum, which distinguishes storage corruption from a version mismatch.

Requirement 3: retrieval has to be hybrid, and it should happen server-side

Semantic search alone doesn’t work for this workload. Ask an agent, “What’s my policy number?” and vector similarity gives you insurance-adjacent chunks. You need the lexical match too, plus a recency bias, because in a conversation, the newer statement usually supersedes the older one.

MongoDB 8.1 added $rankFusion, which does that fusion inside a single aggregation:

retrieval path: $rankFusion (server-side, MongoDB 8.3)

One round trip, p50 15.2 ms, with per-tenant weights over the semantic, lexical, and recency rankings. The alternative is three queries and a reciprocal-rank-fusion loop in your application, which is more code, more latency, and a reranking implementation you now own.

Worth checking your server version before you count on it. My cloud test cluster was on 8.0.30, which predates $rankFusion; the library detects this and falls back to client-side fusion, so it keeps working, but you lose the server-side path without an error telling you so.

In the benchmark, hybrid retrieval through Atlas scored 0.5773 on LLM-judge accuracy, compared to 0.5322 for the same engine retrieving in-process with no database. LLM-judge accuracy here means an LLM grades each answer against ground truth and you take the pass rate using the mem0 evaluation prompts verbatim. Persisting to a database and querying it did not cost accuracy; it gained about 4.5 points, because fused ranking beats the in-process scorer. Retrieval latency went from 165 ms to 187 ms at p50 to get that.

Reader context also stayed flat while this happened, which I’ll come back to at the end. Fair warning on the benchmark: those accuracy figures aren’t comparable to published mem0 or Zep numbers, because this run doesn’t replicate their prompt routing or entity extraction. The comparison that’s valid is Atlas-backed against in-process under identical conditions.

Requirement 4: deletion has to be provable, and it reaches further than you think

This is the requirement I’d underestimated, and it’s the one I’d now lead with when talking to anyone in a regulated industry.

A GDPR Article 17 request means a user’s data must be gone, and “gone” has to be demonstrable. In this implementation, the erasure produces a signed certificate:

certificate: user=<tenant>/<agent> events_wiped=16
signature (RS256): jRKdVEuIbnDfuT5KZwMSE6Hy07RzASc0…
signature verifies: ✓
memory_items left: 0 | snapshots left: 0
quarantine vault left: 0 | learned anchors left: 0
retrieval after forget: 0 hits (expected 0)

Note the two lines before the retrieval check, because they’re the interesting failure mode. This system has an ingest filter that blocks suspicious text before it reaches memory, and blocked text is held in a quarantine collection for human review. It also derives embeddings from blocked text to recognize future variants.

Both of those contain user data. The quarantine vault holds the full original text. The learned anchors are vectors derived from it. An implementation that deletes memory_items and calls the job done has left personal data sitting in its own safety subsystem, which, under Article 17, is not a technicality. Getting this right requires knowing every derived copy, and it’s much easier to enumerate those when they’re collections in one database rather than artifacts spread across four services.

A word on what the signature does and doesn’t prove. The RS256 key is yours: the operator supplies the private key, and the certificate is signed after every wipe step has completed and been counted. So the signature proves the deletion ran to completion under your key and hasn’t been altered since. It is not third-party attestation; if you need an external party to vouch for it, you hand them the certificate and your public key. That’s the same trust model as any self-signed audit record, and it’s worth being upfront about before an auditor asks.

Same argument in general form: the moment your source of truth and your vector index live in separate systems, “delete this user” becomes a distributed systems problem with no transaction boundary and no single place to verify the result.

The security layer, briefly

The ingest filter deserves a note because it’s the part I expected to be hand-wavy, but it wasn’t.

The threat is specific to persistent memory. A one-shot prompt injection is a bad answer, and then it’s over. An injection that gets written into memory is retrieved and re-injected on every future turn, and it will be retrieved precisely because it’s semantically relevant to whatever the attacker aimed it at.

The filter is cosine similarity against a handful of anchor sentences describing what must never enter memory, with a 0.45 threshold. A direct injection attempt scored 0.696 and was blocked. A leetspeak version of the same attack (1gn0r3 4ll pr3v10u5 1n57ruc710n5) scored 0.298 and passed, because the embedder sees gibberish tokens. So there’s a de-obfuscation pass that normalizes character substitutions and homoglyphs and re-checks: 0.633, blocked. I appreciated that the demo shows its first line of defense failing rather than only the version that works.

The part with operational teeth: blocked text is quarantined, not dropped, and the review queue shows only a hash, a 15-character preview, and the score, so an operator can triage without reading customer data. Releases require a reason. Legitimate text that trips the filter (one test case scored 0.632, phrased like an injection but entirely benign) can be released and recommitted. Cosine similarity can’t read intent, so false positives are a certainty, and the workflow assumes them.

Every block also teaches. The embedding of a blocked attempt is stored as a learned anchor, and those anchors live in a collection shared across the whole tenant rather than per agent. In the demo, the attack is blocked at advisor-01. A reworded variant is then sent to advisor-02, an agent with no seed catalog of its own and no history with this attacker, and it’s blocked at 0.938 against the anchor advisor-01 learned. One agent gets hit, the fleet is immune. Releasing a false positive un-learns its anchor, so the operator workflow above doesn’t poison the shared list. I haven’t seen another memory layer demonstrate this, and it’s only trivial because the anchors are documents in the same database every agent already reads: no sync job, no cache invalidation, no second system.

To be clear about scope: this catches the obvious class of attack, the kind that reads like an instruction to the model. It is not a defense against an adversarially optimized payload. An injection phrased to sound like a benign customer statement won’t resemble any anchor and will pass, for the same reason the leetspeak variant passed before de-obfuscation. It raises the cost of the lazy attack and leaves an audit trail, and that’s what it claims to do.

Blocked attempts leave the engine state bit-identical to a run where the attack never happened, while still writing content-free audit markers recording that something was blocked. Audit trail without storing the payload.

Reproducing it

docker run -d --name semvec-atlas-mongo -p 27017:27017 \
    --ulimit nofile=64000:64000 mongodb/mongodb-atlas-local:8.3.8latest

python3 -m venv ~/.venvs/semvec-atlas
~/.venvs/semvec-atlas/bin/pip install -e ".[langgraph,compliance,mcp,dev]" \
    sentence-transformers openai tiktoken

demo/run.sh --no-llm

The — ulimit flag is not decoration. Each test provisions its own database, and a few hundred of those exhausted the default file-descriptor limit, at which point WiredTiger takes a fatal assertion and mongod dies mid-run. What you see is a wall of OSError: connection closed failures that look like application bugs. With the limit raised, the suite goes from a 25-minute partial collapse to 41 seconds.

Everything except the answer text is deterministic, so the numbers above reproduce exactly rather than approximately.

Why you’d want this

If you’re building agents, you’ve already hit the wall, or you’re about to: conversation history grows, context costs grow with it, and every turn gets more expensive than the last. The usual fix is to compress. The part nobody tells you is that compression only pays off if the compressed state is somewhere you can trust.

Here’s what that looked like in practice. Across 5,882 turns of real conversation, the context sent to the model per call stayed at 987 tokens. That held on the 689-turn conversation just like the 369-turn one. Replaying the full history would have cost 27,353. So: 96% less context, and it doesn’t drift upward as the conversation gets longer. Turn 700 costs what turn 7 did.

And it didn’t cost quality. LLM-judge accuracy was 0.5773 going through Atlas against 0.5322 for the same engine holding everything in memory with no database at all. Persisting to a database didn’t tax accuracy, it improved it, because fused ranking beats an in-process scorer.

Fixed cost per turn, and accuracy that holds. That’s what makes an agent affordable to run at turn 1,000 rather than just impressive in a demo.

What MongoDB is doing for this

Fixed-size memory isn’t a new idea. What’s different here is not having to assemble the storage underneath it from four separate services.

Everything in this post ran on one database:

• The turn log, the state snapshot, and the vectors commit together in one transaction, 21.7 ms at p50. No outbox, no saga, no window where a fact exists in your log but not your index.
• Hybrid ranking over semantic, lexical, and recency signals happens server-side in one aggregation via $rankFusion, 15.2 ms at p50. Not three queries and a fusion loop you have to write and own.
• “Delete this user” is one operation you can verify, with a signed certificate, because there is one place the data lives.

Try that on Postgres plus a dedicated vector store, and every one of those becomes distributed-systems work: consistency code you maintain, reranking you implement, and a GDPR request that turns into an archaeology project across four services. The alternative to one database isn’t four databases, it’s four databases plus the glue between them and the compliance gaps in the seams.

The whole thing runs on your laptop today: mongodb/mongodb-atlas-local, one docker run, same code path as Atlas in production.

One last data point for anyone about to optimize the wrong layer. The embedder was 4x the cost of the database write and 5x the cost of retrieval. The database was never the bottleneck.

What agent memory actually asks of a database was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.

What’s This All About? 🤔

In the previous episode, we focused on building an amazingly tiny-but-powerful MongoDB cluster on four RK1 modules. Some basic load tests have been done, but with no specific focus on the behavior of our beloved MongoDB engine. Now is the time to dive a bit deeper into this topic. Special thanks to #Frederic Favelin for helping me tackle the subject in depth 🙂

MongoDB memory-management deep dive

This part separates mechanism from measurement. It documents the kernel and allocator settings, the WiredTiger working sets, the controlled THP ON/OFF procedure, and results from the 500k and 50M documents datasets. Claims about causal THP effects are limited to what the repeated application-level measurements and telemetry support offer.

THP and TCMalloc deployment prerequisites

Enabling Transparent Huge Pages on RK1 requires a kernel compiled with the appropriate option. The stock kernel does not expose the THP interface, so the custom kernel packages must be installed before enabling THP through Ansible. I hadn’t compiled a Linux kernel for several decades, but the process has not changed globally. Node 4 is used to get it compiled properly (compilation takes less than 10 minutes), and then it gets distributed using an Ansible playbook.

setup_rk1cluster.yml installs the two custom kernel packages and holds them so the package upgrade cannot replace them. Both .deb files must be present beside the playbook before running the setup.

The THP configuration needed is achieved by using the related Ansible playbooks as follows.

# MongoDB 8 on ARM64: enable THP and upgraded TCMalloc per-CPU caches
ansible-playbook --ask-vault-pass -i inventory.yml thp_enable.yml

# Optional A/B comparison: disable THP and restart MongoDB one node at a time
ansible-playbook --ask-vault-pass -i inventory.yml thp_disable.yml

Verify the current situation at any time:

# MongoDB 8 THP values: always, defer+madvise, 0, and 1
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
cat /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none
cat /proc/sys/vm/overcommit_memory

MongoDB 8 uses upgraded TCMalloc on supported ARM64 Linux systems. The deployment enables THP before Docker starts MongoDB, sets the required always, defer+madvise, max_ptes_none=0, and vm.overcommit_memory=1 values, and sets GLIBC_TUNABLES=glibc.pthread.rseq=0 in the container so TCMalloc can use per-CPU caches. The thp_enable.yml and thp_disable.yml playbooks provide a serial, restart-and-report A/B comparison for benchmarks. Run the enable playbook again after comparison benchmarks to restore the MongoDB 8-recommended configuration.

The RK1 kernel must expose /sys/kernel/mm/transparent_hugepage with CONFIG_TRANSPARENT_HUGEPAGE enabled. The playbooks cannot enable THP when the kernel omits that interface; install or boot a kernel built with that option first. Check the active kernel with:

uname -r
zgrep CONFIG_TRANSPARENT_HUGEPAGE /proc/config.gz 2>/dev/null || \
  grep CONFIG_TRANSPARENT_HUGEPAGE "/boot/config-$(uname -r)"
ls -ld /sys/kernel/mm/*transparent_hugepage

Controlled THP evaluation on ARM64

Starting with MongoDB 8, Transparent Huge Pages (THP) are recommended on supported Linux systems. Because the stock RK1 kernel does not expose the required THP interface, this cluster uses a custom kernel built with CONFIG_TRANSPARENT_HUGEPAGE=y.

The objective of this benchmark is to determine whether MongoDB’s use of THP produces a measurable application-level benefit on the RK3588 ARM64 platform. Two complementary datasets are used:

The tests compare identical MongoDB, Docker, YCSB, storage, and hardware configurations. Only the host THP state changes. Docker does not isolate a container from the host kernel memory management, so THP applies directly to the containerized mongod process. The MongoDB containers have no CPU, memory, or CPU-affinity limits, and /data/db is a host bind mount rather than Docker’s writable overlay filesystem.

Test Environment

Memory-stack interpretation: THP, TCMalloc, and WiredTiger

The three layers answer different questions. THP is a host-kernel page mapping feature; TCMalloc manages mongod’s user-space allocations; and WiredTiger manages database pages in its own cache. Seeing activity in one layer does not establish a throughput benefit in another.

The dashboard is interpreted as follows:

• In the 500k campaign, host AnonHugePages grows to roughly 4–5 GiB under THP ON, while cache fill rises over time without the sustained high-rate NVMe reads seen in the 50M campaign. The TCMalloc free/unmapped and cache panels evolve during both campaigns; they indicate changes in allocator state, not a standalone THP performance result.
• In the 50M campaign, THP-backed anonymous memory grows further, to roughly 6–7 GiB per data node. The WiredTiger cache is around 80% full, with sustained reads, eviction, and dirty-page cycles. These are distinct sources of latency variability that must be considered alongside the YCSB numbers.

The THP dashboard counters and AnonHugePages are host measurements. A process-specific smaps_rollup/smaps check for mongod would strengthen the attribution of huge-page coverage to MongoDB. No separate TCMalloc ON/OFF or WiredTiger-cache-size A/B test was performed here; THP is the controlled variable, and the other two subsystems provide explanatory telemetry.

Workload and Measurement Method

Both datasets use the same mixed YCSB workload:

• 60% READ, 25% UPDATE, and 15% SCAN
• Zipfian request distribution
• uniform scan-length distribution
• readPreference=primary
• directConnection=true
• 4, 8, 16, 32, and 64 YCSB client threads
• 600 seconds per concurrency level

The configured operation count is deliberately much larger than can be completed during a run. The playbook uses an external timeout to terminate each test after 600 seconds, making the comparison time-bound instead of operation-count-bound. The go-ycsb limitation and shutdown behavior are documented in Part I gotchas.

Dataset creation is separate from the measured workload. Loading uses 32 threads and writeConcern=1 to reduce preparation time; measured runs use writeConcern=majority. The load phase is not included in the benchmark results.

Benchmark Procedure

1. Enable THP and reboot if requested:

  ansible-playbook --ask-vault-pass -i inventory.yml thp_enable.yml

2. Confirm the reported THP settings, then run the benchmark and give it a descriptive prefix for easy analysis:

ansible-playbook --ask-vault-pass -i inventory.yml ycsb_cache_pressure_test.yml \
    -e confirm_unauthenticated_ycsb=true \
    -e load_dataset=true \
    -e file_prefix=thpon

3. Disable THP and reboot if requested:

  ansible-playbook --ask-vault-pass -i inventory.yml thp_disable.yml

4. Confirm the reported THP settings and repeat the benchmark with a different prefix:

ansible-playbook --ask-vault-pass -i inventory.yml ycsb_cache_pressure_test.yml \
    -e confirm_unauthenticated_ycsb=true \
    -e file_prefix=thpoff

The dataset load is opt-in. Load it once before the first comparison run with -e load_dataset=true; subsequent runs reuse the existing dataset and should leave that option unset. Loading uses 32 client threads and primary-only write acknowledgment for faster preparation. The measured workload uses majority write concern and direct primary connections.

For every reuse run, pass the intended ycsb_record_count and database explicitly (for example, -e ycsb_database=ycsbsmall -e ycsb_record_count=500000). The protected playbook compares the actual collection document count with the configured record count before the timed thread sweep and refuses to run on a mismatch. The generated summary CSVs still omit READ error counters, so raw thread logs remain necessary for a final request-validity audit. The record-count failure that motivated this safeguard is described in the Part I gotchas.

Playbook Parameters

The main parameters are defined near the top of ycsb_cache_pressure_test.yml and can be overridden with -e:

The workload is read/update/scan only: 60% reads, 25% updates, and 15% scans.

Inserts are disabled when reusing a populated dataset, avoiding duplicate-key errors. The measured runs use mongodb.write_concern=majority and mongodb.read_preference=primary.

Generated Results

Results are written to ycsb_results/ on the control node. For a prefix of nothp, the playbook creates one raw file per thread level:

ycsb_results/nothp-threads-4.txt
ycsb_results/nothp-threads-8.txt
ycsb_results/nothp-threads-16.txt
ycsb_results/nothp-threads-32.txt
ycsb_results/nothp-threads-64.txt

When load_dataset=true, the loader output is also saved as nothp-load.txt. The thread files contain YCSB throughput, operation counts, average latency, and percentile latency for READ, UPDATE, SCAN, and TOTAL.

Python Report Generator

ycsb_report.py parses the raw thread files and generates an aggregate CSV and PNG chart. Matplotlib is installed by setup_facility-node.s as python3-matplotlib.

The playbook runs the report generator automatically. To regenerate a report manually or process an existing run:

python3 ycsb_report.py \
  --results-dir ycsb_results \
  --prefix nothp \
  --output-prefix nothp

This creates ycsb_result/nothp-summary.csv and ycsb_results/nothp-report.png. The chart contains throughput, average latency, p95 latency, and operation-count panels plotted against client threads. The CSV is intended for further comparison or archival.

Small Dataset: 500,000 Documents

The 500,000-document campaign contains four repetitions under each THP setting, starting with THP off. The load logs confirm recordcount=500000 and report 500,000 inserts. Run 1 is excluded consistently as the post-load transient/warm-up repetition; the tables use the arithmetic mean of the final CSV totals from runs 2–4.

THP effects on throughput are mixed in sign and within roughly 0.5% across these stabilized repetitions. Neither average nor P95 total latency shows a consistent THP advantage. In Figure 1, THP-backed anonymous memory increases to roughly 4–5 GiB on the enabled nodes; allocation and collapse activity appear after THP is switched on, while flow control remains inactive. THP is thus visibly in use, but not associated with a material YCSB performance gain in this campaign.

The summary CSVs contain completed-operation throughput and latency, not READ success/error counters. The load size and configured recordcount are aligned, but a complete request-validity audit requires the raw YCSB thread logs. Error-free READs should be confirmed there before treating these numbers as final.

Sixteen threads is the practical throughput/latency sweet spot for this small-dataset workload. Averaging the THP OFF and ON results exposes the cost of pushing concurrency past the knee:

From 16 to 32 threads, throughput rises by only 3.8% while mean latency almost doubles; at 64 threads, throughput slips slightly below the 32-thread level while mean latency nearly quadruples relative to 16 threads. P95 also rises by roughly 2.4x and 5.8x, respectively. The 32 and 64-thread settings are poor latency trade-offs for this specific small-dataset profile.

Large Dataset: 50 Million Documents

The large dataset is intentionally much greater than the effective WiredTiger cache. It creates sustained cache churn and exercises the full path through WiredTiger eviction, NVMe I/O, dirty-page processing, checkpointing, and replica-set acknowledgment.

The table below uses the arithmetic mean of the final YCSB totals from repeated runs 2–4. Run 1 is consistently excluded as the transient/warm-up repetition. Latency values are the total operation latency.

Unlike the small dataset, the repeated large-dataset results show a modest THP advantage at every tested concurrency level. The largest mean throughput gain is approximately 7.1% at 32 threads, while average and P95 latency are also lower with THP enabled at all five thread counts. The gain is not linear, so it should be interpreted as a workload-level effect rather than a fixed THP improvement factor.

The dashboard confirms that this is a genuine cache-pressure workload:

• WiredTiger cache remains around 80% full on the active data nodes.
• Eviction is sustained throughout the campaigns;
• NVMe reads frequently reach approximately 150–200 MiB/s;
• dirty-cache ratios rise and fall with checkpoint and writeback cycles;
• Replication lag adds occasional latency variability.

Flow control requires a separate interpretation. The flow-control activity in Figure 2 occurs only during the 50M-document loading step, when documents are initially inserted. It is absent from the measured 4/8/16/32/64-thread YCSB sweeps. Flow control is therefore evidence of pressure during dataset creation, but it is not a limiting factor in the reported benchmark results.

THP use is also much greater than in the small experiment. The MongoDB nodes reach approximately 6–7 GiB of AnonHugePages each. THP allocations are continuous, fallbacks remain rare, and collapse/split activity does not show a pathological storm. Docker, therefore, does not prevent THP from operating, and the measured benefit is not accompanied by an obvious THP allocation failure or compaction problem.

The 50M-record workload continues to gain throughput through 64 client threads, unlike the small workload. That gain comes with rapidly increasing latency. The following table averages THP ON and OFF values to isolate the concurrency trade-off from the THP comparison:

At 32 and especially 64 threads, average latency rises much faster in absolute terms, and P95 latency reaches roughly 494–536 ms and 671–718 ms, respectively. Those settings remain useful when batch throughput is the priority, but they are no longer the most balanced choice. Sixteen threads is a reasonable latency-conscious operating point for this large-dataset profile, but not its sweet spot: unlike the small dataset, 32 and 64 threads provide substantial additional throughput. Select those higher settings when the extra completed work justifies the measured latency cost.

Global Conclusion

Together, the two datasets show that MongoDB 8.3.8 successfully and extensively uses Transparent Huge Pages on RK1/RK3588 ARM64, including when mongod runs inside Docker. Allocation fallbacks remain very low, so the comparison is not between an effective THP OFF configuration and a nominal-but-nonfunctional THP ON configuration.

The performance effect depends on the working set:

• With the 500,000-document workload, THP ON and OFF differ by less than approximately 0.5% in mean throughput across the tested thread counts. The 16-thread knee is clear: moving to 32 or 64 adds only 3.8% or 3.5% throughput on average, while average total latency rises by roughly 1.9x or 3.9x, respectively. Raw READ error counters should be checked before final publication of this new campaign.
• With 50 million documents and sustained WiredTiger cache pressure, repeated runs 2–4 show a small but consistent THP benefit: approximately 1.6–7.1% higher mean throughput, accompanied by lower mean and P95 total latency at every tested concurrency level. Throughput continues to scale above 16 threads, but average and P95 latency rise sharply, making 16 threads the most balanced operating point and 32–64 threads throughput-oriented choices.

The evidence, therefore, supports a more nuanced conclusion than either “THP always improves MongoDB” or “THP has no effect”:

On this RK1 ARM64 cluster, THP is neutral for the small, relatively cache-resident workload and modestly beneficial for the 50M-document cache-pressure workload in these repeated runs. The larger workload shows both greater THP-backed memory and higher mean throughput with THP ON; these measurements do not isolate the memory-coverage increase as the cause. WiredTiger, NVMe I/O, replication, and concurrency remain the dominant determinants of overall performance. Sixteen client threads is the small-dataset throughput/latency sweet spot; on the large dataset, 32 and 64 threads can offer substantial additional throughput, but also increase average and P95 latency.

The operations-per-watt chart in Part I describes the platform’s power envelope, but it is based on server-side MongoDB operations rather than YCSB requests. A per-thread and time-aligned power comparison would be required to establish whether THP improves completed YCSB work per watt.

These findings are specific to this RK3588 hardware, MongoDB 8.3.8, custom Linux kernel, Docker deployment, WiredTiger configuration, NVMe subsystem, replica-set topology, and YCSB workload. They should not be generalized into a universal MongoDB THP rule. For this project, the MongoDB 8-recommended THP-enabled configuration remains the operational default, while THP OFF is retained for controlled A/B testing.

Further Performance Optimizations Worth Testing

• A/B test MongoDB in Docker versus bare metal to quantify the actual containerization overhead on RK1. The current container has no resource limits, /data/db is a host bind mount, and the metrics confirm that THP is already active for the containerized mongod, so Docker is not expected to prevent THP operation. Small/no improvement expected, but needs to be checked.
• Benchmark mongodb_wiredtiger_cache_size_gb at 4, 6, and 8 on the 16 GB RK1 nodes. The system currently defaults to 6 to keep more RAM available for Linux page cache, Docker overhead, and monitoring while still preserving a useful WiredTiger working set.
• A/B test mongodb_wiredtiger_collection_compressor between zstd and snappy. The current zstd default is space-efficient, but snappy often improves CPU-bound write throughput on ARM.
• Consider pinning the mongod container to the RK3588 Cortex-A76 cores during write-heavy benchmarks. The big.LITTLE topology can otherwise mix latency-sensitive database threads with background tasks.
• Compare the current direct-primary workload with a replica-set-aware client profile to quantify driver-routing and failover behavior. The present measured workload already uses writeConcern=majority.
• Pin the monitoring containers away from :latest before long benchmark campaigns, so exporter behavior and Prometheus/Grafana feature changes do not skew repeatability.
• With the recent announcement of MongoDB 9 expect some more comparative benchmarks soon

MongoDB Cluster on ARM RK1 modules — Part II. Memory Management Deep-dive was originally published in MongoDB on Medium, where people are continuing the conversation by highlighting and responding to this story.