How I built a hybrid-search, self-cleaning long-term memory for an AI agent that runs entirely inside the app process: no server, no network, about 150 lines of Python.

Every AI agent has the same embarrassing flaw: it forgets you the moment the session ends. You tell it you prefer concise answers, that your service is called ledgerly, that deploys go through staging first. Tomorrow it greets you like a stranger.

The usual fix is to bolt a vector database onto the agent. That works, but it drags in a server, a network hop on every turn, and a privacy question: your user’s memories now live on someone else’s machine.

In this post we build a different kind of agent memory, one that lives inside the agent process using Qdrant Edge. By the end you will have:

• A memory store with episodic, semantic and procedural memories
• Hybrid recall (dense embeddings + BM25) fused with Reciprocal Rank Fusion
• Recency and importance weighting, so fresh and important memories win ties
• Consolidation, where old conversations get compressed into durable facts
• Forgetting, so the store does not grow forever
• A drop-in agent loop that reads and writes memory every turn

Every snippet below was run against qdrant-edge-py 0.8.0.

What “memory” means for an agent

An LLM’s context window is working memory: fast, small, and wiped at the end of each call. Long-term memory has to live outside the model, and cognitive science gives us a useful way to split it into three kinds.

The trick is that all three can live in one vector collection. A kind field in the payload tells them apart, and payload filters let us ask for one kind, two kinds, or everything. That design keeps the code small and lets a single hybrid query rank across all of them.

Why Qdrant Edge instead of a vector server

Qdrant Edge is the in-process flavour of Qdrant. There is no client and no server: you open a shard on local disk and call it like a library, the same way you would use SQLite. The Python package is qdrant-edge-py, and it imports as qdrant_edge.

For agent memory that brings four concrete wins:

• Zero network latency. Recall runs on every single agent turn, so shaving a round trip off each one adds up fast.
• Offline by default. A coding agent on a laptop, a field-service app, or a desktop assistant keeps its memory on a plane.
• Privacy. A user’s memories never leave their device unless you choose to sync them.
• Real Qdrant features. You still get HNSW indexing, payload indexes, named dense and sparse vectors, prefetch plus fusion, and even a built-in BM25 encoder. Nothing here is a toy.

pip install qdrant-edge-py sentence-transformers

When should you not use Edge? If many users or many agent instances must share one memory in real time, a Qdrant server or Qdrant Cloud is the better home. We will see later how the two can work together.

The architecture

The whole system is one Python class, AgentMemory, wrapping one Edge shard on local disk. The agent loop calls it twice per turn: recall() before the LLM answers, and remember() after.

Each memory is stored as a single point with two named vectors and a payload:

• dense: a 384-dimension sentence embedding (all-MiniLM-L6-v2) that captures meaning
• bm25: a sparse vector from Qdrant Edge's built-in BM25 encoder that captures exact words, such as project names, error codes and commands
• payload: text, kind, user_id, importance, created_at and access_count

Two background jobs keep the store healthy. Consolidation turns piles of old episodic memories into a few semantic facts. Forgetting deletes old, low-importance memories nobody needs.

Building it, step by step

Step 1: Create (or reopen) the shard

The shard is a folder on disk. On first run we create it with a dense vector, a sparse BM25 vector with IDF weighting, and payload indexes on the fields we filter by. On later runs we simply load it.

import time, uuid
from pathlib import Path
from sentence_transformers import SentenceTransformer
from qdrant_edge import (
    EdgeShard, EdgeConfig, EdgeVectorParams, EdgeSparseVectorParams, Distance, Modifier,
    Point, UpdateOperation, PayloadSchemaType, QueryRequest, Prefetch, Query, Fusion,
    Filter, FieldCondition, MatchValue, RangeFloat, ScrollRequest, CountRequest, Bm25,
)
DIM = 384
class AgentMemory:
    def __init__(self, path="./agent_memory"):
        self.embedder = SentenceTransformer("all-MiniLM-L6-v2")
        self.bm25 = Bm25()  # built-in sparse encoder, no extra model
        config = EdgeConfig(
            vectors={"dense": EdgeVectorParams(size=DIM, distance=Distance.Cosine)},
            sparse_vectors={"bm25": EdgeSparseVectorParams(modifier=Modifier.Idf)},
        )
        if Path(path).exists():
            self.shard = EdgeShard.load(path)
        else:
            Path(path).mkdir(parents=True)  # create() needs the folder to exist
            self.shard = EdgeShard.create(path, config)
            for field, schema in [("kind", PayloadSchemaType.Keyword),
                                  ("user_id", PayloadSchemaType.Keyword),
                                  ("created_at", PayloadSchemaType.Float),
                                  ("importance", PayloadSchemaType.Float)]:
                self.shard.update(UpdateOperation.create_field_index(field, schema))

Two gotchas I hit: EdgeShard.create() fails if the folder does not exist yet, and Modifier.Idf matters, because without it BM25 scores ignore how rare a word is across your memories.

Step 2: Write a memory

Every memory gets both vectors at write time, plus a timestamp and an importance score between 0 and 1.

def remember(self, text, kind="episodic", user_id="default", importance=0.5, **extra):
        point_id = str(uuid.uuid4())
        self.shard.update(UpdateOperation.upsert_points([
            Point(
                id=point_id,
                vector={
                    "dense": self.embedder.encode(text, normalize_embeddings=True).tolist(),
                    "bm25": self.bm25.embed_document(text),
                },
                payload={"text": text, "kind": kind, "user_id": user_id,
                         "importance": importance, "created_at": time.time(),
                         "access_count": 0, **extra},
            )
        ]))
        return point_id

Step 3: Hybrid recall with RRF

This is the heart of the system. Dense search finds memories that mean the same thing (“ship the app” finds “deploy ledgerly”). BM25 finds memories with the same words (“ledgerly”, “ECONNREFUSED”). We run both as prefetches inside one query and let Reciprocal Rank Fusion merge the two ranked lists.

def _filter(self, user_id, kinds=None, since=None):
        must = [FieldCondition(key="user_id", match=MatchValue(value=user_id))]
        if since is not None:
            must.append(FieldCondition(key="created_at", range=RangeFloat(gte=since)))
        if kinds:
            return Filter(must=must, should=[
                FieldCondition(key="kind", match=MatchValue(value=k)) for k in kinds])
        return Filter(must=must)
def recall(self, query, user_id="default", k=5, kinds=None, since=None,
               half_life_days=14.0):
        flt = self._filter(user_id, kinds, since)
        dense_q = self.embedder.encode(query, normalize_embeddings=True).tolist()
        hits = self.shard.query(QueryRequest(
            prefetches=[
                Prefetch(query=Query.Nearest(dense_q, using="dense"),
                         limit=k * 4, filter=flt),
                Prefetch(query=Query.Nearest(self.bm25.embed_query(query), using="bm25"),
                         limit=k * 4, filter=flt),
            ],
            query=Fusion.Rrf(k=60),
            limit=k * 4,
            with_payload=True,
        ))

Note that Fusion.Rrf must be instantiated with k; passing the bare class raises a type error. The filter is applied inside each prefetch, so one user's memories never leak into another user's results.

Step 4: Re-rank by recency and importance

RRF tells us what is relevant. Humans also weigh what is recent and what mattered. We multiply the fused score by a blend of an exponential recency decay (half-life of 14 days) and the stored importance, then reinforce whatever we return.

now = time.time()
        def final(h):
            age_days = (now - h.payload["created_at"]) / 86400
            recency = 0.5 ** (age_days / half_life_days)
            return h.score * (0.6 + 0.25 * recency + 0.15 * h.payload["importance"])
ranked = sorted(hits, key=final, reverse=True)[:k]
        for h in ranked:  # memories that get used become easier to keep
            self.shard.update(UpdateOperation.set_payload(
                [h.id], {"access_count": h.payload["access_count"] + 1}))
        return ranked

The 0.6 floor means relevance always dominates. Recency and importance only break ties and nudge close calls, which is exactly what you want from memory.

Step 5: Consolidation, from episodes to facts

Left alone, episodic memory becomes a pile of chat logs. Consolidation scrolls through old episodes, asks a summarizer (usually your LLM) for durable facts, stores those as semantic memories, and deletes the originals in one filter-based call.

def consolidate(self, user_id, summarize, older_than_days=1.0, min_items=3):
        cutoff = time.time() - older_than_days * 86400
        old = Filter(must=[
            FieldCondition(key="user_id", match=MatchValue(value=user_id)),
            FieldCondition(key="kind", match=MatchValue(value="episodic")),
            FieldCondition(key="created_at", range=RangeFloat(lt=cutoff)),
        ])
        records, _ = self.shard.scroll(ScrollRequest(filter=old, limit=100, with_payload=True))
        if len(records) < min_items:
            return None
        facts = summarize([r.payload["text"] for r in records])
        for fact in facts:
            self.remember(fact, kind="semantic", user_id=user_id, importance=0.8)
        self.shard.update(UpdateOperation.delete_points_by_filter(old))
        return facts

Step 6: Forgetting on purpose

Forgetting is a feature. Old, unimportant memories add noise to recall and cost disk. One filtered delete handles it.

def forget(self, user_id, max_age_days=90, below_importance=0.3):
        cutoff = time.time() - max_age_days * 86400
        self.shard.update(UpdateOperation.delete_points_by_filter(Filter(must=[
            FieldCondition(key="user_id", match=MatchValue(value=user_id)),
            FieldCondition(key="created_at", range=RangeFloat(lt=cutoff)),
            FieldCondition(key="importance", range=RangeFloat(lt=below_importance)),
        ])))
def count(self, user_id):
        return self.shard.count(CountRequest(filter=self._filter(user_id)))

Does it work?

Here is a quick test with five memories for one user and one for a different user:

m = AgentMemory("./mem_test")
u = "asha"
m.remember("User prefers concise answers with code examples", kind="semantic", user_id=u, importance=0.9)
m.remember("User is building a FastAPI service called ledgerly", kind="semantic", user_id=u, importance=0.8)
m.remember("Debugged a Postgres connection timeout in ledgerly on Tuesday", user_id=u)
m.remember("User asked about pytest fixtures for async code", user_id=u, importance=0.4)
m.remember("To deploy ledgerly: run make build then make deploy-staging", kind="procedural", user_id=u, importance=0.9)
m.remember("Other user likes Rust", kind="semantic", user_id="bob")
for h in m.recall("how do I deploy ledgerly?", user_id=u, k=3):
    print(h.payload["kind"], "|", h.payload["text"])

procedural | To deploy ledgerly: run make build then make deploy-staging
semantic   | User is building a FastAPI service called ledgerly
episodic   | Debugged a Postgres connection timeout in ledgerly on Tuesday

The deploy recipe comes first, Bob’s memory never appears, and after closing and reloading the shard from disk the count is unchanged. The memory survives restarts.

Plugging memory into an agent loop

Memory only matters if the agent uses it. The loop below does three things per turn: recall relevant memories and inject them into the system prompt, answer, then ask the LLM to extract anything worth remembering from the exchange.

import json
SYSTEM = """You are a helpful coding assistant.
Relevant long-term memories about this user (may be outdated):
{memories}"""
EXTRACT = """From this exchange, list durable facts worth remembering about the user,
as JSON: [{{"text": str, "kind": "semantic"|"procedural"|"episodic", "importance": 0-1}}].
Return [] if nothing is worth keeping.
User: {user}
Assistant: {reply}"""
class MemoryAgent:
    def __init__(self, llm, memory, user_id):
        self.llm, self.memory, self.user_id = llm, memory, user_id
    def chat(self, user_msg):
        # 1. Recall before answering
        hits = self.memory.recall(user_msg, user_id=self.user_id, k=5)
        memories = "\n".join(
            f"- [{h.payload['kind']}] {h.payload['text']}" for h in hits) or "- none yet"
        # 2. Answer with memories in context
        reply = self.llm(system=SYSTEM.format(memories=memories), prompt=user_msg)
        # 3. Extract and store new memories
        try:
            items = json.loads(self.llm(system="Reply with JSON only.",
                                        prompt=EXTRACT.format(user=user_msg, reply=reply)))
        except json.JSONDecodeError:
            items = []
        for it in items:
            self.memory.remember(it["text"], kind=it.get("kind", "episodic"),
                                 user_id=self.user_id,
                                 importance=float(it.get("importance", 0.5)))
        return reply

The llm argument is any function that takes a system prompt and a user prompt and returns text, so you can plug in any provider. Here is a version using the Anthropic SDK:

import anthropic
client = anthropic.Anthropic()
def llm(system, prompt):
    msg = client.messages.create(
        model="claude-sonnet-4-6",  # use whichever model you prefer
        max_tokens=1024,
        system=system,
        messages=[{"role": "user", "content": prompt}],
    )
    return msg.content[0].text
agent = MemoryAgent(llm, AgentMemory("./agent_memory"), user_id="asha")
agent.chat("I always ship ledgerly to staging first")
# ...next day, new process...
agent.chat("How should I release ledgerly today?")

On the second call, the system prompt the model sees already contains:

- [procedural] User deploys ledgerly to staging before prod

That is the whole point: the agent “remembers” across processes because the memory lives on disk, right next to the app.

To keep the store tidy, schedule the maintenance jobs, for example once a day or on app start:

def summarize(texts):
    out = llm(system="Reply with a JSON list of short strings only.",
              prompt="Merge these notes into a few durable facts:\n" + "\n".join(texts))
    return json.loads(out)
memory.consolidate("asha", summarize, older_than_days=1)
memory.forget("asha", max_age_days=90, below_importance=0.3)

Syncing with a Qdrant server (optional)

Local-first does not have to mean local-only. A common pattern is to treat the Edge shard as the agent’s fast, private cache and a Qdrant server or Qdrant Cloud collection as the durable backup that follows the user across devices.

The two directions work differently:

• Edge to server (push). Durable memories, usually only semantic and procedural ones, are upserted to a server collection with the regular qdrant-client. Raw episodic chatter can stay on the device.
• Server to Edge (pull). A new device bootstraps from a snapshot of the server data. The Edge API exposes EdgeShard.unpack_snapshot(), update_from_snapshot() and snapshot_manifest() for this; check the Qdrant Edge docs for the current snapshot workflow, since this part of the API is new and evolving.

A minimal push of durable memories looks like this:

from qdrant_client import QdrantClient, models
server = QdrantClient(url="https://YOUR-CLUSTER.cloud.qdrant.io", api_key="...")
def push_durable(memory, user_id):
    durable = Filter(
        must=[FieldCondition(key="user_id", match=MatchValue(value=user_id))],
        should=[FieldCondition(key="kind", match=MatchValue(value="semantic")),
                FieldCondition(key="kind", match=MatchValue(value="procedural"))],
    )
    records, _ = memory.shard.scroll(ScrollRequest(
        filter=durable, limit=500, with_payload=True, with_vector=True))
    server.upsert("agent_memory", points=[
        models.PointStruct(id=r.id, vector={"dense": r.vector["dense"]}, payload=r.payload)
        for r in records
    ])

The server collection needs a matching dense vector config, and you can add the bm25 sparse vector there too if you want hybrid search server-side. Pushing only durable memory keeps bandwidth low and keeps the noisiest, most personal data on the device.

Limitations

• Hybrid beats dense alone for agents. Agent memories are full of identifiers: project names, file paths, error codes, CLI commands. Dense embeddings blur those; BM25 nails them. RRF gives you both without tuning score scales.
• Filter inside the prefetch. Putting the user_id filter in each prefetch, not after fusion, guarantees isolation and keeps the candidate pools full of useful hits.
• Forgetting improves recall. Once the store held a few thousand stale episodes, relevant facts started losing to noisy near-matches. Consolidation plus forgetting fixed that.
• Let the LLM decide what is memorable, but cap it. Extraction prompts can over-remember. Keep importance scores honest and skip storing small talk.

There are honest limits too. The read-modify-write on access_count is not atomic, which is fine for a single agent process but not for many writers. Embedding on every write costs CPU on small devices, so batch writes when you can. And Qdrant Edge is young, so pin your package version.

Where to take it next

• Use Qdrant’s server-side formula queries with a decay function to move recency scoring into the engine
• Add MMR (Mmr is in the Edge API) to diversify recalled memories
• Add a multimodal vector so the agent remembers screenshots and diagrams
• Expose AgentMemory as an MCP server so any agent framework can use it

Resources: Qdrant Edge documentation · Qdrant on GitHub · Hybrid queries in Qdrant

Models don’t have that. If you want one that’s better at chemistry, you fine-tune the whole thing or train a new one. Two labs can’t each build a piece and snap them together, because every layer inside a trained model speaks a private dialect. It only makes sense next to the exact weights it grew up with.

The one test that matters

Two people train two modules separately. They never see each other’s code or data. Then we plug both modules into the same model, and it does nearly as well as a model where everything was trained together.

That’s it. If that works on a toy problem, I’ll believe there’s something here. If it doesn’t, at least it fails fast and cheap.

I like this test because of what it leaves out. The modules don’t need to be small or efficient. There’s no new training algorithm involved. It’s just asking whether two strangers can build parts that understand each other, because if they can’t, nothing else I’d want to build on top of this matters.

A whiteboard everyone shares

The first piece is a shared workspace: a fixed set of slots, say 64 vectors, that every module reads from and writes to.

The picture in my head is a whiteboard in a meeting room. Nobody whispers to anyone else. If you have something to say, you write it on the board, and if you want to know what’s going on, you read the board. Modules work the same way. They never call each other directly.

The board is the socket. Its shape doesn’t change: same number of slots, same width, same spot in the forward pass. As long as a module fits that shape, it can plug in.

What a module promises

The second piece is a contract every module signs. It reads the workspace through attention, does some work, and writes results back.

The middle part is nobody’s business. A module could be a transformer, a CNN, a lookup table, or a hand-written symbolic solver. I only care about the read and the write. If you’ve written a function against someone else’s API, it’s the same deal: agree on what goes in and what comes out, and do whatever you want in between.

The whiteboard and the contract are the easy bits. This next one is where I think the whole idea lives or dies.

Agreeing on the shape of the board doesn’t mean agreeing on what’s written on it. Say two modules were trained apart, and both learned to use slot 12. One uses it to mean “the subject of the sentence.” The other uses it to mean “something red.” Plug them in together and they’ll happily read each other’s notes and get everything wrong. Same alphabet, different language.

So something has to pin down what each slot means. The simplest version I can think of is a frozen reference model, an anchor. Every module trains against it: it learns from workspace states the anchor produced, and whatever it writes has to stay readable by the anchor. Since the anchor never changes, a module built this week and one built next year should still share a dictionary.

What I genuinely don’t know is whether that’s enough. A frozen anchor might also be a ceiling. If the anchor has no way to represent some concept, can a new module still get that idea across? Or does every module end up limited to the vocabulary of whatever we froze on day one? If I had to pick one question to spend a year on, it’d be this one.

Who gets called

The last piece is a router that decides which modules run for a given input. Routers aren’t new. The awkward requirement here is that this one has to accept modules that didn’t exist when it was trained. If you need to retrain the router every time someone ships a module, you’ve lost most of the point.

One way I can imagine it working: each module publishes a small key, a vector living in the same space as the workspace. The router compares the current state of the board against those keys and picks the closest matches. Joining means registering a key. Nobody retrains anything.

How I’d test it

Putting it together, the first experiment looks something like this:

Two teams, separate data, separate code. The only thing they share is the frozen anchor. Plug both modules in, then compare against a baseline with the same parts trained together. A small gap means the idea works. A big gap tells us where it breaks, which is useful too.

Hasn’t someone done this?

Sort of, and I’d be surprised if nobody had come close. The nearest work I know of:

• “Coordination Among Neural Modules Through a Shared Global Workspace” (Goyal et al., 2021). This is almost exactly the whiteboard idea. But the modules are trained together.
• Recurrent Independent Mechanisms (Goyal et al., 2019). Separate modules that compete to update a shared state. Also trained jointly.
• Perceiver (Jaegle et al., 2021). Reads and writes through a fixed latent array, which is very close to how I picture the workspace.

There’s also a nearby line of work on model stitching and on aligning representations between separately trained networks, plus methods that train expert models in parallel and merge them afterward. Those feel relevant to the anchor question in particular.

As far as I can tell, none of them tests the specific thing I care about: modules trained independently, by people who never coordinated, plugged in after the fact. That looks like the gap. I’m going from memory on these citations, though, so if you know of work that already does this, please tell me. I’d honestly rather find out now.

The stuff I’m deliberately ignoring

There are two bigger ideas sitting behind this one, and I’m parking both on purpose.

The first is training without backpropagation. Backprop needs the whole network held in memory while gradients flow backward, which makes training expensive and pretty much rules out a device learning on the fly. The dream is a network where each layer learns from local signals only, so layers can train in parallel and out of sync, a bit like the brain seems to. Hinton’s Forward-Forward algorithm and predictive coding networks are early steps, but as far as I know nothing matches backprop at scale yet. Whoever cracks that changes what hardware we build.

The second is an asynchronous framework where modules update on their own schedules

What If Neural Networks Came in Parts? was originally published in Tutorial Savvy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ask a good LLM what happens if you nudge a glass off the edge of a table and you’ll get a flawless answer: it falls, it probably shatters, there’s water on the floor. Now ask it to show you the next three seconds, from the other side of the room, after you’ve nudged the glass two centimetres to the left instead. It can’t. It knows the sentence. It doesn’t have the scene.

That gap is what large world models are trying to close.

In the last 18 months, world models have gone from a research curiosity to one of the most heavily funded ideas in AI. Google DeepMind’s Genie 3 generates explorable worlds in real time. Fei-Fei Li’s World Labs raised $1 billion in February 2026 and, on September 28, agreed to join AMD. Yann LeCun left Meta to found AMI Labs, which raised $1.03 billion to build world models. NVIDIA’s Cosmos models, trained on 20 million hours of video, are being used to train robots and self-driving cars. Gartner put “physical AI” on its list of top strategic technology trends for 2026.

This post is a practical guide. We’ll cover:

• what a world model is, and what makes one “large”
• how LWMs work under the hood, with block diagrams
• the four families of world models and the flagship systems in each
• where they’re used today, and what’s still hard
• how LWMs differ from LLMs, and why nearly every AI product you use still runs on an LLM

TL;DR: An LLM predicts the next token of text. An LWM predicts the next state of a world (video frames, a 3D scene or an abstract latent state) given an action. That makes LWMs the right tool for robotics, driving, games and simulation, and the wrong (and far more expensive) tool for most knowledge work. The future isn’t one replacing the other. It’s a stack where each does what it’s best at.

1. What is a world model?

A world model is a learned simulator. Give it the current state of an environment and an action, and it predicts what happens next:

next_state = f(current_state, action)

The idea is older than deep learning. In 1943 the psychologist Kenneth Craik suggested that the mind carries a “small-scale model” of external reality, which lets us try out alternatives before committing to one. You use yours constantly. To catch a ball, you predict where it will be, not where it is. To cross a dark room, you remember where the coffee table was.

A machine world model has three jobs:

1. Represent. Compress raw observations (pixels, depth, lidar, joint angles) into a compact state.
2. Predict. Roll that state forward in time, conditioned on actions.
3. Stay consistent. Remember what it has seen, so the wall you walked past is still there when you turn around.

The core loop. The agent imagines many futures cheaply, then acts on the best one in the real world.

The payoff is imagination. An agent with a good world model can try a thousand candidate actions “in its head” and execute only the one that looks best. In 2018, David Ha and Jürgen Schmidhuber showed that an agent could learn to play a video game entirely inside “its own hallucinated dream,” then carry that skill back to the real game. They split the system into Vision (V), Memory (M) and Controller ©. That three-part shape still describes most world models today.

2. So what makes it “large”?

Classic world models were small and narrow: one model per game or robot, trained on that environment’s data. Large world models apply the recipe that made LLMs work:

• Internet-scale data: millions of hours of video instead of one game’s replays.
• Big transformer backbones: billions of parameters.
• Pretrain, then adapt: one general model, then fine-tune or prompt it for driving, robotics or games.
• Prompting: describe a world in text, or hand it a photo, and it generates one.

A note on naming, because it’s messy. World Labs describes its work as building “large world models.” NVIDIA calls Cosmos “world foundation models.” DeepMind called Genie 2 a “foundation world model.” And a 2024 Berkeley paper literally named “Large World Model” (LWM) was a long-context model that reads video and text across a million-token window, which is closer to a multimodal LLM than to a simulator. In this post, LWM means the broad category: large, general-purpose models that learn how worlds look and behave.

3. A short history

World models have deep roots in cognitive science and reinforcement learning, but most of the progress has happened since 2024.

Three threads came together.

Model-based reinforcement learning gave us the loop. Richard Sutton’s Dyna (1990) learned a model of the environment and planned inside it. Ha and Schmidhuber (2018) trained agents in their dreams. DeepMind’s MuZero mastered Go, chess and Atari by planning with a learned model it was never told the rules of. DreamerV3 used one configuration across more than 150 tasks and became the first algorithm to collect diamonds in Minecraft from scratch, with no human data. It was published in Nature in 2025.

Video generation gave us scale. OpenAI’s February 2024 Sora report argued that scaling video models is “a promising path towards building general purpose simulators of the physical world.”

Interactivity tied them together. DeepMind’s Genie (February 2024, 11 billion parameters) learned playable 2D worlds from unlabeled internet video, without any action labels. Genie 2 (December 2024) made playable 3D worlds from a single image, consistent for up to a minute. Genie 3 (August 2025) made them real-time.

4. Inside a large world model

There’s no single LWM architecture, but most systems fill the same four slots: inputs, tokenizers, a prediction backbone and outputs.

Step 1: Tokenize the world

Raw video is enormous. One minute of 720p video at 24 frames per second is 1,440 frames and more than 1.3 billion pixels. So the first job is compression. A video tokenizer (usually a VAE or a vector-quantized encoder) chops a clip into small space-time patches and maps each one to a compact latent token. The rest of the model never sees raw pixels; it works in this compressed space.

Other inputs get their own encoders: depth and camera pose, text prompts (often via a pretrained language model), and, most importantly, actions.

Step 2: Condition on actions

This is what separates a world model from a video generator. A video model answers “what might happen next?” A world model answers “what happens next if I do this?”

Actions can be explicit: key presses, steering angles, robot joint commands. But most internet video has no action labels. Nobody logged which buttons the player pressed. Genie’s clever trick was a latent action model that watches consecutive frames and infers a small set of discrete actions that explain the change. Train it on enough gameplay footage and those latent actions end up meaning things like “jump” or “move left,” without anyone labeling them.

Step 3: Predict the next state

The backbone is usually a large spatiotemporal transformer. It attends within each frame (space), across frames (time, usually causally so the future can’t leak into the past), and to the actions and text it’s conditioned on. Three training objectives dominate:

• Autoregressive prediction: predict the next video tokens one step at a time, the way an LLM predicts the next word. Genie’s dynamics model works this way.
• Diffusion or flow matching: start from noise and iteratively denoise the next frames, conditioned on history and actions. Most photoreal video world models work this way. NVIDIA released Cosmos in both diffusion and autoregressive versions.
• Embedding prediction (JEPA): don’t predict pixels at all. Predict the representation of the future.

That last choice is one of the biggest philosophical splits in the field.

Yann LeCun’s case for JEPA (Joint-Embedding Predictive Architecture) is that the world is full of details nobody can predict, like the exact pattern of leaves in the wind or ripples on a pond, and forcing a model to generate them wastes capacity. Predict an abstract summary instead, and the model can focus on what matters: the glass is falling, the car is braking. The trade-off is that you can’t watch a JEPA model’s imagination. There’s no video to inspect.

Step 4: Remember

Memory is the hardest part. Generate a world frame by frame and small errors compound: a door drifts, a sign changes, a building disappears when you look away. There are three main approaches:

• Frame cache: attend to recent frames. Genie 3’s visual memory reaches back about a minute.
• Pose-indexed spatial memory: store frames along with the camera position they were taken from, and retrieve the nearby ones when you come back. World Labs’ RTFM model does this and runs at interactive frame rates on a single H100 GPU.
• Explicit 3D: build a persistent 3D scene (Gaussian splats or meshes) so the world can’t forget. World Labs’ Marble takes this route.

The imagination loop, in code

Once you have a world model, planning is surprisingly simple. Here’s the core idea in pseudocode. It’s roughly how Meta’s V-JEPA 2-AC plans robot arm movements toward a goal image:

# Planning with a world model (simplified pseudocode)
z    = encoder(camera_frame)       # 1. compress what you see
goal = encoder(goal_image)         #    ...and what you want to see

best_plan, best_score = None, float("-inf")
for plan in sample_action_sequences(n=256, horizon=10):
    z_sim = z
    for action in plan:            # 2. imagine; don't act yet
        z_sim = dynamics(z_sim, action)
    score = -distance(z_sim, goal) # 3. how close did we get?
    if score > best_score:
        best_plan, best_score = plan, score

robot.execute(best_plan[0])        # 4. take one step, observe, repeat

No hand-written reward function and no real-world trial and error. The robot rehearses in latent space, takes one step, looks again, and replans.

5. The four families of world models

What a model predicts determines what it’s good at.

Interactive video world models (Genie 3, Runway GWM, Odyssey, Decart, Cosmos Predict, Waymo’s world model, Wayve GAIA) generate pixels in response to actions. They’re the most visually impressive and the most general. They’re also the most expensive to run, and they drift over time.

Spatial or 3D world models (World Labs Marble, Tencent HunyuanWorld) produce an explicit 3D scene you can walk through, edit and export to other tools. Persistence comes for free. Rich dynamics don’t.

Latent predictive models (Meta’s V-JEPA 2, AMI Labs) predict in embedding space. They’re efficient and good for planning, but hard to show off, since there’s nothing to watch.

Agent-centric RL world models (DreamerV3, MuZero) learn compact dynamics for one environment. They’re narrow but proven, and they’re the source of most of the field’s core ideas.

Real systems increasingly mix families. Cosmos pairs video models with a reasoning vision-language model, and World Labs’ RTFM renders frames in real time on top of a spatial memory.

6. The flagship systems, as of September 2026

Google DeepMind: Genie 3 and Project Genie

Genie 3, announced in August 2025, generates worlds from a text prompt in real time at 720p and 24 fps, and keeps them consistent for a few minutes. It introduced “promptable world events”: type a text command to change the weather or add a new object or character, and the world changes mid-session. DeepMind also uses Genie 3 worlds as a training ground. Its SIMA 2 agent plays and improves itself inside them.

In January 2026, Google opened Project Genie, a prototype for US Google AI Ultra subscribers with a 60-second limit per world. In May it added Street View, so you can generate worlds from real streets. DeepMind’s Jack Parker-Holder was candid about where it stands, calling it “maybe six to 12 months behind video” in quality. And in February, Waymo unveiled the Waymo World Model, built on Genie 3. It generates both camera and lidar data and can simulate rare events like a tornado or an elephant on the road.

World Labs: Marble, RTFM and the AMD deal

Fei-Fei Li’s World Labs came out of stealth in September 2024 with $230 million to build “large world models” with spatial intelligence. Its product Marble (generally available since November 2025) turns text, images, video or rough 3D layouts into persistent 3D worlds, and exports them as Gaussian splats, meshes or video. RTFM, its real-time frame model, renders explorable worlds on a single H100.

In 2026 World Labs launched a World API, raised $1 billion from backers including AMD, Autodesk and NVIDIA, and released Atlas, which generates up to a minute of 1440p video. On September 28, 2026, it announced an agreement to join AMD, with Fei-Fei Li set to become AMD’s Chief Scientist. The deal is expected to close by the end of 2026. TechCrunch reports its value at $8.2 billion.

NVIDIA: Cosmos

NVIDIA launched Cosmos at CES in January 2025 as a platform of open world foundation models for “physical AI,” trained on 9,000 trillion tokens drawn from 20 million hours of real-world data. The family grew into three lines: Predict (generate future video), Transfer (turn structured inputs like depth maps, segmentation or simulator output into photoreal video) and Reason (a vision-language model that reasons about physical scenes). Cosmos 3, released in May 2026, merges text, image, video, sound and action into one “omnimodel” in 4B, 16B and 64B sizes.

Cosmos’s pitch is less “play a world” and more “make training data for robots and cars.”

Meta and AMI Labs: the JEPA bet

Meta’s V-JEPA 2 (June 2025) is a 1.2-billion-parameter model trained on more than a million hours of video. Its action-conditioned version, V-JEPA 2-AC, was trained on just 62 hours of robot data and then planned pick-and-place tasks zero-shot in new labs, with 65–80% success. V-JEPA 2.1 (March 2026) improved real-robot grasping by about 20 points.

Yann LeCun, who championed JEPA at Meta, left to found AMI Labs in Paris. In March 2026 it raised $1.03 billion at a $3.5 billion pre-money valuation to build JEPA-style world models. As of September 2026, it hasn’t released a product and describes itself as still in a research and building phase.

Driving: Waymo, Wayve and Tesla

Self-driving is the most mature commercial use of world models. Besides Waymo’s Genie-based model, Wayve’s GAIA-3 (15 billion parameters, December 2025) and GAIA-4 (August 2026, closed-loop simulation) generate driving scenarios for training and testing, and Tesla has described a “neural world simulator” for evaluating its driving software. The value is obvious: you can’t wait for a real child to run into a real road to test your car, but you can simulate it a million times.

Also worth knowing

Runway’s GWM-1 (December 2025) and GWM Worlds 2 (September 2026) target media and robotics. Microsoft’s Muse, published in Nature, generates gameplay from a real game’s data. Odyssey-3 (September 2026) aims to cover robots, cars and games in one model. Tencent’s HunyuanWorld line offers open-source 3D world generation. And OpenAI, whose Sora report helped popularize the “world simulator” framing, discontinued the Sora app in April 2026.

7. Where world models are used today

The common thread is that reality is a slow and expensive teacher. With a world model, robots can rehearse before they move, cars can crash safely, agents can train on endless worlds, and game studios and VFX teams can prototype in minutes instead of weeks.

8. What’s still hard

• Long-horizon consistency. DeepMind describes Genie 3 as supporting “a few minutes of continuous interaction, rather than extended hours.” Small errors compound.
• Physics that looks right but isn’t. Visual plausibility is not physical accuracy. TechCrunch described Project Genie as “not yet physics-aware.” A robot trained on subtly wrong physics learns subtly wrong skills.
• Limited actions and multiple agents. DeepMind lists a limited action space and interactions between multiple agents as Genie 3 limitations, along with text legibility and geographic accuracy.
• Data. There’s plenty of video, but very little action-labeled video that records what caused each change.
• Evaluation. There’s no MMLU for worlds. Meta released three new benchmarks (IntPhys 2, MVPBench and CausalVQA) alongside V-JEPA 2 because existing ones didn’t measure physical understanding well.
• Compute. This is the biggest reason LWMs aren’t everywhere yet. More on it below.
• Trust and safety. Photoreal, controllable video of real places raises obvious questions about deepfakes and misuse.

9. LLMs vs LWMs: what’s actually different?

At a high level they share a recipe: big transformers, scaling laws, and pretraining on huge datasets followed by post-training. The difference is what they predict.

An LLM lives in a one-dimensional world of text. Its state is the text so far, its unit of prediction is a sub-word token, and a hundred tokens a second feels instant. An LWM lives in space and time. Its state is a whole scene, including whatever is currently off-screen. Its unit is a space-time patch, and it has to produce 24 to 60 frames every second to feel real.

The most important differences, in words:

• The core question. An LLM asks “what word comes next?” An LWM asks “what happens next if I do this?”
• Training data. Recent open LLMs train on roughly 15 to 36 trillion tokens of text and code (Llama 3 used 15T+, Qwen3 about 36T). Cosmos trained on about 9,000 trillion tokens from 20 million hours of video. Each video token carries far less meaning than a text token, so the numbers aren’t directly comparable.
• The scarce ingredient. For LLMs, it’s fresh, high-quality human text. For LWMs, it’s action-labeled data.
• What they understand. LLMs: language, facts, code and symbolic reasoning. LWMs: geometry, intuitive physics, cause and effect, object permanence.
• Time. LLMs are turn-based, and a second of latency is fine. LWMs are continuous and real-time.
• How they fail. LLMs hallucinate facts. LWMs hallucinate physics: drift, melting objects, things that vanish.
• Maturity. LLMs are everywhere, with mature APIs and tooling. LWMs are mostly research previews and early APIs.

Don’t LLMs already have a world model? Partly. Trained on enough text, LLMs absorb a lot about how the world works, which is why they can explain why a glass breaks. But that knowledge comes from descriptions of the world, not from observing it, and it isn’t spatial, continuous or tied to actions. An LLM knows the glass falls. An LWM can show you where the shards land.

10. So why do we still use LLMs, not LWMs?

If world models are closer to how intelligence works in the physical world, why does nearly every AI product (chat assistants, coding tools, support bots, search) still run on LLMs? There are seven reasons.

1. Most valuable digital work is language-shaped

Code, contracts, emails, tickets, spreadsheets, reports: most knowledge work is text in, text out. A world model can simulate a kitchen, but it can’t write your SQL query, summarize a contract or fix a failing test. The problem picks the model, and most software problems are language problems.

2. Text is pre-compressed meaning

“The glass fell and shattered” is five words. Rendering it as video takes hundreds of thousands of tokens. When people write, they’ve already done the hard work of abstraction: deciding what matters and naming it. That’s why an LLM learns so much per token, and why a world model has to wade through huge volumes of redundant pixels to learn the same concept. Worse, most video lacks the thing a world model needs most: a record of the action that caused each change.

3. Serving a world costs orders of magnitude more

World-model figures from World Labs’ RTFM post (October 2025). LLM figures are typical 2026 values.

A chatbot streams about a hundred tokens a second and can pause to think. A world model has to generate frames continuously, for every user, against hard real-time deadlines. World Labs has written that world models “will be very computationally demanding, much more so than LLMs today.” By its estimate, 4K video at 60 fps needs more than 100,000 tokens per second, and keeping a world consistent for an hour would take more than 100 million tokens of context, which it calls “neither feasible nor economically viable” today.

You can see the economics in how these systems are offered. Genie 3 launched as a limited research preview, and Project Genie caps each world at 60 seconds for top-tier subscribers.

4. You can check an LLM’s work

When an LLM writes code, you run the tests. When it states a fact, you can look it up. Checking whether a simulated world obeys physics is much harder, and the benchmarks to do it are still being invented. For a business, being able to verify output is what turns a demo into a product.

5. The ecosystem is years ahead

LLMs come with everything a team needs to ship: stable APIs, fine-tuning, retrieval, function calling, agent frameworks, evaluation harnesses, guardrails, and lots of engineers who know how to use them. LWMs mostly offer research previews, waitlists and first-generation APIs. AMI Labs, with more than a billion dollars raised, hasn’t released a product yet.

6. Language is still the interface

People specify goals in words. Even world models are prompted in text (“a rainy street in Tokyo”), and many use a pretrained language model as their text encoder. NVIDIA built Cosmos Reason, a vision-language model, because a physical-AI system still needs something that can follow instructions and explain its reasoning. Here LLMs aren’t competing with world models; they’re a component inside them.

7. They aren’t substitutes

“Why LLMs and not LWMs?” assumes one should replace the other. They do different jobs. Asking an LWM to draft a strategy memo is like asking a flight simulator to write the pilot’s manual. The useful question is where each one fits.

11. Which one should you reach for?

A practical rule of thumb:

• Use an LLM when the input and output are language, code or structured data, and the job is reasoning, retrieval, writing or tool use.
• Use a vision-language model when you need to understand images or video but can answer in text: inspection, captioning, visual Q&A.
• Use a world model when you need to predict what physically happens next under different actions: generating training data, testing robot or driving policies, rehearsing plans, or building explorable 3D and interactive media.
• Use both when an agent has to act in the physical world. That’s where the field is heading.

12. The future is a stack, not a winner

Each layer does the job it’s best at. The loop between imagination and action is the key idea.

The emerging architecture for physical AI looks less like one giant model and more like a team. A language model understands the request and plans the steps. A world model imagines what each candidate step would do. An action model, often a vision-language-action (VLA) model like Physical Intelligence’s π series, turns the chosen plan into motor commands. Sensors stream back, and the loop repeats.

You can already see the pieces converging. Cosmos 3 folds text, video, sound and action into one model. DeepMind trains SIMA 2 agents inside Genie 3 worlds. Waymo tests its cars against Genie-based simulations of events it may never record in reality. The lines between “language model,” “video model” and “world model” get blurrier every quarter.

References

1. Google DeepMind, “Genie 3: A new frontier for world models” (Aug 2025): https://deepmind.google/discover/blog/genie-3-a-new-frontier-for-world-models/
2. Google DeepMind, “Genie 2: A large-scale foundation world model” (Dec 2024): https://deepmind.google/discover/blog/genie-2-a-large-scale-foundation-world-model/
3. Bruce et al., “Genie: Generative Interactive Environments” (Feb 2024): https://arxiv.org/abs/2402.15391
4. Google, “Project Genie” (Jan 2026): https://blog.google/innovation-and-ai/models-and-research/google-deepmind/project-genie/
5. TechCrunch, “Google’s Genie world model can now simulate real streets with Street View” (May 2026): https://techcrunch.com/2026/05/19/googles-genie-world-model-can-now-simulate-real-streets-with-street-view/
6. Google DeepMind, “SIMA 2” (Nov 2025): https://deepmind.google/blog/sima-2-an-agent-that-plays-reasons-and-learns-with-you-in-virtual-3d-worlds/
7. Waymo, “The Waymo World Model” (Feb 2026): https://waymo.com/blog/2026/02/the-waymo-world-model-a-new-frontier-for-autonomous-driving-simulation
8. World Labs, “Marble” (Nov 2025): https://www.worldlabs.ai/blog/marble-world-model
9. World Labs, “RTFM: A Real-Time Frame Model” (Oct 2025): https://www.worldlabs.ai/blog/rtfm
10. World Labs, 2026 funding announcement: https://www.worldlabs.ai/blog/funding-2026
11. World Labs, AMD announcement (Sep 2026): https://www.worldlabs.ai/blog/amd-announcement
12. TechCrunch, “AMD will acquire Fei-Fei Li’s World Labs for $8.2 billion” (Sep 2026): https://techcrunch.com/2026/09/28/amd-will-acquire-fei-fei-lis-world-labs-for-8-2-billion/
13. TechCrunch, World Labs comes out of stealth (Sep 2024): https://techcrunch.com/2024/09/13/fei-fei-lis-world-labs-comes-out-of-stealth-with-230m-in-funding/
14. NVIDIA, Cosmos launch (Jan 2025): https://nvidianews.nvidia.com/news/nvidia-launches-cosmos-world-foundation-model-platform-to-accelerate-physical-ai-development
15. NVIDIA, “What are world foundation models?”: https://blogs.nvidia.com/blog/cosmos-world-foundation-models/
16. NVIDIA, Cosmos 3 launch (May 2026): https://nvidianews.nvidia.com/news/nvidia-launches-cosmos-3-the-open-frontier-foundation-model-for-physical-ai
17. Meta AI, “V-JEPA 2” (Jun 2025): https://ai.meta.com/blog/v-jepa-2-world-model-benchmarks/
18. V-JEPA 2.1 (Mar 2026): https://arxiv.org/abs/2603.14482
19. TechCrunch, “Yann LeCun’s AMI Labs raises $1.03 billion to build world models” (Mar 2026): https://techcrunch.com/2026/03/09/yann-lecuns-ami-labs-raises-1-03-billion-to-build-world-models/
20. TechCrunch, “World model companies are keeping a lot of secrets” (Sep 2026): https://techcrunch.com/2026/09/20/world-model-companies-are-keeping-a-lot-of-secrets/
21. OpenAI, “Video generation models as world simulators” (Feb 2024): https://openai.com/index/video-generation-models-as-world-simulators/
22. OpenAI, “Sora 2 is here” (Sep 2025; product discontinued Apr 2026): https://openai.com/index/sora-2/
23. Wayve, GAIA-3 (Dec 2025): https://wayve.ai/thinking/gaia-3/ and GAIA-4 (Aug 2026): https://wayve.ai/thinking/gaia-4/
24. Runway, “Introducing GWM-1” (Dec 2025): https://runwayml.com/research/introducing-runway-gwm-1
25. Microsoft Research, “Introducing Muse” (Feb 2025): https://www.microsoft.com/en-us/research/blog/introducing-muse-our-first-generative-ai-model-designed-for-gameplay-ideation/
26. Odyssey, “Introducing Odyssey-3” (Sep 2026): https://odyssey.systems/introducing-odyssey-3
27. Liu et al., “World Model on Million-Length Video and Language with Blockwise RingAttention” (Feb 2024): https://arxiv.org/abs/2402.08268
28. Ha & Schmidhuber, “World Models” (2018): https://arxiv.org/abs/1803.10122
29. Hafner et al., “Mastering diverse control tasks through world models,” Nature (2025): https://www.nature.com/articles/s41586-025-08744-2
30. Schrittwieser et al., “Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model” (MuZero): https://arxiv.org/abs/1911.08265
31. Meta, “Introducing Meta Llama 3”: https://ai.meta.com/blog/meta-llama-3/ · Qwen, “Qwen3”: https://qwenlm.github.io/blog/qwen3/
32. Gartner, “Top Strategic Technology Trends for 2026” (Oct 2025): https://www.gartner.com/en/newsroom/press-releases/2025-10-20-gartner-identifies-the-top-strategic-technology-trends-for-2026

Large World Models(LWM): Teaching AI to Imagine the World was originally published in Tutorial Savvy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Anyone who has shipped something built on an LLM has had this happen. You ask a simple question, like which API version added a certain endpoint, and you get a clean, confident, nicely formatted answer.

It’s wrong. The version number doesn’t exist, and the function it cites was never in the SDK. What bothered me most the first time was the tone: the model sounded exactly as sure as it does when it’s right.

For a while I treated hallucination as a bug that the next model release would fix. It hasn’t gone away, and I now think that’s expected. Hallucination is a side effect of how these models are trained and evaluated. Once you see where it comes from, it’s much easier to tell which mitigations actually work and which ones just make you feel better.

This post covers both. First the why, then the fixes that have held up for me and in the research, and a few that are overrated.

First, what are we actually talking about?

“Hallucination” has turned into a catch-all for “the AI said something wrong,” and that’s too broad to be useful. OpenAI’s definition is the one I like best: plausible but false statements generated by language models. The key word is plausible. A typo is a mistake. A made-up but perfectly formatted citation is a hallucination.

So why do models make things up?

Short version: next-word prediction produces hallucinations naturally, and the way we grade models keeps them around. There’s no single cause. Each stage of a model’s life adds a bit of pressure.

1. Next-word prediction has no concept of “false”

During pretraining the model learns one thing: given this text, what comes next? Nobody labels any sentence as true or false. All it ever sees are examples of fluent language.

The OpenAI paper by Kalai et al. uses an analogy that made this click for me. Train an image model to label photos as “cat” or “dog” and it gets very good. Train it to label each pet photo with the pet’s birthday and it will always make mistakes, because nothing in the photo predicts the birthday.

Lots of facts work like birthdays. Spelling and bracket-matching follow consistent patterns, so those errors go away with scale. A specific person’s birth date, a niche API parameter, or the exact title of a paper doesn’t follow a pattern. Either the model saw it enough times to memorise it or it didn’t.

The paper makes this concrete. If 20% of birthday facts appear exactly once in the training data, you should expect a base model to hallucinate on at least 20% of birthday questions. So hallucination isn’t a mysterious glitch. It’s a predictable statistical error rate.

2. The scoreboards reward bluffing

This is the part people usually miss. Think back to multiple-choice exams with no negative marking. If you don’t know the answer, you guess, because a blank is guaranteed zero and a guess sometimes pays off.

Models get graded the same way. Most benchmarks score plain accuracy, so a model that says “I don’t know” scores the same as one that’s wrong. Over thousands of questions, the model that always guesses climbs the leaderboard. Train against that leaderboard and you’re effectively teaching the model to bluff.

OpenAI’s own numbers show this clearly. On the SimpleQA benchmark, the older o4-mini edges out the newer model on accuracy, 24% vs 22%. But it answers almost everything, so it’s wrong 75% of the time. The newer model says “I don’t know” on about half the questions and cuts its error rate to 26%.

An accuracy-only leaderboard would rank o4-mini higher. As a user, I’d much rather have the other one.

The math behind this is simple. Suppose a correct answer scores 1, an abstention scores 0, and a wrong answer costs λ points. If the model’s confidence is p, answering is worth taking only when:

p \cdot 1 - (1 - p) \cdot \lambda > 0 \quad \Longleftrightarrow \quad p > \frac{\lambda}{1 + \lambda}

With plain accuracy scoring, λ = 0 and the threshold is zero, so guessing always pays. Set λ = 1 and the model should answer only when it’s more than 50% sure. Set λ = 3 and the bar is 75%. That’s the fix Kalai et al. argue for: change how the main benchmarks are scored, rather than adding yet another hallucination benchmark that nobody optimises for.

3. Fine-tuning on facts the model doesn’t know

Gekhman et al. (EMNLP 2024) ran a controlled experiment and found that fine-tuning examples with new facts are learned much more slowly than examples the model already agrees with. Once those new facts are learned, the model’s tendency to hallucinate goes up roughly linearly.

Their explanation is that models get most of their factual knowledge in pretraining. Fine-tuning mostly teaches them how to use what they already know. If you push brand-new facts in through SFT, what the model really learns is “answer confidently even when you don’t actually know.”

4. Decoding and context

The last push comes at inference time. Sampling with temperature above zero will sometimes pick a lower-probability token, and every token after that is conditioned on it. Zhang et al. call this hallucination snowballing. They found GPT-4 could recognise 87% of its own mistakes when asked separately, yet it still defended them once they were in the answer.

Context causes problems too. If the retrieved documents are missing the answer, or the answer sits in the middle of a long prompt, the model falls back on memory. Liu et al. showed that accuracy drops noticeably when the relevant passage is in the middle of a long context rather than at the start or end.

Mitigations that actually help

Nothing gets you to zero. What works in practice is layering: give the model the facts, make abstaining a legitimate option, then check the output before a user sees it. Here’s the pipeline I’d build today:

Each step deals with one of the causes above. Here they are in order of how much they bought me for the effort.

1. Ground the model in real sources (RAG, done properly)

If pretraining can’t reliably store rare facts, don’t ask the model to recall them. Retrieve them. The original RAG paper by Lewis et al. showed that pairing a generator with a retrieval index gives “more specific, diverse and factual language” than a model working from memory alone.

What the diagrams on vendor slides leave out is that retrieval quality is the ceiling. If the right chunk isn’t retrieved, the model will still produce an answer, just a made-up one. The details that made the biggest difference for me:

• Measure retrieval separately. Track recall@k on a labelled set before you look at a single generated answer.
• Put the best evidence at the start or end of the context, not buried in the middle.
• Say it explicitly: “Answer only using the sources below. If they don’t contain the answer, say so.”
• Require citations per claim, so every sentence points back to a chunk you can check.

2. Make “I don’t know” a first-class answer

This follows straight from the scoring argument. If your prompt, your eval, and your product all treat abstaining as failure, you’ll get bluffing. So:

• Tell the model in the system prompt that abstaining is acceptable, and show an example of it doing so.
• Score your internal evals with a penalty for wrong answers (the λ from earlier), not plain accuracy.
• Design the UI so “I couldn’t find this in our docs, here’s who to ask” is a normal outcome, not an error screen.

3. Detect uncertainty by sampling

This is my favourite technique because it’s so simple. Ask the same question several times at a non-zero temperature. If the model knows the answer, the samples agree. If it’s guessing, they scatter.

Farquhar et al. (Nature, 2024) formalised this as semantic entropy. You sample several answers, cluster them by meaning rather than exact wording (“Paris” and “It’s Paris, France” count as the same answer), and compute entropy over the clusters. High entropy flags a likely confabulation. It costs N times the inference, so I only run it on high-stakes questions.

4. Verify after generating, independently

Chain-of-Verification (CoVe) from Meta works in four steps: draft an answer, plan verification questions, answer those questions independently, then write a revised final answer.

The word “independently” is doing the heavy lifting. Remember the snowballing result: once a model has committed to a claim, it tends to defend it. Answering the checks in a fresh context, without the draft visible, breaks that loop. In RAG systems I do a lighter version: a second, cheaper call checks each cited claim against its source chunk and drops the claims that aren’t supported.

5. Hand deterministic work to tools

Dates, arithmetic, ID lookups, current prices, and user permissions shouldn’t be generated from memory. Give the model a tool and have it call the tool. A function call either returns the right value or fails with an error. It never invents a plausible number.

6. Fine-tune for behaviour, not facts

Given the Gekhman result, I’ve stopped trying to put knowledge into weights through SFT. Fine-tuning is great for format, tone, and teaching the model when to abstain. Facts belong in retrieval, where you can update them, cite them, and audit them.

Mitigations that are overrated

Some fixes get recommended a lot and don’t do much.

“Do not hallucinate” in the system prompt. The model has no switch to turn off. Explicit permission to abstain helps, as covered above. A bare instruction not to make things up does almost nothing.

Temperature 0 as a fix. Greedy decoding makes the output repeatable, not correct. A model that’s confidently wrong at temperature 0.7 is usually confidently wrong at 0 too, just the same way every time. You also lose the variation between samples

“Just wait for a bigger model.” Bigger models know more, so they hallucinate less on common facts. But OpenAI’s paper makes a point that stuck with me: some questions can’t be answered, so accuracy will never reach 100%. And knowing your own limits can actually be easier for a small model. Calibration comes from training incentives, not parameter count.

Asking “Are you sure?” in the same chat. Because of snowballing, the model tends to double down, or swing the other way and apologise for a correct answer. Verification works when it runs in a fresh context, not as a follow-up turn.

Treating RAG as done. Retrieval addresses factuality. It does nothing for faithfulness, where the model has the right document and still paraphrases it wrongly. You need the verification step for that.

Checklist

If you’re putting an LLM feature in front of users, run through this list. It’s short on purpose.

• Build an eval set of 100 to 200 real questions, including ones your sources can’t answer
• Score it three ways (correct, wrong, abstained), with a penalty for wrong answers
• Measure retrieval recall on its own, before measuring generation
• Tell the model it’s allowed to abstain, and give it an example
• Require a citation for every factual claim, and check the citations automatically
• Route dates, math, lookups, and permissions through tools, not generation
• Add a verification pass (CoVe-style or a claim-vs-source check) for high-stakes answers
• Use sampling consistency or semantic entropy where the extra cost is worth it
• Log every abstention and flagged answer, since they show you where your knowledge base has gaps
• Design the “I don’t know” state in the UI as carefully as the answer state

References

1. Kalai, Nachum, Vempala & Zhang, Why Language Models Hallucinate (OpenAI, 2025). Also see OpenAI’s summary post, which has the SimpleQA numbers.
2. Huang et al., A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (ACM TOIS, 2024)
3. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020)
4. Farquhar, Kossen, Kuhn & Gal, Detecting hallucinations in large language models using semantic entropy (Nature, 2024)
5. Dhuliawala et al., Chain-of-Verification Reduces Hallucination in Large Language Models (Meta AI, 2023)

Why LLMs Hallucinate (and What Actually Helps) was originally published in Tutorial Savvy on Medium, where people are continuing the conversation by highlighting and responding to this story.

You would have noticed this in claude the same questions get asked twice, the same system prompt gets sent on every single call, and the same large files get read into context over and over just so the model can tell you “yes, this function calls the database.”

Cacheup is a small, dependency-free proxy that sits in front of api.anthropic.com and fixes all three problems at once without touching your prompts by hand or standing up Redis. Point your SDK at http://127.0.0.1:8787 instead of the real endpoint, and it starts saving you tokens immediately.

The core idea: three tiers, each solving a different kind of waste

Cacheup doesn’t try to be one clever cache. It’s three layered defenses, each targeting a different flavor of redundant work:

Tier 1 — Semantic response cache. Before anything hits the network, Cacheup embeds your latest message with a dependency-free hashed n-gram embedding (no model download, no GPU) and compares it against prior requests that shared the exact same model, system prompt, and tools. If cosine similarity clears 0.96, you get the old answer back a near-instant X-Cacheup: hit; layer=semantic. match to identical context is deliberate it stops the cache from confidently returning a stale answer to a differently-configured session.

The exact cache. If there’s no semantic near-match, Cacheup hashes the fully normalized request (whitespace collapsed, keys sorted) into a cache key and checks a layered backend in-memory LRU first, then SQLite. A hit here is a byte-for-byte repeat of a prior call, and it’s promoted back up into memory so the next repeat is even faster.

Tier 2 — Reusable fragments. Register a big, stable block of context once a DB schema, a style guide, a tool definition — and reference it by ID ({"cacheup": {"fragments": ["schema:v3.2"]}}) instead of pasting it into every request. Cacheup splices it into the system prompt server-side and strips the fragment metadata before the request ever reaches Anthropic.

Tier 3 — Native prompt caching. On a real miss, prepare_anthropic_request() marks up to four deterministic cache_control: {"type": "ephemeral"} breakpoints: the first and last system blocks, the last tool definition, and the final content block of the latest message. That's Anthropic's own prompt caching, applied automatically so you don't have to hand-place breakpoints on every call.

The three tiers compose cleanly: a semantic hit skips the network entirely; an exact-cache hit skips it too; a genuine miss still gets to keep its expensive prefix cached upstream. Every response header tells you exactly which layer answered — hit; layer=semantic, hit; layer=memory, hit; layer=sqlite, or miss; source=anthropic — plus an estimated X-Cacheup-Tokens-Saved.

The second half: don’t even let Claude read the haystack

Caching repeated requests is only half the problem. The other half is requests that didn’t need to spend frontier-model context in the first placereading ten files to answer one question, or generating boilerplate test code from a pattern that’s already sitting in the repo. For that, Cacheup ships a local implementation of Spotify’s Portal/Shunt pattern a conservative router that decides, per task, whether the work should stay on Claude or get delegated to a cheaper worker mode.

The policy is deliberately narrow and ordered so the safe default wins ties:

1. Is this an edit? Editing, debugging, architecture, and safety-critical reasoning always stay on Claude no exceptions, checked first.
2. Is the output predictable and reference-driven? Given a spec and a reference file (e.g. “write tests for UserService, modeled on OrderTest.java"), code-writer generates the file — and if you pass a target path, the generated code is written straight to disk and never even enters Claude's context.
3. Is this bulk, low-reasoning I/O? More than one path, or any single file over the configurable SHUNT_MIN_LINES (default 350), routes to bulk-reader, which ingests the files and answers a focused question instead of dumping raw file contents into the model's context window.
4. Otherwise: it stays on Claude. Small or reasoning-heavy work is exactly what the primary model is for.

You can inspect this decision before committing to it via POST /v1/route, which returns the route, the mode, and the plain-English reason — useful for debugging why something got delegated (or didn't). And delegated results are cached too, through the same LayeredCache machinery, so a repeated bulk-read of the same files doesn't re-invoke the Portal CLI.

The rules aren’t hidden in Python eithershunt.yaml declares them as hookable policy (on: Read, on: Bash, on: Edit, etc.), so the same "big reads get delegated, edits stay on Claude" contract can be enforced at the tool-call layer, not just the HTTP layer.

Why the architecture stays boring on purpose

Nothing here requires a vector database or a model download. The semantic tier uses a hashed embedding over SQLite or process memory specifically so the project runs cold, with zero setup.

Try it

python3 -m unittest discover -v
python3 server.py

export ANTHROPIC_API_KEY=sk-ant-...
curl http://127.0.0.1:8787/v1/messages \
  -H 'content-type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{"model":"claude-sonnet-4-5","max_tokens":128,"system":"You are a concise coding assistant.","messages":[{"role":"user","content":"What is dependency injection?"}]}'

Run it twice. The first response comes back X-Cacheup: miss; source=anthropic. The second assuming it's a normalized repeat comes back hit, with a token-savings estimate attached

Cacheup: A Multi-Tiered Cache in Front of Claude, So You Stop Paying for Work You’ve Already Done was originally published in Tutorial Savvy on Medium, where people are continuing the conversation by highlighting and responding to this story.

A new model called Jev bets that most of your “AI decisions” were never language problems in the first place.

Here’s an uncomfortable truth about most “AI-powered” systems: they don’t need an AI that can write. They need one that can judge.

Is this order fraudulent? Which warehouse should fulfill it? Is this API key request risky? Does this search result actually match the query?

None of these need a paragraph. They need a label, fast.

That’s the bet behind Jev, a model TypeSafe AI shipped on September 15, 2026. It can’t hold a conversation. It can’t write code. It won’t generate so much as a sentence. And that’s not a limitation it’s the entire pitch.

The model isn’t broken. It’s just refusing to do a job it was never good at.

The tax we’ve been paying without noticing

Tool calling and structured outputs made LLMs much easier to bolt onto software. But under the hood, nothing changed the model is still generative. Even a one-word answer gets produced token by token, with input cost, generation latency, and output cost stacked on top.

Now picture that inside a loop that calls a model over and over just to pick a next step:

def run_agent(task):
 memory = [task]
 while not finished(memory):
 step = brain.decide(memory) # generation call #1
 outcome = perform(step)
 memory.append(outcome)
 status = brain.decide(memory) # generation call #2, just to check progress
return memory

Two of those calls never needed a written answer just a label. That’s the tax.

So what does it actually do?

Strip away the branding and Jev is a semantic decision engine. You feed it two things:

- State plain text or JSON describing what’s happening right now
- Questions decisions you want made about that state, each with a predefined answer shape

Three answer types, no freeform text allowed:

• Choice (pick one option, probability per option),
• Score (place the input on a scale you define), and
• Noul(a yes/no question that returns a probability instead of a flat boolean).

Here’s a differently-shaped example than a support ticket an incoming order review:

{
 "model": "jev-latest",
 "state": "Order #48213: $2,400 electronics, shipping to a new address, paid with a card added 10 minutes ago.",
 "questions": {
 "likely_fraud": {
 "type": "noul",
 "instructions": "Does this order show signs of card fraud?"
 },
 "fulfillment_center": {
 "type": "choice",
 "instructions": "Which warehouse should ship this order?",
 "criteria": {
 "east": "Closest to East Coast addresses, electronics in stock",
 "west": "Closest to West Coast addresses, limited electronics stock",
 "hold": "Do not ship until reviewed"
 }
 }
 }
}

And the response your code actually acts on:

result = jev.ask(order_payload)
if result["likely_fraud"] > 0.85:
 hold_for_manual_review(order_id)
elif result["fulfillment_center"]["confidence"] < 0.5:
 hold_for_manual_review(order_id)
else:
 ship_from(result["fulfillment_center"]["choice"])

People keep calling this “a smart switch statement.” That’s meant as an insult, but it’s the whole point ordinary code owns the branching. The model just supplies the fuzzy judgment code can’t calculate on its own.

LLM vs. Jev: not a rivalry, a division of labor

TypeSafe says Jev evaluates every declared question about a state in parallel, in one request. The company’s numbers: 70–500ms latency, roughly $0.042 per million input tokens, free output, with headline claims up to 200x faster and 400x cheaper than comparable LLM workflows.

Worth noting: those multiples come from TypeSafe’s own benchmarks. Treat them as a ceiling, not a promise.

The number matters more than the label

This unlocks a simple pattern, shown here with the fraud-check example instead of a ticket:

verdict = jev.ask(order_payload)
  if verdict["likely_fraud"] > 0.9:
   block_order(order_id)
  elif verdict["likely_fraud"] > 0.5:
   flag_for_analyst(order_id)
  else:
   approve_order(order_id)

Thresholds live in your code, not the modelreviewable, testable, adjustable per use case.

“It can’t hallucinate”

Jev genuinely can’t return an answer outside the schema you gave it. But it can absolutely, confidently, land on the wrong valid option.

Type safety guarantees the shape of the answer. It says nothing about whether the answer is right.

Where it actually belongs: next to an LLM, not instead of one

A model-routing example, using a different pair of tiers than the original “fast/powerful” split:

tier = jev.choice(
 state=incoming_request,
 options={
 "light": "Simple lookups, formatting, short factual answers",
 "heavy": "Multi-step reasoning, novel code, ambiguous requests",
 },
)
handler = light_weight_model if tier == "light" else frontier_model
response = handler.run(incoming_request)

And a tool-gating example for an agent about to touch a file system:

risk = jev.score(
 state=f"About to run: {proposed_command}",
 scale=["safe", "reversible", "destructive"],)
if risk["label"] == "destructive" or risk["confidence"] < 0.7:
 request_human_approval(proposed_command)
else:
 execute(proposed_command)

Where it earns its keep today

That maps cleanly onto support/ops triage, search reranking and citation checks, safety screening, bulk classification, and with a caveat real-time interfaces, since Jev only reads text/JSON, never pixels.

Where it’s clearly the wrong tool

No writing, no summarizing, no code generation, no explaining itself. Weak at arithmetic, exact string matching, date math. Struggles with decisions needing several hidden reasoning steps. Can’t pull an unknown value out of nowhere.

And the boring-but-true rule: if an if-statement already solves it, keep the if-statement.

How to roll it out without shooting yourself in the foot

# Shadow mode: log Jev's answer, but let the old system decide
old_result = legacy_router(request)
shadow_result = jev.ask(request)
log_comparison(old_result, shadow_result)
return old_result

Pick one bounded, low-risk decision. Write the rubric first. Collect messy, adversarial examples. Run it quietly in shadow mode. Set thresholds from what you observe. Automate the safest slice first. Version everything like code.

The real takeaway

Jev isn’t interesting because it beats an LLM at writing. It’s interesting because it refuses to try.

Jev: Teaching Software to Make Small Decisions Without Calling in a Language Model was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

A low-latency voice pipeline for local inference, streaming speech recognition, LLM reasoning, and text-to-speech synthesis.

Voice interfaces are becoming one of the most important interaction layers for AI products.

Whether it’s assistants, copilots, smart devices, or customer support tools, real-time voice interaction is moving from novelty to necessity.

But building a fast, reliable, production-quality voice AI system remains difficult. Developers often have to stitch together multiple components across ASR, LLMs, and TTS systems, while depending on vendor-specific ecosystems or closed APIs.

We built Vaani to solve that problem.

Today, we’re releasing Vaani v1.0.0 — the first stable open-source release of our real-time voice AI stack.

Why We Built Vaani

The core challenge in voice AI is latency.

A user expects an assistant to respond quickly, naturally, and continuously. However, many systems are optimized primarily for model quality, rather than the end-to-end responsiveness required for real-time conversations.

We wanted to build a system that:

• Works locally
• Supports real-time streaming
• Is transparent and inspectable
• Keeps the stack open and extensible
• Allows developers to benchmark and tune latency
• Supports both local and remote inference

Vaani is built around that idea.

What Is Vaani?

Vaani is an open-source, real-time voice AI stack designed for end-to-end conversational interaction.

It combines streaming speech recognition, language-model generation, and text-to-speech synthesis into a single streaming pipeline.

At its core:

Microphone → ASR → LLM → TTS → Speaker

The architecture is designed around streaming data rather than traditional batch processing, allowing different stages of the pipeline to work concurrently.

This makes Vaani suitable for:

• Conversational AI assistants
• Voice copilots
• Smart devices
• Local AI applications
• Voice-driven software
• Research and experimentation

What Vaani Includes

Vaani brings together several components required for a complete voice AI system:

• Streaming ASR with VAD-based speech detection
• Partial and final transcript handling
• LLM token streaming with TTFT instrumentation
• Sentence-level TTS chunking
• Low-latency audio playback
• Barge-in and interrupt handling
• gRPC deployment for remote inference
• Benchmarking utilities for latency and throughput analysis

The goal is to make these components work together as a coherent system rather than forcing developers to build the entire orchestration layer themselves.

System Architecture

The architecture is intentionally modular.

Speech input is processed in small audio chunks. A lightweight VAD layer detects speech activity and identifies speech boundaries.

The resulting audio stream is passed to the ASR layer, which produces partial and final transcripts.

These transcripts are then converted into prompts for the LLM. Instead of waiting for the complete response, Vaani streams generated tokens as soon as they become available.

The generated response is then passed to the TTS layer, which converts the text into smaller audio chunks that can be played back progressively.

This creates a continuous pipeline:

Audio Input → VAD → ASR → LLM → TTS → Audio Output

The streaming architecture is what makes the system suitable for real-time interaction.

Performance Goals

Vaani was designed around a simple principle:

The most important metric isn’t just what the model says — it’s how quickly the user hears it.

Our target latency profile is:

ComponentTargetVAD + ASR boundary~20–80 msLLM TTFT~60–180 msTTS first chunk~40–120 msPlayback output~6–20 ms

Overall local first response<500 ms

These targets are intended as an optimization baseline for conversational systems where responsiveness matters as much as correctness.

Benchmark Snapshot

We benchmarked Vaani with 10 concurrent clients.

First Audio Latency — P50

ModelLatencyVaani1008.65 msElevenLabs Conversational*680 msOpenAI Realtime1089 ms

First Audio Latency — P95

ModelLatencyVaani1009.66 msElevenLabs Conversational*1560 msOpenAI Realtime1630 ms

Throughput

ModelResponses/secVaani 1.97 ElevenLabs Conversational*2.40 OpenAI Realtime1.70

The benchmark demonstrates that Vaani can operate competitively within the conversational voice AI space while remaining open, inspectable, and developer-controlled.

More importantly, the benchmark gives us a measurable baseline for future optimization.

These results should not be interpreted as a claim that Vaani is universally better than commercial voice platforms. Different systems can use different hardware, regions, models, network conditions, audio configurations, and measurement methodologies.

Our goal is to establish a transparent baseline and continuously improve it.

Why This Release Matters

Vaani v1.0.0 is more than an experiment.

It provides a foundation for developers and researchers who want to:

• Run a voice AI stack locally
• Benchmark latency and throughput
• Experiment with different models
• Build custom voice agents
• Deploy remote inference through gRPC
• Modify and extend the entire pipeline

The release includes:

• Local deployment mode
• gRPC server/client mode
• Build validation
• Test coverage
• Benchmarking utilities
• Package metadata
• Modular inference components

What’s Next?

Vaani v1.0.0 is only the beginning.

Our upcoming focus areas include:

• Further latency optimization
• Better deployment ergonomics
• Greater model flexibility
• Improved multi-turn memory
• More advanced profiling and benchmarking
• Broader hardware compatibility
• Improved streaming performance

The long-term goal is to make Vaani a flexible building block for the next generation of voice-native AI applications.

Introduction

The AI industry is obsessed with models.Every week a new model is released. Bigger context windows. Better reasoning. Faster inference. Lower costs.

Yet most AI systems fail for a surprisingly simple reason:

The model doesn’t know enough about your business.

An LLM can explain quantum mechanics, write production-grade code, and solve complex reasoning problems, but ask it:

• What was the latest update to our internal policy?
• Which API endpoint was deprecated last week?
• What customer issues were reported this month?
• What does our private documentation say about deployment?

And suddenly the model becomes almost useless.

The problem isn’t intelligence. The problem is knowledge.

Modern organizations generate enormous amounts of information:

• Documentation
• Wikis
• Source code
• Databases
• Customer conversations
• PDFs
• Slack messages
• Meeting notes
• APIs
• Analytics
• Product specifications

Unfortunately, this information is fragmented across dozens of systems.

This fragmentation creates a massive gap between organizational knowledge and AI capabilities.

The solution is a Harness Layer.

A Harness Layer acts as the intelligence infrastructure that continuously gathers, transforms, validates, enriches, stores, and serves knowledge to AI applications.

Instead of asking:

“How do we make the model smarter?”

A Harness Layer asks:

“How do we make knowledge accessible, trustworthy, and continuously updated?

Understanding the Core Problem

An organization with:

• 500,000 documents
• 40 Git repositories
• 15 databases
• 20 SaaS platforms
• 10 years of historical data

An AI assistant connected directly to these systems faces several challenges.

Challenge 1: Format Diversity

Knowledge exists in many formats:

PDF
DOCX
TXT
Markdown
JSON
XML
CSV
SQL Tables
REST APIs
Source Code
Images
Videos

Every format requires different processing logic.

Challenge 2: Knowledge Fragmentation

The same information may exist in multiple locations.

Example:

Feature Documentation → Notion

API Documentation → GitHub

Release Notes → Confluence

Customer Issues → Zendesk

Product Specs → Google Drive

Without consolidation, AI receives incomplete context.

Challenge 3: Constant Change

Knowledge never remains static. Every day, documents change, APIs evolve, new code is pushed, policies are updated, and teams create new content. As a result, a static knowledge base becomes obsolete quickly.

Challenge 4: Information Quality

Not all knowledge is reliable. Some content can be outdated, duplicated, incorrect, incomplete, or contradictory. Without proper validation, AI systems amplify these problems rather than resolving them.

What Exactly Is a Harness Layer?

A Harness Layer is a centralized intelligence system responsible for converting raw organizational data into AI-ready knowledge.

Think of it as:

Raw Data
     ↓
Harness Layer
     ↓
Trusted Knowledge
     ↓
AI Applications

The Harness Layer does not replace databases.It does not replace vector stores.It does not replace AI models.Instead, it coordinates all of them.

Its job is to ensure that AI always receives the best possible information.

The Anatomy of a Harness Layer

A production-grade Harness Layer typically consists of eight major subsystems.

1. Connector Framework

The Connector Framework is responsible for collecting data.

It acts as the entry point into the knowledge ecosystem.

Examples:

GitHub
GitLab
Bitbucket
Notion
Confluence
Google Drive
Dropbox
MongoDB
PostgreSQL
MySQL
Snowflake
REST APIs
GraphQL APIs
Local Files
S3 Buckets

A connector abstracts away platform-specific complexities.

Instead of writing separate logic for every source, the Harness Layer standardizes ingestion.

Example Interface:

class Connector:
    def connect()
    def fetch()
    def sync()
    def validate()

Every connector follows the same contract.

2. Parsing Layer

Raw data is rarely usable because a PDF, JSON file, or Git repository is not knowledge in itself. Knowledge emerges only after parsing, where the parsing layer extracts meaningful content from these raw formats.

Examples:

PDF Parser

Extracts:

• Headings
• Sections
• Tables
• Metadata

Markdown Parser

Extracts:

• Titles
• Hierarchies
• Code blocks
• Links

Source Code Parser

Extracts:

• Classes
• Functions
• Comments
• Dependencies

Database Parser

Extracts:

• Records
• Relationships
• Schemas

The goal is to transform every source into a unified representation.

3. Normalization Layer

Different systems describe identical concepts differently.

Example:

Customer
Client
Consumer
Account Holder

All may represent the same entity.

Normalization creates consistency.

Tasks include:

• Deduplication
• Standard naming
• Metadata mapping
• Relationship mapping

Without normalization, retrieval quality collapses.

4. Knowledge Enrichment Layer

This is where intelligence begins. The enrichment layer enhances raw content.

Techniques include:

Summarization

Converts long documents into concise descriptions.

Classification

Assigns categories.

Example:

Engineering
Finance
Legal
HR
Marketing

Entity Extraction

Identifies:

• Companies
• People
• Technologies
• Products

5. Validation Layer

A critical but often ignored component.

Validation ensures knowledge quality.

Checks include:

Freshness Validation

Is content outdated?

Integrity Validation

Is content corrupted?

Duplicate Detection

Has this already been indexed?

Reference Validation

Do linked resources still exist?

Trust Scoring

Assign confidence levels.

Example:

Official Documentation = 100
Internal Wiki = 80
Community Notes = 60
Archived Content = 20

Retrieval can prioritize high-trust content.

6. Scheduling & Orchestration Layer

Knowledge pipelines must run continuously.

Common schedules:

Every Minute
Every 5 Minutes
Hourly
Daily
Weekly
Monthly

Advanced systems support:

• Event-based triggers
• Webhooks
• Real-time streaming

Popular orchestration tools:

• Airflow
• Temporal
• Dagster
• Prefect

7. Knowledge Storage Layer

After processing, knowledge is stored.

Most Harness Layers use multiple storage systems.

Object Storage

Stores raw files.

Example:

S3
MinIO
GCS

Relational Storage

Stores metadata.

Example:

PostgreSQL
MySQL

Vector Storage

Stores embeddings.

Example:

Pinecone
Qdrant
Weaviate
Milvus
pgvector

Graph Storage

Stores relationships.

Example:

Neo4j
TigerGraph
ArangoDB

8. Retrieval Layer

The retrieval layer is the heart of AI interaction.

It determines:

What knowledge should be sent to the model?

Techniques include:

Keyword Search

Traditional text matching.

Semantic Search

Vector similarity.

Hybrid Search

Keyword + Semantic.

Metadata Filtering

Example:

Department = Engineering
Created After = Jan 2026
Author = Platform Team

Reranking

Uses a smaller model to improve relevance.

This dramatically improves answer quality.

How Knowledge Bases Fit Into a Harness Layer

Many people think:

Knowledge Base = Harness Layer

This is incorrect.

A Knowledge Base is only one component.

Relationship:

Sources
   ↓
Harness Layer
   ↓
Knowledge Base
   ↓
Retrieval
   ↓
LLM

The Harness Layer builds and maintains the Knowledge Base.

The Knowledge Base stores the processed knowledge.

Building a Strong Knowledge Layer

The strongest Harness Layers share six characteristics.

Characteristic 1: Source Agnostic

Adding a new connector should take hours, not weeks.

Characteristic 2: Self-Healing

Failures automatically recover.

Characteristic 3: Incremental Updates

Only changed data gets reprocessed.

Characteristic 4: Versioned Knowledge

Historical snapshots remain available.

Characteristic 5: Trust Scoring

Every piece of knowledge has confidence levels.

Characteristic 6: Observability

Everything is measurable.

Metrics:

• Sync Success Rate
• Retrieval Accuracy
• Knowledge Freshness
• Query Latency
• Coverage Percentage

Reference Open-Source Architecture

harness-layer/
│
├── connectors/
│   ├── github/
│   ├── notion/
│   ├── postgres/
│   └── drive/
│
├── parsers/
│   ├── pdf/
│   ├── markdown/
│   ├── code/
│   └── json/
│
├── enrichers/
│   ├── embeddings/
│   ├── summaries/
│   ├── classifiers/
│   └── entities/
│
├── validators/
│   ├── freshness/
│   ├── trust/
│   └── duplicate/
│
├── schedulers/
│
├── retrieval/
│
├── vectorstore/
│
├── api/
│
└── dashboard/

Introduction

Imagine you are shown thousands of images of cats and dogs.

A traditional computer program would require manually defining rules:

• Cats have pointy ears.
• Dogs have longer snouts.
• Cats usually have vertical pupils.

The problem is obvious: reality is messy. Some dogs have pointy ears. Some cats have round faces. Lighting, angles, and backgrounds constantly change.

Modern AI solves this differently. Instead of programming rules, we allow models to learn representations of the world.

Representation Learning is arguably the most important idea in modern machine learning. It is the reason deep learning surpassed traditional machine learning and enabled systems like ChatGPT, image generators, recommendation engines, autonomous vehicles, and scientific discovery models.

At its core, representation learning answers a simple question:

How can a machine automatically discover useful features from raw data?

The answer transformed artificial intelligence.

What Is Representation Learning?

A representation is simply a way of describing information.

For example, consider the word:

Apple

Humans immediately understand multiple meanings:

• Fruit
• Technology company
• Brand
• Food item

Computers see:

A p p l e

just characters.

Representation learning converts raw data into mathematical forms that capture meaning.

Instead of storing:

Apple

the model may represent it as:

[0.82, -0.14, 1.23, 0.56, ...]

This vector contains learned information about the concept.

The model learns:

• Apple is related to fruit
• Apple is related to technology
• Apple is similar to Microsoft
• Apple is different from Banana in some contexts

These learned vectors are called embeddings.

The Evolution of Feature Engineering

Before deep learning, machine learning depended heavily on handcrafted features.

Traditional Approach

Suppose we want to classify emails as spam.

Engineers manually create features:

contains_discount
contains_offer
contains_money
number_of_links
email_length

The model learns from these predefined features.

Pipeline:

Raw Email
    ↓
Manual Feature Engineering
    ↓
Machine Learning Model
    ↓
Prediction

Performance is limited by human creativity.

Deep Learning Approach

Modern systems learn features automatically.

Pipeline:

Raw Email
    ↓
Neural Network
    ↓
Learned Representation
    ↓
Prediction

The model discovers patterns humans might never think of.

This shift from manual feature engineering to automatic representation learning fueled the AI revolution.

The Geometry of Intelligence

A surprising fact about modern AI:

Intelligence often emerges from geometry.

Models transform information into points inside high-dimensional spaces.

Consider three words:

King
Queen
Prince

After training:

King   → (2.1, 1.3, ...)
Queen  → (2.0, 2.4, ...)
Prince → (1.7, 1.0, ...)

Distances now encode meaning.

Words with similar meanings appear close together.

Visualization:

King -------- Queen
  |
  |
Prince

while unrelated concepts remain far away.

King -------------------- Refrigerator

The model has learned a semantic space.

This space becomes the foundation of reasoning.

Embeddings: The DNA of Modern AI

Embeddings are dense numerical representations learned from data.

Think of them as compressed knowledge.

Word Embeddings

One of the first breakthroughs was:

• Word2Vec
• GloVe
• FastText

Example:

King - Man + Woman ≈ Queen

The model learned gender relationships purely from text.

Mathematically:
v(King)-v(Man)+v(Woman) ~=v(Queen)

This was one of the first demonstrations that neural networks could capture abstract concepts.

Sentence Embeddings

Instead of individual words:

I love machine learning

becomes:

[0.21, -0.87, 0.55, ...]

Now entire meanings can be compared.

Examples:

"I like AI"

"I enjoy artificial intelligence"

produce vectors that are extremely close.

Applications:

• Semantic search
• Document retrieval
• Chatbots
• RAG systems

Image Embeddings

Images can also become vectors.

Cat Image

To

[0.67, -0.42, 1.54, ...]

Images of cats cluster together.

Visualization:

Cats
 ● ● ● ●

Dogs
         ● ● ● ●

Cars
                 ● ● ●

The model learns visual concepts automatically.

Latent Spaces: Where Meaning Lives

One of the most fascinating ideas in AI is the latent space.

A latent space is a compressed hidden representation learned by a model.

Imagine a face generator.

Every face:

• Age
• Gender
• Hair style
• Expression
• Lighting

can be encoded into a vector.

Face
 ↓
Encoder
 ↓
Latent Vector
 ↓
Decoder
 ↓
Reconstructed Face

The latent vector captures essential information.

Autoencoders

Autoencoders are designed specifically for learning latent spaces.

Architecture:

Input
  ↓
Encoder
  ↓
Latent Space
  ↓
Decoder
  ↓
Output

Goal:
Input ~= Output

The bottleneck forces compression.

The network learns the most important features.

Variational Autoencoders (VAEs)

VAEs make latent spaces smooth and continuous.

Instead of encoding:

Face A → Point A
Face B → Point B

the model learns distributions.

This enables generation.

Move smoothly between points:

Young Face
      ↓
Middle Age Face
      ↓
Old Face

AI-generated images emerged from this idea.

Contrastive Learning: Learning Through Comparison

Humans learn by comparing.

A child learns dogs because they are:

• Similar to other dogs
• Different from cats

Contrastive learning follows the same principle.

Core Idea

Bring similar examples together.

Push different examples apart.

Mathematically:
d(z_i,z_j) -> 0

for positive pairs.
d(z_i,z_k) -> Infinity

Example

Take an image:

Dog

Create two augmented versions:

• Rotated
• Cropped

The model sees both.

Since they originate from the same image:

Positive Pair

Representations become closer.

Different images:

Dog
Car

become farther apart.

SimCLR

One landmark method was:

SimCLR

Pipeline:

Image
 ↓
Augmentation
 ↓
Encoder
 ↓
Projection Head
 ↓
Contrastive Loss

No labels required.

The model learns useful features automatically.

This dramatically reduced dependence on labeled datasets.

Self-Supervised Learning

Representation learning evolved into self-supervised learning.

The idea:

Create labels automatically from data itself.

Examples:

BERT

Mask words:

The cat sat on the [MASK]

Predict:

mat

GPT

Predict next token:

Artificial Intelligence is "transforming"

CLIP

Train on image-text pairs.

Image ↔ Caption

The model learns a shared representation space.

Images and text become directly comparable.

This powers modern multimodal systems.

Representation Learning Inside ChatGPT

Large Language Models are massive representation learners.

When ChatGPT processes:

Explain quantum mechanics

it converts text into embeddings.

Pipeline:

Text
 ↓
Tokenization
 ↓
Embeddings
 ↓
Transformer Layers
 ↓
Higher-Level Representations
 ↓
Prediction

Each transformer layer learns increasingly abstract concepts.

Early layers:

Grammar

Middle layers:

Syntax

Higher layers:

Reasoning
Knowledge
Concept Relationships

The model continuously refines representations before generating output.

Representation Learning in Search Engines

Modern search engines no longer rely purely on keywords.

Query:

best laptop for machine learning

Traditional search:

Keyword Matching

Modern search:

Query Embedding
 ↓
Vector Search
 ↓
Semantic Matching

Now the engine understands intent.

This is why searching:

car insurance

can return pages discussing:

vehicle coverage

even without exact keyword overlap.

Representation Learning in Recommendation Systems

Recommendation systems are giant embedding machines.

User:

Watched:
Interstellar
Inception
The Matrix

Model learns:

User Vector

Movies become vectors too.

Recommendations emerge from vector similarity.

Similarity(User,Movie)

Higher similarity:

Recommend

Lower similarity:

Ignore

This drives:

• Netflix
• YouTube
• Spotify
• Amazon

recommendations.

Representation Learning in Healthcare

Healthcare AI increasingly relies on learned representations.

Examples:

Medical Imaging

MRI scans:

Raw Pixels
 ↓
CNN
 ↓
Latent Representation
 ↓
Disease Detection

Drug Discovery

Protein structures become embeddings.

Researchers compare:

Protein A
Protein B

inside learned biological spaces.

This dramatically accelerates drug development.

Companies such as DeepMind leveraged representation learning in systems like AlphaFold to predict protein structures with unprecedented accuracy.

The Future: Foundation Models Learn Universal Representations

The trend is clear.

AI is moving toward universal representations.

Modern foundation models learn:

• Language representations
• Image representations
• Audio representations
• Video representations
• Scientific representations

inside a shared latent space.

Future systems may learn representations for:

• Biology
• Physics
• Robotics
• Human behavior

allowing transfer of knowledge across domains.

Key Takeaways

Representation Learning is the process of automatically discovering useful features from raw data.

It powers:

• Large Language Models
• Recommendation Systems
• Search Engines
• Computer Vision
• Medical AI
• Autonomous Systems

The fundamental building blocks are:

1. Embeddings — numerical representations of meaning.
2. Latent Spaces — compressed worlds where concepts live.
3. Contrastive Learning — learning through similarity and difference.
4. Self-Supervised Learning — creating supervision from unlabeled data.
5. Feature Extraction — discovering useful patterns automatically.

In this blog, we’ll build a Behavioral Health Emotion Memory System using:

• Machine Learning (emotion embeddings)
• Qdrant (vector memory database)
• NLP models (emotion detection)
• RAG-style retrieval (context-aware responses)

This architecture is highly relevant for:

• Mental health bots
• Therapy assistants
• Behavioral AI systems
• Reflective journaling apps

The Core Problem: Emotional Context Loss in AI Systems

Traditional AI chat systems store chat logs as plain text. But emotional signals like:

• “I feel anxious about exams”
• “I’m stressed and overwhelmed”
• “I can’t sleep lately”

are treated as just text — not psychological context.

This leads to:

• Generic responses
• No emotional continuity
• Poor personalization

We solve this using ML-based emotion embeddings + Qdrant semantic memory.

System Architecture (Behavioral AI + Vector Memory)

Implementation

Step 1: Using ML for Emotion Classification

We first integrate a fine-tuned emotion detection model (NLP-based).

from transformers import pipeline

emotion_model = pipeline(
    "text-classification",
    model="j-hartmann/emotion-english-distilroberta-base",
    return_all_scores=True
)

text = "I feel extremely anxious about my future"
emotion_scores = emotion_model(text)
print(emotion_scores)

This ML model detects nuanced emotional states like:

• Anxiety
• Sadness
• Joy
• Fear
• Anger

Now we convert this emotional text into embeddings for vector storage.

Step 2: Generating Emotion-Aware Embeddings

Instead of generic embeddings, we use sentence transformers to capture psychological semantics.

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")

entry = "Feeling overwhelmed with workload and deadlines"
vector = embedder.encode(entry)
print(len(vector))  # 384-dimensional vector

These vectors represent emotional meaning, not just keywords.

Step 3: Setting Up Qdrant as an Emotional Memory Database

Now we integrate Qdrant as a long-term behavioral memory layer.

from qdrant_client import QdrantClient
from qdrant_client.http.models import VectorParams, Distance

client = QdrantClient(":memory:")

client.create_collection(
    collection_name="emotion_memory",
    vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)

Why Qdrant?

• Fast similarity search
• Payload filtering (emotion tags)
• Scalable for long-term user memory
• Perfect for RAG-based therapy agents

Step 4: Storing Emotional Events with ML Metadata

We now store emotional entries along with detected emotions.

from uuid import uuid4

journal_entry = "I feel lonely and mentally drained today"
vector = embedder.encode(journal_entry)
emotion = emotion_model(journal_entry)[0][0]["label"]

client.upsert(
    collection_name="emotion_memory",
    points=[
        {
            "id": str(uuid4()),
            "vector": vector,
            "payload": {
                "text": journal_entry,
                "emotion": emotion
            }
        }
    ]
)

This creates a personalized emotional knowledge base.

Step 5: ML-Powered Emotional Retrieval (Context-Aware Search)

Now when a user writes a new entry, the system retrieves similar emotional history.

query = "I feel stressed and exhausted"
query_vector = embedder.encode(query)

results = client.search(
    collection_name="emotion_memory",
    query_vector=query_vector,
    limit=3
)

for r in results:
    print(r.payload)

Even if the wording changes, Qdrant retrieves emotionally similar past states using ML embeddings.

This is extremely powerful for mental health AI.

Step 6: Blending with RAG for Therapeutic AI Responses

Now we combine retrieval + LLM:

Workflow:

1. User expresses emotion
2. ML detects emotional state
3. Qdrant retrieves similar emotional memories
4. LLM generates empathetic, context-aware response

Example prompt injection to LLM:

User Current Emotion: Anxiety
Past Emotional Patterns: [Stress during exams, sleep issues]
Generate a supportive response.

This creates a psychologically aware AI agent instead of a generic chatbot.

You can reach out to me if you have any queries

LinkedIn: https://www.linkedin.com/in/rithvik1709/