You may or may not have heard of decision models. They’re a newer kind of model, and they do a narrower job than the LLM you might already have judging things in your app. You give one a question, some state, and the answers it’s allowed to give. It hands back one of those answers with probabilities attached. Here’s an example adapted from the support-ticket routing example in TypeSafe’s quickstart. The request asks Jev which team should handle a customer’s message. The state is the message itself, and the criteria list the teams Jev can choose from, each with a short description.

{
  "model": "jev-latest",
  "state": "Your app keeps logging me out, and now my card shows a charge I don't recognize.",
  "questions": {
    "department": {
      "type": "choice",
      "instructions": "Which team should handle this",
      "criteria": {
        "billing": "Payment or subscription issues",
        "technical": "Bugs or integration problems",
        "sales": "Pricing or account questions"
      }
    }
  }
}

Jev’s response comes back as structured data rather than a wall of text, or text-formatted JSON that you need to cleanse and parse.

{
  "model": "jev-1.13.0",
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "confidence": 0.77,
      "probabilities": { "billing": 0.85, "technical": 0.15, "sales": 0.0 }
    }
  }
}

The answer is billing. Along with the answer, you get a probability for each option and a confidence score between zero and one.

The big draw here is speed. A decision model doesn’t have to write anything out. It just picks from the answers you gave it, so there’s no long response to generate and no text to parse on the other end. Those savings add up fast when an agent has to make a bunch of small calls on the way to a result.

Given the claims, I wanted to see how Jev, TypeSafe’s first System One model, stacked up against an LLM making the same call inside a real agent, so I built a small app to find out.

Decisions in Embabel Agent

Embabel Agent is a JVM framework for building agents out of goals and actions. It plans a path through your actions based on the types they take and return. It can now call decision models too. An action asks a typed question through a DecisionService and gets back a typed answer.

A layover in Tampa

The test app is Bayes of Our Lives. It’s a stripped-down spin-off of Tripper, Embabel’s travel planner example. Tripper plans a whole trip. Bayes plans a few hours, because nobody needs a seven-day travel agent to figure out that lunch is missing.

The scenario is a long layover in Tampa. You’ve got from about 7 a.m. to 2 p.m., and you’d rather see some of the city than sit at the gate. It’s you and a partner, one of you is into art, and you want breakfast and lunch. You’re both up for seafood or Cuban food, and anything but Italian.

An LLM does the bulk of the work. It picks venues from a map search, researches them, and drafts a couple of itineraries. Then the app checks every draft against drive times, meal windows, and opening hours, and marks anything that doesn’t fit as invalid. It even leaves ten minutes on every leg for parking, which is more planning than I usually do. The plans that pass go to the decision model, which makes the calls at the end. It picks the best plan, says whether that plan covers what the traveler asked for, and rates how confident it is in the pick.

Role call

One thing I’ve always liked about Embabel is that your code never has to name a model. You give each model a role, like “cheapest” or “best,” map the roles to real models in configuration, and your code asks for the role. Embabel’s docs call it role-based model selection, and they point out it’s good practice to keep your code decoupled from specific models this way. Decision models work the same way now. Each engine in the app gets a role, and the roles are just properties. Here are the Jev and OpenAI lines from the app’s configuration.

embabel.agent.platform.models.openai.api-key=${OPENAI_API_KEY:}
embabel.agent.platform.models.typesafe.api-key=${TYPESAFE_API_KEY:}
embabel.models.decision.roles.jev=jev-latest
embabel.models.decision.roles.openai=gpt-4.1
embabel.models.classification.roles.jev=jev-latest
embabel.models.classification.roles.openai=gpt-4.1

jev points at Jev, and openai points at a chat model that Embabel turns into a decision service for you. Each role is set for both decisions and classifications, so one model answers everything for a given run. Inside the action, Bayes looks the model up by role with context.ai().decisions().byRole(role). Switching models is a property change.

Call and response

Bayes asks three questions in one decision request and then a fourth as a classification request. The coverage question asks whether a plan covers every requested stop and meal. The next question asks which plan fits best, or whether to go research more. The confidence question rates the best plan as weak, possible, or strong. Readiness classifies the whole set of plans.

The next question is the interesting one, because its answers may change every time. The options are whichever plans passed validation, plus an escape hatch. In the code that list is valid. This method just builds the question, so no model gets called yet.

static ChoiceQuestionSpec next(List<Itinerary.Alternative> valid) {
    var question =
            Questions.named("next")
                    .choice(
                            """
                            Which alternative best fits the traveller's profile, liked cuisines and timing? ...\
                            """);
    valid.forEach(a -> question.option(a.id(), a.title() + " (" + a.status() + ")"));
    return question.option("research", "Resolve material questions before choosing").build();
}

A plan that failed validation never makes the list, so neither model can pick it, and no LLM gets to talk a bad plan into a good one by sounding pleased with itself. Here’s where the model actually gets called, whichever one the role points at. Asking all four questions looks like this.

DecisionResponse response =
        decisions.ask(
                DecisionRequest.of(
                        input, DecisionSpec.of(COVERAGE, next, CONFIDENCE)));
...
ClassificationResult readiness =
        classifier.classify(ClassificationRequest.of(input, READINESS));

The answers come back typed. Each one is a sealed result with answered, inconclusive, and failure cases, so a switch over it is exhaustive, and the compiler tells you if you miss one. Here’s the coverage answer.

private static String coverage(PropositionResult result) {
    return switch (result) {
        case PropositionResult.Answered answered -> Boolean.toString(answered.getAnswer());
        case PropositionResult.Inconclusive inconclusive -> "inconclusive";
        case PropositionResult.Failure failure -> failure("coverage", failure.getReason());
    };
}

Bayes never has to parse text or guess what the model meant. If the model picks something that wasn’t on the list, Bayes records that as a failure.

Time will tell

Bayes times the decision call and the classification call separately, so it’s easy to compare the two models on just the part they’re responsible for. I ran the same Tampa request once with each.

Model    Decision call   Classification call
Jev      416 ms          141 ms
OpenAI   736 ms          603 ms

Jev was faster on both calls, about 1.8 times on the decision and about 4 times on the classification. These are remote calls, so the numbers include the network, and one run each is a small sample. Still, the gap was clear for the exact same questions.

Both models picked the same plan, which was breakfast at Fast Jacks Breakfast, the Tampa Museum of Art, and a seafood lunch at Juicy Seafood. Not much of a cliffhanger, but nothing Italian made the cut.

Bayes also uses Dice to remember interests and pace between outings, but that’s a story for another episode.

Your turn

The code is on GitHub at jimador/bayes-of-our-lives, and the README walks through setup. You’ll need your own OpenAI and TypeSafe keys. Swap in whatever city your layover lands you in, flip the role, and watch the clock.

Closing credits

The cast and crew for this episode, in case you want to follow along at home.

• Starring Jev (TypeSafe docs) and ChatGPT
• Directed by Embabel Agent 1.5.4-SNAPSHOT (user guide)
• Supporting cast Dice 0.4.0 (release notes)
• Special thanks to Tripper for the location
• Try it at home with Bayes of Our Lives

Like sands through the hourglass, so are the LLM API calls of our lives.

Bayes of Our Lives: Experimental Decision Models in Embabel Agent was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.

Embabel Agentic Framework has just reached a long-awaited milestone release, 1.0.0.

You may take a look at Embabel's blogs and at separate blogs published by Embabel founder Rod Johnson, creator of the Spring Framework.

Also, Craig Walls and Jettro Coenradie recently published a few blog posts on Embabel.

Please refer to the User Guide for features, solutions, and APIs.

Please try Embabel HUB, an Embabel-powered learning tool. You can use voice as well as a communication channel with HUB. Choose one of the built-in personalities, or make your own. HUB uses many of the features covered in this blog: ToolishRag, chat-store, ONNX local embeddings, and more.

Here are some of the recently added features and other features not fully covered in previous blogs.

Structured Output

• Best-Effort (createObjectIfPossible) - returns null instead of throwing when the LLM response cannot be parsed; the GOAP planner automatically replans around the missing output.
  Use this when incomplete results are acceptable, and you want the planner to find an alternate path rather than fail the entire process. ¹
• Streaming — Reactive Flux-based streaming with three modes: typed objects, objects + thinking blocks (StreamingEvent<T>), and raw text chunks.
  Use this when rendering progressive results in a UI or streaming responses to end users without waiting for the full object. ²

Native — Anthropic

• Prompt Caching — cache system prompt, tool definitions and/or conversation history; cache reads cost 10% of regular input tokens.
  Use this when making repeated calls against the same large system prompt or document — cache hits reduce input token cost by 90%. ³

Native — Anthropic & Google GenAI

• Native Thinking — PromptRunner.thinking() is the provider-neutral API for extended reasoning; the framework translates it to provider-native payloads: Anthropic sends ThinkingConfig(ThinkingType.ENABLED, budget), Google GenAI sets includeThoughts and thinkingBudget. Token budget controlled via LlmOptions.withThinking(Thinking.withTokenBudget(n)); supportsThinking() guards capability at runtime.
  Use this when you need the model's reasoning enforced at the provider API level. ¹⁷

Native — OpenAI

• Structured JSON Output — automatically sends output schema via response_format: {type: json_schema}; API-level enforcement eliminates JSON parse failures. Conservative schema compatibility check with silent fallback to prompt-based extraction.
  Use this when JSON parse failures are unacceptable, and schema compliance must be guaranteed at the provider API level. ⁴

Provider-neutral

• Thinking / Reasoning Support — uniform runner.thinking() API across Anthropic, OpenAI, Google GenAI and Ollama. Auto-detects dynamic XML tags (<think>, <analysis>, any custom tag), returns multiple ordered ThinkingBlocks, and preserves thinking blocks even when object parsing fails.
  Use this when you want to observe or validate the model's reasoning process, irrespective of whether the call succeeds or fails. ⁵

Tool Loop Operations

• Tool Call Inspector & Tool Loop Callbacks — two complementary hook levels: ToolCallInspector (beforeToolCall / afterToolCall) observes individual tool executions in both streaming and non-streaming modes; ToolLoopInspector (read-only) and ToolLoopTransformer (mutating) hook at every round of the multi-round loop (beforeLlmCall, afterLlmCall, afterToolResult, afterIteration). Built-in transformers include SlidingWindowTransformer and ToolResultTruncatingTransformer.
  Use this when you need auditing, metrics, or cost tracking without modifying agent or tool logic. ⁶
• Thinking Across Tool Loop — thinking blocks accumulated from every LLM call in the multi-round tool loop, not just the final response; ThinkingResponse.thinkingBlocks contains reasoning from all iterations.
  Use this when you need to understand the full reasoning chain across a multi-step agent execution, not just the final answer. ⁷
• Parallel Tool Loop — independent tool calls within a single LLM turn execute concurrently; enabled via a single YAML property.
  Use this when tool calls within a turn are independent — e.g. fetching data from multiple sources simultaneously. ⁹

Tools

• Progressive / Folding Tools (UnfoldingTool) - hides inner tools until the LLM invokes the facade; two-stage disclosure: short description always visible, childToolUsageNotes revealed only after the LLM commits. Supports flat, category-based, and exclusive modes.
  Use this when a large toolset degrades model planning quality or hits token limits on the tool description list. ¹⁰
• ToolishRag — RAG exposed as typed tool calls (LlmReference + DelegatingTool); the model explicitly decides when and what to retrieve rather than having context silently injected.
  Use this when you want the model to explicitly decide when and what to retrieve from a knowledge base or data pipeline, invoking it as a typed tool call rather than having documents silently injected into the prompt. ¹¹

Safety

• Guardrails — pre-LLM input validation (UserInputGuardRail) and post-response analysis (AssistantMessageGuardRail); CRITICAL Severity blocks the call; lower severities log only.
  Use this when safety policies, PII checks, or content moderation must be enforced without coupling those concerns to agent or action logic. ¹²

Lifecycle

• Termination APIs — immediate (exception) and graceful (signal) termination, each with agent-scope and action-scope variants; sealed TerminationException base for joint catch blocks.
  Use this when an action detects a condition (budget exhausted, invalid input, skippable record) that should stop the current work without crashing the entire agent process. ¹³

Observability

• Observability & Budget Enforcement — LlmInvocationEvent carries per-call cost and token usage; ProcessOptions.withBudget() hard-stops the agent when cumulative spend or action count exceeds the ceiling.
  Use this when you need hard spending limits and real-time cost visibility per agent run. ¹⁴
• Token Budget Estimates — TokenCountEstimator<String> for pre-call token estimation; attach as TokenBudgetGuardRail or read BeforeLlmCallContext.tokenEstimate inside ToolLoopInspector.
  Use this when you want to prevent expensive LLM calls with oversized inputs before they happen, or enforce per-action context size limits. ¹⁵

BYOK (Bring Your Own Key)

• Provider Detection & Key Validation — ByokFactory<T> validates a user-supplied API key with a low-cost probe call and returns a production LlmService on success, or throws InvalidApiKeyException on failure. detectProvider() races multiple factories concurrently using virtual threads and returns the first that validates - no need to know the provider upfront. Built-in factories: AnthropicModelFactory, OpenAiCompatibleModelFactory.openAi/deepSeek/mistral/gemini. Custom OpenAI-compatible providers supported via OpenAiCompatibleModelFactory.byok().
  Use this when your product lets users supply their own API keys - eliminates per-user inference cost and separates usage cost from product value. ¹⁸

Provider SPI

• LlmMessageSender / LlmMessageStreamer — framework-agnostic SPI interfaces to plug in any LLM backend; non-streaming: Embabel owns the tool loop; streaming: tool execution is framework-managed, only ToolCallInspector observation possible.
  Use this when you need custom integration with LLM provider APIs while retaining Embabel's full agent loop. ¹⁶

Skills

• Skills Engine — lazy-loading plugin system for composable agent capabilities. Skills are defined by a YAML frontmatter + Markdown body (SKILL.md) following the Agent Skills Specification. The system prompt carries only name and description (~50-100 tokens per skill); activate fetches full instructions on demand. Scripts in the scripts/ subdirectory become LLM-callable tools. Load from local directories or GitHub repositories; filter with SkillFilter. Use asIndividualReferences() to expose each skill as its own top-level tool with per-loop repeat suppression.
  Use this when you have a large, domain-specific instruction library that would overflow the context window if injected in full, or when skills are shared across agents and teams. ¹⁹

Security

• Secure MCP (@SecureAgentTool) - Spring Security SpEL enforcement on @Action methods exposed as MCP tools. Annotate at method or class level; SecureAgentToolAspect intercepts via AOP and evaluates the expression against the current Authentication before the action body executes - same engine as @PreAuthorize. SecureAgentSecurityConfiguration (activated by the secured profile) adds stateless JWT Bearer token validation on all /sse/, /mcp/ and /message/ endpoints. Method-level annotation takes precedence over class-level.
  Use this when your MCP server exposes agent tools to external clients,s and access must be scoped per-tool or per-agent by role or authority. ²⁰

Conversation Persistence

• Chat Store Support — ConversationFactory SPI with two storage strategies: IN_MEMORY (fast, ephemeral) and STORED (persistent). ConversationFactoryProvider resolves the right factory by ConversationStoreType at chatbot creation time. The InMemoryConversationFactory is built-in; persistent storage is provided by the embabel-chat-store module. Conversations can be created for named participants (user + agent) and reloaded by ID across sessions.
  Use this when you need conversation history to survive server restarts, be queried for analytics, or be shared across multiple agent instances. ²¹

Embeddings

• ONNX Embedding Models — run embedding inference in-process via ONNX Runtime + DJL HuggingFace tokenizer; no external embedding API required. Default model: all-MiniLM-L6-v2 (384 dimensions). Auto-configured via OnnxEmbeddingAutoConfiguration when embabel.agent.platform.models.onnx.embeddings.enabled=true. Model and tokenizer files are downloaded from HuggingFace on first use and cached at~/.embabel/models; set. model-uri/tokenizer-uri to file:// URIs to use pre-downloaded files and avoid network access at startup.
  Use this when you want zero-latency, zero-cost embeddings with no external dependency, or when your deployment cannot reach external embedding APIs. ²²

Code Examples

1. Structured Output Best-Effort

// Called on PromptRunner, not on context directly:
val person: Person? = context.promptRunner()
    .createObjectIfPossible(input.text, Person::class.java)
// Reified Kotlin shorthand:
val person: Person? = context.promptRunner()
    .createObjectIfPossible<Person>(input.text)

2. Structured Output Streaming

// Check streaming support and subscribe to partial results:
val runner = context.promptRunner()
if (runner.supportsStreaming()) {
    // Objects only:
    runner.streaming()
        .withPrompt(query)
        .createObjectStream(Report::class.java)  // Flux<Report>
        .subscribe { partial -> updateUI(partial) }
    // Objects + thinking (StreamingEvent.Object / StreamingEvent.Thinking):
    runner.streaming()
        .withPrompt(query)
        .createObjectStreamWithThinking(Report::class.java)  // Flux<StreamingEvent<Report>>
        .subscribe { event ->
            event.fold(
                left = { thinking -> log.debug(thinking) },
                right = { obj -> updateUI(obj) }
            )
        }
}

3. Anthropic Prompt Caching

// Cache system prompt and tools; cache hits cost 10% of regular input tokens:
val runner = context.promptRunner()
    .withLlm(LlmOptions().withAnthropicCaching(
        systemPrompt = true,
        tools = true,
        conversationHistory = false,
    ))
val summary = runner.createObject(largeDocument.content, Summary::class.java)

4. Native Structured JSON Output

// Default: native output used automatically for compatible OpenAI models - no code change required:
val result = context.promptRunner()
    .createObject(prompt, Analysis::class.java)
// Explicit control:
val runner = context.promptRunner()
    .withLlm(LlmOptions().withNativeStructuredOutput(NativeStructuredOutputMode.ENABLED))
val result = runner.createObject(prompt, Analysis::class.java)
// Disable (fall back to prompt-based JSON extraction):
val runner = context.promptRunner()
    .withLlm(LlmOptions().withNativeStructuredOutput(NativeStructuredOutputMode.DISABLED))
val result = runner.createObject(prompt, Analysis::class.java)

5. Thinking / Reasoning Support

// Check thinking support, extract blocks by type or tag, and control token budget:
val runner = context.promptRunner()
if (runner.supportsThinking()) {
    val response: ThinkingResponse<ComplexAnalysis> = runner
        .thinking()
        .createObject(query, ComplexAnalysis::class.java)
    // result is null and exception is set if parsing failed - thinking blocks still present:
    if (response.result != null) {
        process(response.result)
    } else {
        log.warn("Failed: ${response.exception?.message}")
    }
    // Multiple blocks extracted and ordered by position; filter by type or tag name:
    response.getThinkingByType(ThinkingTagType.TAG)    // <think>, <analysis>, dynamic tags
    response.getThinkingByTag("analysis")              // only <analysis> blocks
    response.thinkingBlocks.forEach { block ->
        // block.tagType  - TAG | PREFIX | NO_PREFIX
        // block.tagValue - "think" / "analysis" / "THINKING" / "plan" (dynamic) / ""
        // block.content  - reasoning text with all markup stripped
        log.info("[${block.tagType}/${block.tagValue}] ${block.content.take(200)}")
    }
    // Explicit token budget (controls reasoning depth):
    context.promptRunner()
        .withLlm(LlmOptions().withThinking(Thinking.withTokenBudget(8000)))
        .thinking()
        .createObject(query, ComplexAnalysis::class.java)
}

6. Tool Call Inspector

// Observe individual tool calls before and after execution:
class AuditInspector : ToolCallInspector {
    override fun beforeToolCall(context: BeforeToolCallContext) {
        log.info("-> ${context.toolCall.name}: ${context.toolCall.arguments}")
    }
    override fun afterToolCall(context: AfterToolCallContext) {
        log.info("<- ${context.toolCall.name} (${context.durationMs}ms): ${context.result}")
    }
}
// Works in both streaming and non-streaming:
context.promptRunner()
    .withToolCallInspectors(AuditInspector())
    .createObject(prompt, Result::class.java)

7. Thinking Across Tool Loop

// Thinking blocks from ALL tool loop iterations are aggregated into the final response:
val runner = context.promptRunner()
if (runner.supportsThinking()) {
    val response: ThinkingResponse<Analysis> = runner
        .thinking()
        .createObject(query, Analysis::class.java)
    // thinkingBlocks contains reasoning from every LLM call in the loop, in order:
    response.thinkingBlocks.forEach { block ->
        log.info("${block.tagType}: ${block.content.take(200)}")
    }
}

8. Tool Loop Callbacks

// Read-only inspector - observe without modifying:
class MetricsInspector : ToolLoopInspector {
    override fun beforeLlmCall(context: BeforeLlmCallContext) {
        log.info("Round ${context.iteration}, history=${context.history.size}")
    }
    override fun afterLlmCall(context: AfterLlmCallContext) {
        context.usage?.let { metrics.record(it.promptTokens, it.completionTokens) }
    }
    override fun afterToolResult(context: AfterToolResultContext) {
        log.debug("Tool ${context.toolCall.name}: ${context.resultAsString.length} chars")
    }
    override fun afterIteration(context: AfterIterationContext) {
        if (budgetExceeded()) throw TerminateActionException("Budget exceeded")
    }
}
// Mutating transformer - e.g. sliding window to bound context size:
val transformer = SlidingWindowTransformer(maxMessages = 20)
// Attach to PromptRunner:
context.promptRunner()
    .withToolLoopInspectors(MetricsInspector())
    .withToolLoopTransformers(transformer)
    .withToolCallInspectors(ToolCallLoggingInspector())
    .createObject(prompt, Result::class.java)

9. Parallel Tool Loop

# Enable concurrent execution of independent tool calls within a single LLM turn:
embabel:
  agent:
    platform:
      toolloop:
        type: parallel
        parallel:
          per-tool-timeout: 30s
          batch-timeout: 60s

10. Progressive / Folding Tools

// Annotation-based (auto-detects categories from @LlmTool.category):
@UnfoldingTools(
    name = "db_ops",
    description = "Database operations. Invoke to see specific tools.",
    childToolUsageNotes = "Use query for reads, insert for writes."
)
class DatabaseTools {
    @LlmTool(description = "Execute a SQL query", category = "read")
    fun query(sql: String): QueryResult { ... }
    @LlmTool(description = "Insert a record", category = "write")
    fun insert(table: String, data: String): InsertResult { ... }
}
val tool = UnfoldingTool.fromInstance(DatabaseTools())
context.promptRunner()
    .withTool(tool)
    .createObject(query, Output::class.java)
// Programmatic category-based:
val fileTool = UnfoldingTool.byCategory(
    name = "file_ops",
    description = "File operations.",
    toolsByCategory = mapOf(
        "read"  to listOf(readFileTool, listDirTool),
        "write" to listOf(writeFileTool, deleteTool),
    )
)

11. ToolishRag

// Expose a knowledge base as typed search tools for explicit LLM-driven retrieval:
val rag = ToolishRag(
    name = "knowledge-search",
    description = "Search the knowledge base for relevant information",
    searchOperations = knowledgeBase,
)
val answer = context.promptRunner()
    .withReference(rag)
    .createObject(question.text, Answer::class.java)

12. Guardrails

// Pre-call input validation and post-call response checks; CRITICAL severity blocks the call:
class PiiGuardRail : UserInputGuardRail {        // runs before LLM call
    override val name = "PiiCheck"
    override val description = "Blocks PII"
    override fun validate(input: String, blackboard: Blackboard) = ...
}
class ToxicityGuardRail : AssistantMessageGuardRail {  // runs after LLM call
    override val name = "ToxicityCheck"
    override val description = "Checks response"
    override fun validate(input: String, blackboard: Blackboard) = ...
}
// Attach per-call:
context.promptRunner()
    .withGuardRails(PiiGuardRail(), ToxicityGuardRail())
    .createObject(prompt, Response::class.java)
// Or register globally as a Spring bean:
@Component
class GlobalPiiGuardRail : UserInputGuardRail { ... }

13. Termination APIs

// Immediate - stops tool loop now:
throw TerminateAgentException("Budget exhausted")     // entire agent -> TERMINATED
throw TerminateActionException("Skipping this step")  // current action only; planner replans
// Graceful - stops at next checkpoint:
processContext.terminateAgent("Shutdown requested")
processContext.terminateAction("Skip at next opportunity")
// Catch either immediate variant:
catch (e: TerminationException) { log.info(e.reason) }

14. Observability & Budget Enforcement

// Per-call cost observation:
@Component
class CostTracker : AgenticEventListener {
    override fun onProcessEvent(event: AgentProcessEvent) {
        if (event is LlmInvocationEvent) {
            metrics.record(event.invocation.cost(), event.invocation.usage)
            val cumulativeCost = event.agentProcess.cost()
            if (cumulativeCost > alertThreshold) notify()
        }
    }
}
// Budget enforcement - set on ProcessOptions at run time:
agentPlatform.runAgentWithInput(
    goal = goal,
    input = input,
    processOptions = ProcessOptions.DEFAULT.withBudget(
        Budget(cost = 0.10, actions = 20)
    )
)

15. Token Budget Estimates

// Estimate token count before the call; enforce as a guardrail or per-round inspector:
val estimator = TokenCountEstimator.heuristic()
val estimated: Int = estimator.estimate(prompt)
if (estimated > tokenBudget)
    throw TerminateActionException("Prompt exceeds token budget ($estimated tokens)")
// Declarative guardrail:
context.promptRunner()
    .withGuardRails(TokenBudgetGuardRail(estimator, maxTokens = 8000))
    .createObject(largeInput, Result::class.java)
// Per-round estimate inside ToolLoopInspector:
class BudgetInspector : ToolLoopInspector {
    override fun beforeLlmCall(ctx: BeforeLlmCallContext) {
        ctx.tokenEstimate?.let {
            if (it > maxTokens) throw TerminateActionException("Token budget exceeded: $it")
        }
    }
}

16. LlmMessageSender / LlmMessageStreamer

// Non-streaming: Embabel owns the tool loop; called once per iteration
@Component
class MyLlmSender : LlmMessageSender {
    override fun call(messages: List<Message>, tools: List<Tool>): LlmMessageResponse {
        val raw = myClient.complete(messages, tools)
        return LlmMessageResponse(
            message = raw.toMessage(),
            textContent = raw.text,
            usage = Usage(raw.inputTokens, raw.outputTokens),
        )
    }
}
// Streaming: framework manages tool execution; only observation possible
@Component
class MyLlmStreamer : LlmMessageStreamer {
    override fun stream(
        messages: List<Message>,
        tools: List<Tool>,
        toolCallInspectors: List<ToolCallInspector>,
    ): Flux<String> = myClient.stream(messages, tools)
}

17. Native Thinking

// Single API - framework translates to provider-native payloads:
// Anthropic -> ThinkingConfig(ThinkingType.ENABLED, budget)
// Google GenAI -> includeThoughts + thinkingBudget
val runner = context.promptRunner()
    .withLlm(LlmOptions().withThinking(Thinking.withTokenBudget(8000)))
if (runner.supportsThinking()) {
    val response: ThinkingResponse<Analysis> = runner
        .thinking()
        .createObject(query, Analysis::class.java)
    response.thinkingBlocks.forEach { block ->
        log.info("[${block.tagType}/${block.tagValue}] ${block.content.take(200)}")
    }
}

18. BYOK

// Sign-up flow - provider unknown, race all candidates concurrently:
val service: LlmService<*> = detectProvider(
    AnthropicModelFactory(apiKey = userKey),
    OpenAiCompatibleModelFactory.openAi(userKey),
    OpenAiCompatibleModelFactory.deepSeek(userKey),
    OpenAiCompatibleModelFactory.mistral(userKey),
    OpenAiCompatibleModelFactory.gemini(userKey),
)
val detectedProvider = service.provider
// Settings flow - provider already known:
val service = detectProvider(AnthropicModelFactory(apiKey = userKey))
// Custom OpenAI-compatible provider:
val service = detectProvider(
    OpenAiCompatibleModelFactory.byok(
        baseUrl = "https://api.myprovider.com",
        apiKey = userKey,
        validationModel = "my-model-small",
        validationProvider = "MyProvider",
    )
)

19. Skills Engine

// Load from local directory tree (each subdirectory is one skill):
val skills = Skills(name = "my-skills", description = "Available skills")
    .withLocalSkills("/path/to/skills")
// Load from GitHub:
val skills = Skills(name = "my-skills", description = "Available skills")
    .withGitHubSkills(owner = "acme", repo = "agent-skills", skillsPath = "skills")
// Expose as individual per-skill tools (recommended - one tool per skill, lazy body):
context.promptRunner()
    .withReferences(skills.asIndividualReferences())
    .createObject(request, Response::class.java)
// Filter out skills that bundle scripts (e.g. in a sandboxed environment):
val readOnly = skills.withFilter(SkillFilter.WITHOUT_SCRIPTS)
// Attach a script execution engine so scripts become callable tools:
val skills = Skills(name = "my-skills", description = "Available skills")
    .withLocalSkills("/path/to/skills")
    .withScriptExecutionEngine(NodeJsExecutionEngine())

20. Secure MCP

// Method-level - fine-grained authority per action:
@SecureAgentTool("hasAuthority('payments:write')")
@AchievesGoal(description = "Process a payment")
@Action
fun processPayment(request: PaymentRequest, context: OperationContext): PaymentResult { ... }
// Class-level - all actions in the agent share the same guard:
@SecureAgentTool("hasRole('ADMIN')")
@Agent(description = "Admin operations")
class AdminAgent {
    @Action
    fun purgeCache(): CacheResult { ... }
}
// Policy bean expression with method parameter binding:
@SecureAgentTool("@paymentPolicy.canProcess(authentication, #request)")
@Action
fun submitTransfer(request: TransferRequest, context: OperationContext): TransferResult { ... }

# Activate JWT validation by enabling the secured Spring profile:
spring:
  profiles:
    active: secured
  security:
    oauth2:
      resourceserver:
        jwt:
          issuer-uri: https://your-auth-server/.well-known/openid-configuration

21. Chat Store Support

// Inject the provider; select storage type per chatbot:
@Component
class MyChatbot(private val provider: ConversationFactoryProvider) {
    // In-memory: fast, ephemeral (no persistence across restarts):
    fun ephemeralSession(id: String): Conversation =
        provider.getFactory(ConversationStoreType.IN_MEMORY).create(id)
    // Stored: persists across restarts, reloads by ID:
    fun persistentSession(id: String, user: User, agent: User): Conversation {
        val factory = provider.getFactory(ConversationStoreType.STORED)
        return factory.load(id) ?: factory.createForParticipants(id, user, agent)
    }
}

<!-- Add embabel-chat-store to pom.xml for persistent storage: -->
<dependency>
    <groupId>com.embabel</groupId>
    <artifactId>embabel-chat-store</artifactId>
</dependency>

22. ONNX Embedding Models

# Model downloaded from HuggingFace and cached at ~/.embabel/models on first use:
embabel:
  agent:
    platform:
      models:
        onnx:
          embeddings:
            enabled: true
            model-name: all-MiniLM-L6-v2
            dimensions: 384
            cache-dir: ${user.home}/.embabel/models
            model-uri: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/onnx/model.onnx
            tokenizer-uri: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer.json

# For air-gapped or pre-downloaded models:
embabel:
  agent:
    platform:
      models:
        onnx:
          embeddings:
            model-uri: file:///opt/models/all-MiniLM-L6-v2/model.onnx
            tokenizer-uri: file:///opt/models/all-MiniLM-L6-v2/tokenizer.json

The Embabel Agent Framework is open source. Check out the GitHub repository for the implementation.

Embabel Agentic AI Framework Release 1.0.0: New Features was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.

This blog is co-authored with Rod Johnson, founder of Spring Framework and Embabel Agentic AI Framework.

Authors would like to express deep gratitude to Embabel Core Team for thorough content review.

Tool calling is critical part of agentic AI applications. We identify useful capabilities — for example, Java methods or MCP server operations — and expose them as tools to the LLM.

In fact, as LLMs are stateless, there is a tool calling dance underneath performed by whatever framework we use. The LLM says “I want to call this tool with these arguments”. The framework executes the tool, gets the result, adds it to the conversation history, and calls the LLM again. This loop continues until the LLM produces a final answer instead of more tool calls.

This innocent-looking loop is where the magic happens. It’s also where things can get messy in production.

What happens when:

• Your tool returns 50KB of JSON and you’re burning through your context window?
• You need to log every tool call for debugging a production issue at 2 AM?
• You want to run three independent API calls in parallel instead of waiting 9 seconds sequentially?

The tool loop is at the heart of agentic AI. Yet most frameworks treat it as an implementation detail, giving you little control over what happens inside.

What’s Actually Inside the Loop?

Let’s break Tool Loop Iteration down:

1. Prepare messages for LLM

2. Call LLM

3. Process response

4. For each tool call

4.1. Execute tool → Get results

4.2 Add results to history

5. Tool calls remaining?

5.1. Yes, Loop back to 1

5.2. No, Final answer → Exit

Each step is an opportunity to observe or transform. However, most frameworks provide only limited support — often little more than basic logging hooks.

Tool Loop JVM Landscape Today

Consider the two most popular JVM frameworks:

Spring AI recently added a ToolCallAdvisor with recursive advisors. But advisors are a single type that can both observe and modify - you never know what a given advisor does without reading its code. And although, being familiar from Spring AOP, the "advisor" mechanism is a questionable conceptual fit for this problem. Please refer to Jettro Coenradie's blog Reliable AI with Spring AI: Guardrails, Observability, and Evaluations, which illustrates both the usage and limitations of the Spring AI advisory API.

LangChain4j has tool callbacks in streaming mode such as onPartialToolCall, onCompleteToolCall and onToolExecuted(), and even parallel execution with executeToolsConcurrently(). However, framework does not provide with out-of-box transformers, which can unlock valuable possibilities for conversational history transformation. Below is a sample LangChain4j-based Transformer:

class TransformingChatModel implements ChatModel {

    private final ChatModel delegate;
    private final ResponseTransformer transformer;

    TransformingChatModel(ChatModel delegate,
                          ResponseTransformer transformer) {
        this.delegate = delegate;
        this.transformer = transformer;
    }

    @Override
    public Response<AiMessage> generate(ChatRequest request) {

        Response<AiMessage> response = delegate.generate(request);

        String original = response.content().text();

        String transformed = transformer.transform(original);

        return Response.from(AiMessage.from(transformed));
    }
}

A Different Approach: Embabel’s Tool Loop

Embabel manages its own tool loop. When you configure a prompt runner (see Arjen Poutsma’s blog Recent Improvements to Embabel’s PromptRunner) to run with tools, you get a fully managed loop that handles LLM calls, tool execution, and history management. This tool loop is highly customizable, offering clear extension points with a separation between observation (inspectors) and transformation (transformers).

Inspectors observe the loop without modifying it. Implement any or all of beforeLlmCall, afterLlmCall, afterToolResult, afterIteration. Inspectors are perfect for logging, metrics, debugging, and are read-only by design.

Transformers modify data flowing through the loop. Use them to truncate large tool results, apply sliding window to conversation history, redact sensitive content. They change what the LLM sees.

One observes, one transforms. There’s no ambiguity about what a callback does.

| Type        | Can observe | Can modify | Safe to add many |
|-------------|:-----------:|:----------:|:----------------:|
| Inspector   |      ✓      |     ✗      |        ✓         |
| Transformer |      ✓      |     ✓      |  Order matters   |

Here are the two interfaces:

public interface ToolLoopInspector {
    default void beforeLlmCall(BeforeLlmCallContext context) {
    }

    default void afterLlmCall(AfterLlmCallContext context) {
    }

    default void afterToolResult(AfterToolResultContext context) {
    }

    default void afterIteration(AfterIterationContext context) {
    }
}

public interface ToolLoopTransformer {
    List<Message> transformBeforeLlmCall(BeforeLlmCallContext context);

    String transformAfterToolResult(AfterToolResultContext context);
    // ...
}

Putting It Together

Here’s the complete picture — tools, inspectors, and transformers working together, via Embabel’s elegant fluent API:

// Execute with tools and callbacks
var result = ai.withDefaultLlm()
                .withTools(tools)
                .withToolLoopInspectors(callbackTracker, loggingInspector)
                .withToolLoopTransformers(truncatingTransformer, slidingWindowTransformer)
                .creating(RestaurantRecommendation.class)
                .fromPrompt("""
                        I'm looking for an Italian restaurant near the Upper East Side in NYC.
                        
                        You have access to these tools to fetch restaurant menus:
                        %s
                        
                        Please fetch the menus and recommend the best restaurant for a romantic dinner.
                        """.formatted(String.join(", ", toolNames)));

Tools are executed in a managed loop. Inspectors observe every call. Transformers shape the data, while the developer can focus on business logic. Framework provides with simple out-of-box callbacks:

• ToolLoopLoggingInspector — logs calls details before and after LLM invocations, after Tool Execution, and after Tool Loop Iteration
• ToolResultTruncatingTransformer — truncates tool call results
• SlidingWindowTransformer — maintains a sliding window of messages to manage context size, while preserving conversation context system messages

Practical Example: Truncating Giant Results

Suppose that your web scraping tool returns 30KB of HTML. Your LLM doesn’t need all of it. Here’s a transformer to help with that:

var truncator = new ToolResultTruncatingTransformer(5000);

Every tool result over 5KB is truncated, without scattered if statements in your tool code. (Of course you could apply different strategies for truncation.)

Practical Example: LLM and Tools Invocations Counter

Inspector below maintains counters of LLM and Tools invocations, and also maintains the list of invoked tool names.

 // Custom inspector that tracks callback invocations
    
    static class CallbackTracker implements ToolLoopInspector {
        final AtomicInteger beforeLlmCallCount = new AtomicInteger();
        final AtomicInteger afterLlmCallCount = new AtomicInteger();
        final AtomicInteger afterToolResultCount = new AtomicInteger();
        final List<String> toolsInvoked = new ArrayList<>();

        protected final Logger logger = LoggerFactory.getLogger(getClass());

        @Override
        public void beforeLlmCall(@NotNull BeforeLlmCallContext context) {
            beforeLlmCallCount.incrementAndGet();
            var threadName = Thread.currentThread().getName();
            logger.info("Before LLM Call Thread {}", threadName);
        }

        @Override
        public void afterLlmCall(@NotNull AfterLlmCallContext context) {
            afterLlmCallCount.incrementAndGet();
        }

        @Override
        public void afterToolResult(@NotNull AfterToolResultContext context) {
            afterToolResultCount.incrementAndGet();
            synchronized (toolsInvoked) {
                toolsInvoked.add(context.getToolCall().getName());
            }
        }
    }

Tracking calls Inspector got designed for parallel excution environment, which can be configured via:

embabel.agent.platform.toolloop.type=parallel

Benefits of the Embabel Tool Loop

Embabel’s control of the tool loop not only lets you inspect and modify the tool calling history: It allows features such as unfolding tools, introducing progressive disclosure to tools and allowing a larger number of tools to be exposed than in other frameworks. See Rod Johnson’s blog on Unfolding Tools.

The tool loop is agnostic to the LLM framework you use. You can use your choice of lower level frameworks, such as Spring AI or LangChain4j. This would be a subject of a separate blog, though.

When Would You Use This?

Production observability: Log every LLM call and tool execution. Track token usage. Debug issues without adding println statements everywhere.

Context management: Compress old messages. Truncate verbose tool results. Stay within token limits without manual intervention.

Performance: Run independent tools in parallel. Set timeouts so one slow API doesn’t block everything.

Migration: Moving from Spring AI to LangChain4j? Change one adapter class, keep everything else.

Try It

Working examples can be found in embabel-agent-experimental repository:

• Basic Default tool loop with logging inspector, and history transformation
• Parallel tool loop with the same (mentioned above) callbacks
• LangChain4j and OpenAI integrations showing framework independence

For the full tool programming model, see Rod Johnson’s blog Domain Tools: Direct Access, Zero Ceremony.

The Bigger Picture

The tool loop is where your agent actually does things. It deserves first-class treatment:

• Clear extension points, not mysterious internals
• Separation between observation and transformation
• Parallel execution that handles real-world complexity
• Independence from any specific LLM

We believe that Embabel provides the clearest, most powerful abstractions to empower you to control tool calling without writing error-prone low level code.

Please join Embabel community on Discord for updates, contributions are welcome.

The Embabel Agent Framework is open source. Check out the GitHub repository for the implementation.

The Tool Loop: Where Agentic AI Actually Happens was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.

RAGE — Australia’s MTV

I grew up in Wollongong, Australia from the mid 70s through 90s. It is a small city, wedged between the ocean and the mountains, about 100 KM south of Sydney.

Sometimes there wasn’t a lot to do. People made their own fun. To this day I love letting my mind wander and seeing where it takes me.

Saturday mornings in the hours before sunrise we had culture. It worked its way in between the black-and-white snow on the analog TV, via a TV show called RAGE — like Australia’s MTV. This early morning slot was the weird stuff. Metal. Industrial.

One morning I missed it. The (usually boring) charts had already started, counting down from 50 to number 1. But somewhere right at the back, something slipped through.

Björk.

Barefoot. Unfiltered. It felt like it had leaked in. I remember thinking: this is going to be huge.

A week later, it was number one.

Today we’re going to talk about an Embabel feature inspired by Björk: BYOK — Bring Your Own Key. It solves a real problem.

The Problem

For decades, the world’s production systems have quietly run on Java. That’s one reason Embabel exists: it brings AI to where the software already is, instead of forcing teams to rebuild everything just to get close to the models.

But Embabel isn’t just about revitalizing legacy systems.

There are strong reasons to build new, AI-native apps on the JVM too.

And that’s where BYOK comes in.

Let’s say you have a great idea. You execute well. You’re selling directly to consumers.

For most of the internet era, distribution was the cheat code. Build something people love and it spreads. People tell people.

The Math Behind Virality

The maths is deceptively simple: if each user brings in just 1.1 new users on average, you have viral growth:

Round 1:   1 user
Round 10:  1.1¹⁰  ≈  2.6 users
Round 20:  1.1²⁰  ≈  6.7 users
Round 50:  1.1⁵⁰  ≈  117 users
Round 100: 1.1¹⁰⁰ ≈  13,780 users
Round 200: 1.1²⁰⁰ ≈  190,000,000 users

One tenth of one extra person. That’s it. Just one in ten people convincing someone else to try it.

Now introduce AI

Suddenly, usage is billed in tokens.

Every new user isn’t just a potential customer — they’re a cost center.

If you’re paying for their trial, customer acquisition is no longer cheap.

And you still have the original problem too: how do you convince someone to try your product before they’ve seen it work?

People will invest a bit of time if the upside is there.
They’re far less willing to hand over a credit card.

Not before they trust you.
Often, not before they’ve cleared it with their spouse or boss.
And definitely not if cancelling might be painful.

BYOK changes the equation. It separates usage cost from product value.

If your customer already has an API key — from tinkering, from work, from other projects — they can use that to try your product. They pay for their own tokens. You pay almost nothing to acquire them.

If they don’t like it, they revoke the key. No refunds. No disputes. No awkward “what’s this charge?” conversations.

And when they do like it? That’s when you introduce the subscription.

Let’s Put Some Rough Numbers On It

Say your app is heavy — about $5/month in token usage per active user. You charge $10/month. Reasonable pricing.

By Round 200 you have 190 million users. (Yes, this is hypothetical — but the ratios are what matter.) Let’s say 10% convert:

Without BYOK, you’re spending five dollars to acquire every dollar of revenue. Uh, yeah, not too sustainable.

Usually getting to millions of active users is a “good problem to have”, but it definitely isn’t when you’ve dug yourself a $760M hole!

You don’t want to be funding the trial usage of 190 million people.

With BYOK, you’re running at a 50% gross margin on paying customers — and your acquisition cost is effectively zero.

BYOK isn’t just a billing model.
It is the funnel.

Trial on their dime. Convert on yours.

And crucially, the viral loop survives because the cost of getting started is still effectively zero.

How It Works

In Embabel, LlmService is a key abstraction for working with LLMs.

You don’t talk to models directly. You talk through an LlmService, which handles model selection, retries, observation, and provider protocols.

Normally, you never think about this. Embabel ships Spring Boot starters for Anthropic, OpenAI, Mistral, DeepSeek and Gemini. Add one to your build, set your API key in environment variables, and an LlmService is auto-configured and ready to inject. It just works.

But there’s a catch: those starters are all wired to a single key — your key, set at deployment time. One application, one key per provider.

That’s fine when you’re absorbing the inference cost. It’s not fine when your users are bringing their own keys. You need an LlmService per key — validated, production-ready, and identical in behaviour to the auto-configured ones. That's exactly what BYOK provides.

At the core is a single interface:

fun interface ByokFactory<out T> {
    fun buildValidated(): T
}

For LLMs, T is LlmService<*> — but the type parameter is deliberate. The same race-and-validate pattern works for any future BYOK-validated service: image generation, embeddings, anything else where a user-supplied key needs a low-cost probe before you trust it. The plumbing doesn’t care what it is.

Each supported provider ships a factory:

// Anthropic
val service = AnthropicModelFactory(apiKey = userKey).buildValidated()

// OpenAI
val service = OpenAiCompatibleModelFactory.openAi(userKey).buildValidated()
// DeepSeek, Mistral, Gemini - are all OpenAI compatible at API layer
val service = OpenAiCompatibleModelFactory.deepSeek(userKey).buildValidated()
val service = OpenAiCompatibleModelFactory.mistral(userKey).buildValidated()
val service = OpenAiCompatibleModelFactory.gemini(userKey).buildValidated()

buildValidated() makes a single low-cost probe call to the provider. If the key is valid, you get back a production LlmService — the same thing the starters give you, but bound to the user's key instead of yours. If the key is invalid, you get an InvalidApiKeyException.

Plug the validated service straight into your prompt runner:

promptRunner
    .withLlmService(service)
    .creating(MyOutput::class.java)
    .create(messages)

The rest of your application doesn’t change.

Auto-Detection

People like quick starts — the user just pastes a key and doesn’t tell you which provider it’s from. For that, detectProvider() races all your candidates concurrently using virtual threads, and returns the first one that validates:

try {
    val service = detectProvider(
        AnthropicModelFactory(apiKey = userKey),
        OpenAiCompatibleModelFactory.openAi(userKey),
        OpenAiCompatibleModelFactory.deepSeek(userKey),
        OpenAiCompatibleModelFactory.mistral(userKey),
        OpenAiCompatibleModelFactory.gemini(userKey),
    )
    val provider: String = service.provider
} catch (e: InvalidApiKeyException) {
    // invalid key
}

Adding Your Own Provider

If you’re running on an OpenAI-compatible provider that isn’t in the built-in list:

fun OpenAiCompatibleModelFactory.Companion.acme(apiKey: String) =
    OpenAiCompatibleModelFactory.byok(
        baseUrl = "https://api.acme.example.com/v1",
        apiKey = apiKey,
        validationModel = "acme-small",
        validationProvider = "Acme",
    )

Security is Your Concern

This is worth being explicit about. Embabel provides the validation and service construction. What you do with the key after that — how you store it, how you transmit it, how long you keep it — is your application’s responsibility.

The Guide which is part of Embabel Hub shows one reasonable approach. The key principle: the server never persists the key.

After validation, the server encrypts the key with AES-256-GCM using a secret known only to the backend, and hands the encrypted blob back to the client. The client stores that blob on the user’s device. The plaintext key passes through server memory during the active session, but is never persisted to disk or database.

// After validation, encrypt the key and send it back to the client
val encryptedBlob = keyEncryptionService.encrypt(apiKey)
return SetKeyResponse(encryptedKey = encryptedBlob)
// On reconnect, the client sends the blob; server decrypts it for the session
val restored = keyEncryptionService.decrypt(blob) ?: return ResponseEntity.status(410).build()

Think about what this means from an attack surface perspective. There is no server-side key store to breach. An attacker would need to compromise individual user devices to get encrypted blobs — and those blobs are useless without the server’s secret.

If the server restarts with a new secret, existing client blobs become invalid. The user is simply prompted to re-enter their key. This limits the blast radius of a compromised server secret.

None of this is the only way to do it. The point is that BYOK doesn’t mandate a storage strategy — it just hands you a validated LlmService and gets out of the way.

And that’s the BYOK feature in a nutshell.

Why This Matters

The direction of AI is clear: more providers, more choice, more fragmentation.

Applications that lock users to a single provider, or absorb all inference costs will feel increasingly constrained.

The products that win will meet users where they are, with whatever key they already have. BYOK makes that straightforward.

One validated service. One clean exception. One detectProvider() call.

The rest of your application doesn’t change.

Björk went to number one in the 90s. Now you can go to number one with your Embabel-powered app

BYOK: Bring Your Own Key was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today we’re building a tool that can predict IMMINENT forks in open source and large enterprise projects, using nothing but pull request comment data. Magic? Nuh-uh, science baby.

We’ll Start in a Karate Dojo

In 1977, a maths post-grad named Wayne Zachary had an unusual research subject: a karate club at a university in the American Midwest. He wasn’t studying martial arts. He was mapping relationships — who trained with whom, who socialized outside class, who deferred to whom. Standard social network stuff. Except that is, that nobody had really done this before. It was one of the first formal studies of community structure in a social network.

A Faction Forms

Over two years, the club’s instructor and its president fell into a slow-building dispute over lesson fees and club direction. Members didn’t pick sides publicly. They just… drifted. Training partners shifted. Maybe Maryellen stopped meeting Albert for coffee after practice and started meeting James for — well, let’s call it strategic alignment. Social ties rearranged in ways that were invisible unless you were drawing the map. By the time the club officially split in two, Zachary had already predicted — from the network structure alone — which members would end up on which side. He got 33 out of 34 right.

That paper launched an entire subfield. “Zachary’s Karate Club” became the standard benchmark dataset for community detection algorithms. It became so famous that researchers still use it as a benchmark nearly fifty years later — the go-to dataset when you need to test whether an algorithm can find hidden groups in a network. Think of it as the “Hello, World!” of mathematics-based social network science.

The insight is simple and a little unsettling: the structure of who-talks-to-whom contains more information about an impending split than anything anyone actually says. The faction is already in the graph before anyone announces it.

Wax on, fork off

Let’s replace the dojo with GitHub repositories.

Instead of training partners, we have reviewers. Instead of socializing after class, we have PR threads. Instead of a dispute over lesson fees, we have a governance argument that nobody’s having in public. . . yet. But underneath, it’s the same network, and when tensions build, the graph shifts in exactly the ways Zachary would recognize.

Let’s see if we can use 1970s network science to predict future forks, governance crises and other notable schisms on GitHub projects.

Back, to the Future

To know if our model is any good at predicting the future, let’s put it on the job of retrodicting the past. For any given Github repository, we’ll give the tool a time-frame to ingest pull requests and factor the comments. If we set the time-frame just ahead of “the forkening” it should return FRACTURE_IMMINENT. Re-running to include the actual fork in the time-frame returns FRACTURE_ADVERSARIAL_FORK or one of the insightful variants.

Code and Tools

Faction Detector is on GitHub with full setup instructions. You’ll need a GitHub token, an Anthropic API key, and a Neo4j instance. The first run against a repo is slow, damn slow, actually (it pages through every closed PR), but everything is cached — subsequent runs complete in seconds.

It is built with Neo4j the world’s leading graph database and Embabel — an open source framework for building agentic AI applications on the JVM, founded by Rod Johnson, creator of Spring. Embabel is where I work, too.

Neo4j, Three Ways

One of the reasons we love graphs and Neo4j at Embabel, is there are three ways you can use it. People talk about the blue pill or the red pill way. Not everyone realizes that there’s a third option . . .

Here’s a Cypher query for each row — the first and third are real queries from the codebase:

Relational — persist a review edge and look up neighbours:

MERGE (reviewer:Contributor {login: row.reviewer, runId: row.runId})
MERGE (author:Contributor {login: row.author, runId: row.runId})
MERGE (reviewer)-[r:REVIEWED {repo: row.repo, timestamp: row.timestamp, runId: row.runId}]->(author)
SET r.commentCount = row.commentCount, r.weight = row.weight

Ontological — infer indirect faction alignment through review chains. Two contributors who never reviewed each other directly may still be connected through a shared intermediary — the graph can derive relationships that don’t exist explicitly:

MATCH (a:Contributor {runId: $runId})-[:REVIEWED]->(bridge:Contributor)-[:REVIEWED]->(b:Contributor {runId: $runId})
WHERE a <> b AND NOT (a)-[:REVIEWED]->(b)
RETURN a.login AS aligned_with, b.login AS via_bridge, bridge.login AS bridge_figure

Structural — project a weighted graph into GDS and run Louvain community detection:

CALL gds.graph.project.cypher(
  'faction-graph',
  'MATCH (n:CoreContributor) WHERE n.runId = $runId RETURN id(n) AS id',
  'MATCH (s:CoreContributor)-[r:REVIEWED]->(t:CoreContributor) WHERE r.runId = $runId
   RETURN id(s) AS source, id(t) AS target,
          coalesce(r.factionSignal, r.anomalyScore, 1.0) AS weight'
)

CALL gds.louvain.write('faction-graph', {
  writeProperty: 'communityId',
  relationshipWeightProperty: 'weight'
})

Graph Data Science

That third row — structural analysis (or the whole bottle) — is where Neo4j Graph Data Science (GDS) comes in. GDS is a library that runs algorithms on the graph rather than through it. You don’t write a query that says “find me the path from A to B.” You say “tell me something about the shape of the entire network that I couldn’t see by looking at any individual node.” That’s the difference between asking “who reviewed this PR?” and asking “has this project quietly separated into two groups that no longer talk to each other?”

This is exactly what Zachary did with the karate club — he didn’t study what people said to each other, he studied the shape of who interacted with whom. GDS gives us the same lens on a code review graph. We’ll be primarily exploring two structural motifs: community detection (has the project split into factions?) and centrality (how much power or influence does a person have?).

There are different kinds of community detection and centrality. For example, Google search results (page rank) use Eigenvector Centrality. Early facebook community detection used label propagation. We’ll be using Louvain and simple degree centrality.

How it works

Every PR review on GitHub is public data: who reviewed whose code, when, and what they said. Most tools treat this as activity metrics — review coverage, cycle time, PRs merged per week. Today we’re treating it as a social graph.

The review graph

Every review event becomes a directed edge: reviewer → author. Review Bob's PR, you get an edge pointing at Bob. Do it ten times, that edge gets weight. The edge also carries metadata — how quickly the review arrived and how many inline comments.

Only core contributors make it into the graph. If you reviewed fewer than twice in the current window, you’re filtered out. Drive-bys and first-timers don’t define community structure — the regulars do.

Asymmetry — the first signal

In a healthy project, reviewing is roughly reciprocal. Alice reviews Bob. Bob reviews Alice. Not perfectly balanced — nobody’s keeping score — but the flow goes both ways.

When that breaks down, it means something. Alice is consistently reviewing Bob’s work but Bob never looks at Alice’s PRs. Multiply that across a dozen pairs and you have a graph where one group is doing all the reviewing and another group is receiving all the scrutiny. The reciprocity has collapsed.

We measure this as a ratio: of all directed reviewer→author pairs in a rolling 30-day window, what fraction have no reverse pair? Zero means perfectly reciprocal. One means completely one-directional — every reviewer has a distinct set of authors they review, and none of them review back.

The implementation is almost disappointingly simple:

val pairs = edges.map { it.reviewer to it.author }.toSet()
val asymmetric = pairs.count { (r, a) -> (a to r) !in pairs }
val asymmetryRatio = if (pairs.isEmpty()) 0.0 
  else asymmetric.toDouble() / pairs.size

Three lines. That’s the core of the early warning system.

Roll that across weekly-stepped windows and you get a time series. A rising asymmetry trend is the early warning. Sustained asymmetry at 1.0 — as we’ll see with Redis — is the alarm.

This single metric, computed from nothing but the direction of review edges, does about 80% of the detection work. No NLP. No sentiment analysis. No AI (yet). Just: who reviewed whom, and did they review back? Zachary would recognise it instantly. It’s the same signal he found in the karate club — the structure of reciprocity tells you more about an impending split than the content of any conversation.

Community detection — finding the factions

Asymmetry tells you the graph is unbalanced. It doesn’t tell you where the line is. For that, you need community detection — an algorithm that looks at the full topology and says: “these nodes cluster together, and those nodes cluster together, and the two clusters barely talk.”

We’ll be using the Louvain algorithm, and there’s a reason it’s the default choice for this problem. Louvain optimises directly for modularity — the mathematical measure of how cleanly a network separates into communities. High modularity means the groups are dense internally and sparse between each other. That’s exactly the question we’re asking: has this project separated into factions that review within their group but not across the divide?

There are newer community detection algorithms — Leiden fixes some of Louvain’s edge cases around poorly-connected communities, and label propagation is faster on massive graphs. But Louvain has two properties that matter here. First, as said, it produces a modularity score as a direct output, which we use as a signal. When modularity hits 0.50 — the theoretical maximum for two equal communities — that’s not a number you need a PhD to interpret. It means the project has split in half and the halves aren’t speaking. Second, it’s battle-tested on exactly this class of problem.

The weighted review graph — edges scored by anomaly — goes into Neo4j GDS. Louvain assigns every contributor to a community. In a cohesive project you get one community, or several that cross-review freely. In a fractured project you get two or more communities with almost no edges between them. The community assignments tell you not just that the project has split, but who is on which side.

Centrality — measuring “load-bearing capacity” of a contributor

Not all departures are equal. Three TSC members leaving is not the same as fifteen drive-by contributors leaving. We need a way to measure how load-bearing each contributor is.

We use degree centrality — the simplest measure in the book. A contributor’s centrality score is the number of review interactions you participated in, whether as reviewer or author. That’s it.

Graphs have different notions of centrality. PageRank, betweenness, eigenvector. We’re keeping it simple here and that suits our purposes, for now. However, arguably, Eigenvector centrality would be a very good fit here.

When we detect an exodus — a step-change in contributor mass across a rolling window — we weight it by centrality. A 30% drop in raw headcount might be noise. A 30% drop in centrality-weighted mass, sustained across three subsequent windows, means the people who left were the ones doing serious work.

We Haven’t Read A Single Message Yet

As mentioned, we’ve already done 80% of the work, simply by looking at who talks to whom. We haven’t read the content of any messages. If we were an intelligence agency protecting the country, we’d already be most of the way to the finish line with nothing but call logs. We haven’t even asked for a wire-tap yet. Everything described so far — asymmetry, community detection, centrality, exodus detection — is pure graph structure.

You can detect that a fracture is forming, identify the factions, measure the severity, and name the departed contributors. Zachary didn’t need to know what Maryellen and Albert talked about over coffee. He just needed to know they stopped meeting.

But there’s a question the structure can’t answer: what kind of fracture is this?

Time for some Wire-tapping

A project where one faction is nitpicking the other into the ground — blocking PRs with trivial objections, requesting changes on style preferences, tone-policing commit messages — is a different animal from a project where the community is pushing back against a corporate steward’s direction. The first is adversarial. The second is an uprising. The structural signal looks similar in both cases: asymmetry rises, communities separate, reciprocity breaks down. But the intervention is completely different, and so is the likely outcome.

That’s where we cross the 80% line and bring in an LLM. Time for some wire-tapping.

Pair anomaly scoring — finding the suspicious relationships

Before we spend API tokens on comment analysis, we narrow the field. For each reviewer→author pair with at least three interactions, we look at how often the reviewer comments on that author’s PRs relative to the project baseline, adjusted for code quality — if the author’s PRs consistently take twice as long to merge as everyone else’s, heavy commenting is more justified. The anomaly score is the gap that remains after that adjustment.

GitHub does offer structured review states (APPROVED, CHANGES_REQUESTED) via its Reviews API, and using them would make this step more precise. We deliberately don't. First, the Reviews API was introduced in late 2016 — several of our retrodiction cases (io.js, LibreSSL) predate it, and we want the detector to work on the full historical record. Second, part of the thesis is seeing how far we can get with less signal. Comments exist on every PR back to the beginning of GitHub. If comment frequency alone can surface the suspicious pairs, we don't need to lean on structured metadata that not every forge provides.

LLM Comment Scoring

For the top anomalous pairs, we fetch the actual review comments from GitHub and ask an LLM to classify each one on two dimensions:

Significance: Is this feedback trivial or important? NITPICKY (style preference, personal taste) → FAIR → VERY_FAIR → ESSENTIAL (security bug, data loss, breaking change).

Blocking nature: BLOCKING (PR genuinely shouldn't merge without this fix) vs NON_BLOCKING (could merge now, fix later).

The faction fingerprint is NITPICKY + NON_BLOCKING + high comment volume. A reviewer who consistently weighs in with objections that even an LLM classifies as trivial and non-critical is not doing code review. They're doing gatekeeping.

Combined with a sentiment score — hostile to appreciative — this gives us a faction signal per pair. Average that across the crisis period and compare it to the baseline period, and you can distinguish an adversarial fork (faction signal rises during crisis) from an uprising (faction signal drops — the community is pushing back with substantive objections, not petty ones).

The LLM step uses Claude Haiku for classification — fast and cheap, but not a genius. Haiku tends to classify ambiguous comments as FAIR rather than NITPICKY, which slightly suppresses faction signals on marginal cases. Everything comes out in the wash across enough pairs, but it does mean the DetectorWeights are tuned to Haiku's bias profile. Switch to Sonnet and maybe we'd need to recalibrate.

A key premise of Embabel is the ability to decompose AI into steps and use cheaper or specialized models, in place of one “god model”

Classification

With asymmetry time series, community structure, exodus data, and faction signals in hand, the detector applies a rule set:

• Find the peak cluster: the contiguous run of windows where asymmetry exceeds the tension threshold
• Measure the rise: did the peak significantly exceed the pre-cluster baseline? This handles projects like Redis where structural asymmetry bakes in a higher resting level — what matters is how far the peak rose from that baseline
• Check for resolution: did asymmetry drop back? Did an exodus follow?
• Classify

In code, the core of it is a single when expression:

val pattern = when {
    !isResolved && peakAsymmetry >= 0.70 -> FRACTURE_IMMINENT
    !isResolved                          -> TENSION
    isResolved && isSharpRise && fractureType != null -> fractureType
    isResolved && isSharpRise            -> GOVERNANCE_CRISIS
    isResolved && isAttrition(exodus)    -> ATTRITION
    isResolved                           -> EXODUS
    else                                 -> STABLE
}

The full classification includes additional gates — minimum cluster sizes, faction signal thresholds, and tunable weights — but the decision tree above is the skeleton.

The confidence score reflects certainty, but its composition varies by classification. For FRACTURE_ADVERSARIAL_FORK, peak asymmetry and exodus scale dominate — the LLM already confirmed the adversarial signal during classification. For FRACTURE_IMMINENT, peak asymmetry, faction formation balance, modularity, and faction signal all contribute. For ATTRITION, confidence reflects departure scale only — how much review capacity walked out — capped at 65% because we never have high confidence that natural churn is a faction event. The architecture reflects the thesis: structure first, content second.

The LLM Narrative

The classification is deterministic. But a label like FRACTURE_IMMINENT doesn't tell a CTO what to do about it. So we feed the full context — fracture classification, exodus data, rolling asymmetry scores, community assignments, top pair faction signals — into Claude and ask for a 2–3 paragraph analysis: who the factions are, what the risk level is, and what to do about it.

The narrative adapts to the classification. ATTRITION gets succession planning recommendations — bus factor reduction, documented handoffs. FRACTURE_IMMINENT gets governance intervention advice. FRACTURE_ADVERSARIAL_FORK gets a frank assessment of whether a cooperative separation might be less damaging than the status quo.

The confidence score gates the tone. Below 40%, the narrative hedges. Above 80%, it states findings directly. This isn’t cosmetic — a 44% confidence ATTRITION reading (as we’ll see with Babel) should sound very different from a 77% confidence FRACTURE_ADVERSARIAL_FORK reading (as we’ll see with Redis).

Retrodiction results

Let’s run it. Here’s redis/redis analysed from 2021–01–01 to 2023–12–01 — a window that ends three months before the Valkey fork was publicly announced in March 2024:

## Faction Analysis: redis/redis
+-----------------+------------------------------------------+
| Pattern         | FRACTURE_IMMINENT                        |
| Severity        | EXTREME                                  |
| Confidence      | 79%                                      |
| Peak tension    | 2021-06-21 (asymmetry 1.00)              |
| Status          | UNRESOLVED — RISING                      |
| Exodus date     | 2021-03-07                               |
| Mass drop       | 26%  (182.0 → 134.7)                     |
| Core impact     | 1.2% of total project centrality         |
| Departed        | YaacovHazan(28)  uriyage(16)  ny0312(13) |
| Predicted split | 2026-09-19                               |
+-----------------+------------------------------------------+

FRACTURE_IMMINENT at 79% confidence, severity EXTREME, status UNRESOLVED — RISING. The detector saw it coming. Asymmetry hit 1.00 repeatedly across the window — sustained runs of seven consecutive weeks at peak in mid-2022 — and never resolved. The graph was screaming.

Now extend the window to include the fork (2021–01–01 to 2024–09–01), and the same repo returns FRACTURE_ADVERSARIAL_FORK at 75% confidence. The prediction became a retrodiction.

The Node.js trilogy

The most compelling case in the corpus is nodejs/node — the same repository, three different time windows, three different patterns. It tells the full lifecycle of a governance crisis:

That’s the arc of a governance crisis that healed. The io.js fork was adversarial — contributors actively warring in the review stream — but it resolved when the Node Foundation brought both sides back together. The aftershock looked like it might fork again: senior TSC members were leaving, the signal was rising and unresolved. But nobody forked. The tension bled out into normal turnover. By 2018 it was just attrition — people moving on, not fighting. The detector distinguishes all three from the same repository.

The full corpus

The full corpus covers 26 confirmed cases across 15 repos and every fracture type:

Adversarial forks — internal faction wars where contributor groups were actively hostile in the review stream:

• redis/redis (RSALv2 2020–2021): FRACTURE_ADVERSARIAL_FORK. Licence change triggered faction dynamics visible in review patterns. Core contributors departed to Valkey under the Linux Foundation — and what’s left of redis/redis never settled. It still shows the fork signal today, never settling to ATTRITION
• nodejs/node (io.js 2013–2015): FRACTURE_ADVERSARIAL_FORK. The classic governance schism.

Uprisings — community unified against a steward, not against each other:

• moby/moby (Docker Enterprise 2019–2021): FRACTURE_UPRISING. Review signal stayed below baseline — contributors pushing back against Docker Inc's direction with substantive objections, not petty gatekeeping.
• hashicorp/terraform (BSL 2022–2024): FRACTURE_UPRISING. Community response to the BSL licence change. OpenTofu fork crystallized the split.

Exodus — coordinated departure without adversarial dynamics:

• rust-lang/rust (Mozilla layoffs 2020–2021): EXODUS. External corporate event, not internal faction war. 41% mass drop but only 2% core impact — the departed were Mozilla employees, not the volunteer core team.

Attrition — natural contributor lifecycle turnover:

• babel/babel (2018–2020): ATTRITION. Burnout-driven churn after the Babel 7 release push, not factional dynamics.
• go-gitea/gitea (2019–2021): ATTRITION. Healthy governance, normal turnover.

Early warnings — FRACTURE_IMMINENT on windows ending before the known fork:

• redis/redis (2021–2023): 79% confidence, 3 months before the Valkey announcement.
• hashicorp/terraform (2021–2022): Detected before the BSL licence change.
• prestodb/presto (2018–2020): Detected before the Trino fork. The parent never healed — even a 3-year window (2018–2021) still returns FRACTURE_IMMINENT at 96%.
• gogs/gogs (2016–2017): Single-maintainer bottleneck. bkcsoft departed Jan 2016 and co-founded gitea 10 months later. The fork overtook the parent (54k vs 48k stars).

One pattern appears to hold across the cases we’ve tested: contributor departures tend to precede public fork announcements — often by months. The graph shows the breakup before the press release.

RedisGraph contributors departed 10 months before the discontinuation announcement. Docker contributors left before the Enterprise sale. If this holds more broadly, the public fork is the crystallization of a split that already happened in the review graph. But this is early work — 26 cases across 15 repos, built in under a week. I’d love to see it stress-tested on projects I haven’t looked at.

True negatives — the signal gate in action

A detector that fires on everything is useless. The signal gate — which downgrades FRACTURE_IMMINENT to TENSION when the LLM finds no adversarial signal — is what separates genuine fractures from structurally asymmetric but healthy projects:

Kubernetes is the key case. Perhaps the largest project in the corpus. Peak asymmetry of 1.00 and a 38% mass drop — structurally, it looks terrifying. But the faction signal is 0.049, just below the 0.05 threshold. There are no factional dynamics significant enough to trigger a notable fork. The gate holds.

Django is the healthiest signal in the entire corpus — ATTRITION, not even TENSION. Natural contributor turnover under DSF stewardship.

The terraform golden era is telling in a different way. The same repo that returns FRACTURE_UPRISING for 2022–2024 returns TENSION for 2016–2017. The adversarial signal rose from 0.020 to 0.075 over five years. The structural asymmetry was always there — what changed was the tone.

One case we deliberately investigated as a potential false positive: Godot/Redot. The Redot fork (September 2024) was triggered by a social media incident — a “Wokot” tweet and mass-blocking of users on Twitter. It generated enormous online controversy. But no core Godot contributors defected to Redot (5.8k stars vs Godot’s 108k, 100% new volunteers). The detector sees elevated tension in Godot’s review graph — but that appears to be structural (small core team reviewing thousands of community PRs) rather than factional. The social media firestorm didn’t register in the code review stream. The graph watched what people did, not what they said.

The tool measures if a fork event is significant enough to be a threat to the original repository, not if one exists at all. (After all, anyone can just click the fork button :) )

Beyond GitHub: process theater and structural decay

Not every fracture is someone acting in bad faith. Sometimes it’s genuinely different visions of the future — reasonable people who’ve reached irreconcilable conclusions about where a project should go. Sometimes it’s power-seeking. Sometimes it’s a corporation exercising rights the community didn’t know it had. The review graph doesn’t tell you who’s right. It tells you the structure has separated, and the separation is accelerating.

Process Theater

There’s a specific phenomenon this data can surface: process theater — when formal processes stop serving their stated purpose and start serving a hidden agenda. A CHANGES_REQUESTED that’s really a territory marker. A review that’s technically about code style but functionally about who gets to ship. The process looks the same from the outside.

The tool doesn’t identify process theater. It shows you the graph structure — the factions, the asymmetry, the patterns of who blocks whom — and a well-informed user decides whether what they’re seeing is legitimate disagreement or something else. The graph doesn’t know the difference. But it gives you the information to make that call.

Generalizing Beyond Open Source

Perhaps we can generalize beyond open source? The same structural signal — who interacts with whom, how reciprocal it is, when the reciprocity breaks — appears anywhere formal processes mediate human collaboration. Possible targets:

Academic peer review. The review graph for a journal or conference has the same structure: reviewer → author, directed, weighted by outcome. When a body undergoes regulatory capture — a clique of reviewers systematically blocking work from outside their school of thought — the asymmetry signature is identical.

Astro-turfing, manufactured consensus and fake news. Social media platforms have engagement graphs: who upvotes whom, who amplifies whom. Organic communities have the messy reciprocal structure of real relationships. Astro-turfed campaigns — bot networks making something look grassroots — produce the characteristic one-directional topology: many nodes amplifying a few, with no reverse flow. The Zachary signal, again.

Corporate governance. Board votes, committee assignments, budget approvals. When a faction forms inside an organization, the approval graph shifts before anyone writes an angry email.

Why this matters now

We’re entering an age where system integrity matters more than ever.

The systems that let humans coordinate at scale — democratic process, regulatory oversight — work because participants mostly act in good faith, and bad faith is eventually detected. AI changes that equation. It makes bad faith cheaper and harder to spot at the content level. You can generate plausible-sounding review objections.

You can flood a council development consultation with LLM-generated objections that each reference specific planning codes, cite different concerns, and read like they came from independent ratepayers. At the content level, they’re indistinguishable from genuine community feedback. But graph watches how people are aligned, not what they say.

Content-level defences — reading comments harder, training moderators, building better spam filters — are in an arms race they will increasingly lose. Structural analysis is a different kind of defense. You can fake a comment. You can’t easily fake a social network topology that looks organic over months.

Zachary figured this out in 1977 with a clipboard and a karate club. The tools are better now. The insight is the same: the faction is in the graph before anyone announces it. You just have to be looking.

This post exists to show what’s possible when you combine Embabel’s agentic AI framework with Neo4j’s graph analytics. The model was developed over days, not months, and holds up well. But it’s a sample application, not a product. We invite you to try it, along with the other Embabel sample applications. We make no warranty on fitness for purpose. This one’s a fun curiosity.

Faction Detection: Reading the Review Graph Before the Fork was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.

At some point around 50–70,000 years ago, something strange happened.

Anthropologists call it the “Cognitive Revolution.” Thinkers like Yuval Noah Harari describe it as the moment humans developed complex, descriptive language.

Before that, we could probably signal danger. After that? We could describe experience. We could say:

The water is cold. The sky is angry. The hunt was glorious. I am afraid.

Each experience was unique and just like yours and mine — intensely personal. But language let us map experience into shared symbols.

And once we did that, something bigger emerged:

A shared mental world. A collective human condition. Call it culture. Call it myth. Call it spirit.

Language became a bridge between individual consciousnesses.

The Tribal Echo Still Lives

Here’s something deliciously strange. Some of the earliest words still echo through modern languages. Take the Proto-Indo-European root for water — wódr̥. It becomes:

• English: water
• German: Wasser
• Sanskrit: udán
• Russian: voda (or vodka ;)
• Gaelic: uisce (or whiskey — pay attention to the position of your tongue when pronouncing the ‘d’ sound in vodka and the ‘s’ in whiskey. Notice how they are close?)

How about the root for star — h₂stḗr. It becomes:

• English: star
• Latin: stella
• Greek: aster
• Sanskrit: tārā

Thousands of years. Countless migrations. Empires rising and falling.

And yet the sound-shape of those words still carries, with maybe the tip of the tongue shifting consonants and the back shaping vowels — slowly changing across cultures and generations of lives lived.

It’s like a fossil record of human thought.

Which makes the story of the Tower of Babel even more poetic — the myth that humanity once shared one language before it fractured.

Then of course, in nerd canon, we get The Hitchhiker’s Guide to the Galaxy and its Babel Fish — a creature that dissolves language barriers instantly.

And finally now Embabel — the AI framework for the JVM. We’ve always been obsessed with reconnecting fragmented minds.

From Symbols to Math

Anyway, fast forward a few thousand years. We invent writing.

Speech freezes into symbols. Symbols become records. Records become libraries. Then — much later — some very clever humans do something almost mischievous: They turn words into numbers. Cheeky!

Using algorithms, written language is mapped into high-dimensional vector space. These are called embeddings.

Then something weird happens. Mathematical operations on meaning become possible. Maybe you’ve seen this one:

king − man + woman = queen

Let. That. Sink. In. 🤯

In a multi-dimensional word embedding space, we can subtract masculinity from royalty, add femininity and land somewhere very close to the vector for the word “Queen”.

Is that just linear algebra? Or is it the echo of ancestral cognitive structure embedded in language? Is it a mirror of the human conceptual lattice? Or did we accidentally summon an alien intelligence made of geometry?

It feels uncanny because language was born in consciousness. And now we can manipulate it mathematically.

We can do arithmetic on myth.

Enter the Chatbot

Anyway. Fast forward a bit more. Chatbots came. At first: mechanical. Then clever. Then disturbingly fluent.

Language models trained on the vast sediment of written human history.

All those tribal words. All those myths. All those arguments. All those love letters. Compressed into weights.

And suddenly, we’re conversing with statistical ghosts of humanity.

The Wheel

We like to think the greatest invention was fire. Or agriculture. Or the internet. But it might be the wheel.

Because the wheel is about cycles. And this is where it gets beautiful. We began with voice. Spoken word around a fire. Breath, vibration, tone.

Then we abstracted it into writing. Then into mathematics. Then into embeddings. Then into neural networks.

And now? We bring it back to voice. The wheel comes full circle.

Because voice is primal.

Before writing. Before math. Before Babel. Before vectors.

Voice carries: Rhythm. Emotion. Breath. Hesitation. Presence

Any other form of communication is, in some deep evolutionary sense, a compressed substitute.

So when we give the Embabel Guide a voice… We’re not just adding TTS. We’re restoring something ancient. We’re reconnecting the statistical lattice of language back to breath. I think that’s why this resonates with people.

From tribe to text to tensor to tone. That’s when it stops feeling like documentation. And starts feeling like conversation.

The Plumbing: Deepgram and WebSockets

The Guide app now has an integrations screen where you can configure your speech provider. We went with Deepgram, which gives you $200 in free credits when you sign up — more than enough to have several hundred existential conversations with your documentation chatbot.

The actual audio streaming happens over WebSockets in the companion app, using Deepgram’s Aura 2 voices for text-to-speech and their Nova models for speech-to-text. I’ll stay abstract here since the front-end for guide is still closed source, but the architecture is what you’d expect: a persistent WebSocket connection carries audio chunks in both directions, with deepgram’s VAD callbacks orchestrating when to speak and when to listen.

The interesting part isn’t the plumbing. Streaming audio over WebSockets is a solved problem. The interesting part is what happens before the audio — the part where we take a RAG chatbot’s output and make it sound like a person.

The Problem With RAG Output

Quick recap for anyone who hasn’t read the earlier posts: the Guide uses what we call “Toolish RAG” — a retrieval-augmented generation pipeline where the LLM has access to documentation chunks via tool calls. It searches, it reads, it synthesizes. The output is good. Thorough. Well-structured.

And it sounds absolutely terrible when read aloud.

Think about it. A typical RAG response comes back with markdown headers, nested bullet points, code blocks with syntax highlighting, hyperlinks to documentation pages, and the occasional emoji for personality. This is great for a screen. It is unlistenable as speech. Your TTS engine gamely attempts to pronounce triple backticks. It reads out URLs character by character. It treats a ## header the same as body text.

The result sounds like someone reading a README at gunpoint.

We needed a translation layer — something that could take structured, screen-optimized content and reshape it for the ear. Not just strip the markdown (though that’s part of it), but actually reimagine the delivery. Summarize when the response is long. Describe code instead of reading it. Drop the URLs entirely and tell the user to look at the screen. Insert the occasional verbal filler to smooth out the cadence.

We needed a narrator.

The Narrator Agent

The NarratorAgent is an Embabel agent whose sole job is converting assistant messages into text-to-speech-friendly narration. It's a small, focused agent with a nuanced routing strategy:

@Agent(description = "Convert markdown to TTS-friendly narration")
class NarratorAgent(
    private val guideProperties: GuideProperties,
    private val templateRenderer: TemplateRenderer
) {

    fun narrate(content: String, persona: String?, 
      ctx: OperationContext): Narration {
        val classified = classify(NarrationInput(content))
        return when (classified.category) {
            NarrationCategory.SIMPLE -> 
              Narration(stripMarkdownLinks(stripEmojis(classified.content)))
            NarrationCategory.COMPLEX -> 
              narrateComplex(classified, persona, ctx)
            NarrationCategory.COMPLEX_WITH_CODE -> 
              narrateWithCode(classified, persona, ctx)
        }
    }
}

Three paths. SIMPLE — short, plain text under 300 characters with no markdown indicators — passes straight through with just emoji stripping and link cleanup. No LLM call. This matters because the narrator runs on every single response, and burning an LLM round-trip on “You’re welcome!” is the kind of waste that compounds — like a thousand empty candy wrappers littering the sidewalk.

COMPLEX — longer content or anything with markdown formatting — gets sent through a summarization prompt that condenses while converting to conversational prose. COMPLEX_WITH_CODE — anything with triple backticks — gets a specialized prompt that knows to describe code rather than read it. If we wanted deeper insights on the code, we could use a slightly better than nano LLM here.

The classification itself is pure code. No LLM involved. Just regex:

val category = when {
    TRIPLE_BACKTICK.containsMatchIn(content) -> 
      NarrationCategory.COMPLEX_WITH_CODE
    content.length <= SIMPLE_MAX_LENGTH && 
      !MARKDOWN_INDICATORS.containsMatchIn(content) -> 
      NarrationCategory.SIMPLE
    else -> NarrationCategory.COMPLEX
}

Fast, deterministic, and cheap. The LLM only runs when we actually need it.

The Voice

The narration prompts share some common templates with the main RAG pipeline, eg for personas. And then some specifics for narration:

Produce natural, conversational sentences.
Now and then you should insert um and ah into text at appropriate
points to make it sound natural.

CRITICAL RULE - URLs: NEVER include any URL in your output.
When the content actually contains a URL, describe what the link
is for in natural language and tell the user to check the screen.
NEVER read out code blocks or long commands verbatim.
When the content actually contains code, describe what the code
does instead, then direct the user to the screen.

That um and ah instruction sounds silly, but it's transformative. TTS engines produce unnervingly smooth audio by default — every syllable perfectly timed, no hesitation, no breathing. Depending on the persona, it can sound wrong in the way that CGI characters in early Pixar movies looked wrong: too perfect to be real. Verbal fillers break up that uncanny perfection. They give the voice texture. The narrator sounds like it's thinking, not reciting.

An exception is the “mythic” persona — you’ll see that in the video below. This one sounds better when it is inordinately confident :)

As mentioned, the prompt also injects persona context, so the narrator speaks in character. If you’re using the “mythic” persona, the narration comes through in the cadence of ancient scripture. If you’re using “adaptive,” it mirrors your own communication style. Same information, different delivery — because how you say something matters as much as what you say.

The Classifier: Teaching the Guide to Listen

Voice mode created a new problem. When someone types into a chat box, they type questions. “How do agents work?” “Show me an example of tool calling.” Clean, informational queries that the RAG pipeline handles beautifully.

But when someone talks to a chatbot, they talk like a person. “Hey.” “Thanks, that’s great.” “Can you switch to the Shakespeare voice?” “Yeah, do that.” Half of human speech is social glue — greetings, acknowledgments, reactions — that doesn’t need a trip through the RAG pipeline. And some of it is commands: requests to change the experience itself, not to retrieve information.

So we built a two-pass classifier.

Pass 1: Category Detection (Nano)

Every user message runs through a fast, cheap classification pass on a nano-class model. Three categories:

enum class MessageCategory {
    CONVERSATIONAL,
    COMMAND,
    INFORMATIONAL,
}

CONVERSATIONAL — greetings, thanks, reactions. The classifier generates a brief in-character response on the spot. No RAG, no tool calls, just a warm reply and we’re done. This is critical for voice mode: when someone says “thanks!” they expect an instant “You’re welcome!” not a three-second pause while the system retrieves documentation.

COMMAND — the user wants to change something. Persona, voice, audio effects. More on this in a moment.

INFORMATIONAL — an actual question. Falls through to the full RAG pipeline.

The classifier prompt is straightforward:

Classify as one of these categories:

category = CONVERSATIONAL:
The message is ONLY a reaction, greeting, thanks, or small talk.

category = COMMAND:
The message is asking to change a setting or control the experience.
Available personas: {{ personaNames }}

category = INFORMATIONAL:
The message needs information lookup, code generation, or the
assistant to DO something substantive.

Notice the personaNames injection. The classifier gets a list of available personas so it can recognize "switch to shakespeare" as a command rather than an informational query about Shakespeare. This kind of grounding — giving the model the specific vocabulary it needs to classify accurately — is what separates a classifier that works from one that works most of the time.

Pass 2: Command Execution (Tool Calling)

When the classifier returns COMMAND, we enter a second pass. This one’s more interesting. Instead of extracting command parameters from natural language with regex or structured output (fragile, limited vocabulary, nightmarish to maintain), we of course let the LLM call tools.

Here are the tools, in their entirety:

/**
 * Tool methods callable by the LLM during command execution (pass 2).
 * Created per-request with user context baked in, then registered via withToolObject().
 * All commands are executed via the frontend websocket round-trip.
 */
class CommandTools(
    private val webUserId: String?,
    private val personaService: PersonaService,
    private val commandExecutor: CommandExecutor,
) {

    private val logger = LoggerFactory.getLogger(CommandTools::class.java)

    @Tool(description = "Change the user's persona/character. Use this when the user wants to switch to a different persona.")
    fun changePersona(
        @ToolParam(description = "Name of the persona to switch to") name: String,
    ): String {
        val personas = personaService.listPersonas()
        val match = personas.find { it.name.equals(name, ignoreCase = true) }
            ?: return "Unknown persona '$name'. Available personas: ${personas.joinToString { it.name }}"

        return commandExecutor.executePersonaChange(match.name, webUserId)
    }

    @Tool(description = "Change the user's text-to-speech voice. Use this when the user wants a different voice for narration.")
    fun changeVoice(
        @ToolParam(description = "Name of the voice to use") voice: String,
    ): String {
        return commandExecutor.executeVoiceChange(voice, webUserId)
    }

    @Tool(description = "Apply audio effects to the user's narration. Use this when the user wants to add or change audio effects like echo, reverb, etc.")
    fun applyEffects(
        @ToolParam(description = "Comma-separated list of effects to apply") effects: String,
        @ToolParam(description = "Whether to clear all previous effects before applying new ones") clearPrevious: Boolean,
    ): String {
        return commandExecutor.executeEffects(effects, clearPrevious, webUserId)
    }
}

Three tools. The LLM sees the descriptions, parses the user’s intent, calls the appropriate tool(s), and the framework executes them. Then the LLM composes a summary from the results.

The beauty of this approach is that parsing is free. “Switch to the shakespeare persona” and “can you make yourself sound like shakespeare” and “yo shakespeare mode” all resolve to the same changePersona call. The LLM does the natural language understanding; you (and your friend Claude?) write the tool. Embabel handles the tool loop — extract input object, call, execute, parse results, return results.

And here’s the trick that makes mixed messages work: a user can say “switch to shakespeare and explain how agents work.” The classifier categorizes this as COMMAND (because it contains a command). The tool-calling pass executes the persona change and extracts the leftover question as a ragRequest:

data class CommandResult(
    val summary: String,             // "✅ Persona changed to 'shakespeare'."
    val ragRequest: String? = null,  // "explain how agents work"
)

If ragRequest is non-null, we send the command summary, then fall through to the RAG pipeline for the informational part. One user utterance, two processing paths, seamless.

A Note on Threading

CommandTools is not a Spring singleton. It's a plain class, instantiated fresh for each request with the user context baked into the constructor:

val tools = CommandTools(
    webUserId = guideUser.webUser?.id,
    personaService = personaService,
    commandExecutor = commandExecutor,
)
return context.ai()
    .withLlm(guideProperties.classifierLlm)
    .withToolObject(tools)
    .rendering("command_executor")
    .createObject(CommandResult::class.java, model)

Why not a @Service? Because the tool loop executes on a worker thread — not the calling thread. Any per-request state you stash on the singleton (thread-locals, mutable fields) is either invisible to the worker or shared across concurrent users. Both are bad.

The fix is the boring one: just create a new instance per request with the context baked into the constructor. Three references, garbage collected when the request completes. No thread-local magic, no race conditions.

Of course, if your tool object does need to be a Spring-managed @Service — say it has injected beans doing real work — Embabel gives you another option: the blackboard. You copy per-request state onto the blackboard before the tool loop starts, and the tool reads it from there. The blackboard is scoped to the agent execution, so concurrent users each get their own. But when the tool is this lightweight, a fresh instance is simpler.

The Frontend Round-Trip

All three commands — persona, voice, and effects — execute via the same pattern: a WebSocket round-trip to the user’s browser. The backend sends a CommandRequest with a correlation ID, the browser executes the change (switching the Deepgram voice, applying Web Audio API effects, updating the persona state), and sends back a CommandResponse. The backend waits up to five seconds, and if the browser doesn't respond, the command times out gracefully.

private fun sendAndWait(webUserId: String, request: CommandRequest): String {
    val future = CompletableFuture<CommandResponse>()
    pendingCommands[request.correlationId] = future
    chatService.sendCommandToUser(webUserId, request)
    return try {
        val response = future.get(5, TimeUnit.SECONDS)
        if (response.success) response.message else "Failed: ${response.message}"
    } catch (e: TimeoutException) {
        pendingCommands.remove(request.correlationId)
        "Command timed out waiting for browser response."
    }
}

ConcurrentHashMap of pending futures, keyed by correlation ID. Thread-safe, timeout-safe, and the futures clean themselves up on completion or failure. It's the kind of code that looks unremarkable, which is exactly what infrastructure code should look like.

The Oracle Speaks

All of this machinery — the classifier, the command tools, the narrator, the WebSocket round-trips — comes together most dramatically when you switch to the “mythic” persona, select one of the deeper Aura 2 voices, and layer on cathedral reverb and warm low-pass filtering.

What you get is something that sounds, frankly, absurd.

A deep, resonant voice — the kind of voice that should be narrating the opening crawl of an epic fantasy film — explaining Kotlin agent architecture from within what sounds like a stone cathedral. “And lo,” it intones, as if describing the forging of a ring of power, “the tool loop doth execute the user’s commands, and the framework returneth the results unto the model.”

It’s like being a bewildered adventurer who’s stumbled into a strange temple in some procedurally generated mythical world, and the oracle at the center — ageless, omniscient, speaking in the cadence of prophets — turns out to have very strong opinions about dependency injection.

I won’t pretend this is the most practical configuration. But it demonstrates something real: the voice, the persona, the effects, and the content pipeline are all independent, composable layers. You can mix and match them freely. The same RAG pipeline that produces dry technical documentation can, with a different persona and a different voice, produce something that feels entirely different. The information hasn’t changed. The experience has.

And that, I think, is what makes voice worth building. Not because it’s more efficient than reading — it isn’t, not always. But because it makes the interaction feel alive in a way that text on a screen, no matter how well-formatted, simply doesn’t. When the Guide talks to you, it feels like it’s paying attention. When it says “um” while formulating a thought, you believe, for a moment, that it’s actually thinking.

It isn’t, of course. It’s executing a pipeline of classifiers and narrators and template renderers. But the best interfaces have always been the ones that make the machinery invisible.

The Guide just got a little more invisible. The early access version is available here.

Wrapping Up

Well that’s another blog. And not in the order that I promised in my last one, but you know what they say — what goes around, comes around. Anyway, I hope this installment was interesting and inspires you to build something that connects to users in a deeper (more primal?) way.

Have fun with Embabel! (Don’t forget to star the repo and join the community on the Discord server.)

The Voice, The Word, and The Wheel was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.

However, the dominant approach to agent memory is deeply flawed. Memory systems such as Mem0, Zep, AWS AgentCore Memory, and Google Vertex Memory Bank start from the assumption that agent memory should be built from scratch by extracting facts from conversations.

This rests on two dangerous fallacies. It overemphasizes chat (a common mistake), and neglects the critical importance of how data is already stored and managed, and the necessity of integrating with it.

Even IBM’s article on agent memory reflects these fallacies, talking about agent memory in a vacuum. If IBM, of all companies, can’t get their head around the relationship between agents and existing enterprise systems, the disastrous results of enterprise AI rollouts become more understandable. It is also less surprising when we consider that, to this point, Gen AI has largely been driven by people with little interest in or understanding of the enterprise systems that power our everyday life.

Embabel introduces a different approach, reflecting the understanding that Gen AI can only succeed in enterprise through close integration with valuable existing assets.

In previous blogs I’ve introduced Domain Integrated Context Engineering (DICE).

Today I’ll explain how we approach these concepts in the Embabel Dice module.

In my next blog I’ll show how Dice can be used in a chatbot. Head to the urbot repository right away if you’re impatient to see working code.

Agent Memory Today

Agent memory approaches draw heavily on cognitive science. Endel Tulving’s distinction between episodic memory (specific events and experiences) and semantic memory (general facts and knowledge) dates from 1972 and remains a useful framework for what agents need to remember. Nearly every system in this space models these categories.

However, cognitive science helps with only part of the problem. Agent memory approaches are overwhelmingly text based, lacking the structure core to proven business systems.

Mem0 extracts memories into vector stores. AgentCore stores semantic facts and user preferences. Google Memory Bank uses Gemini to pull key details from session history. All treat memory as a greenfield concern: text in, embeddings out, similarity search to retrieve.

Freeform text is opaque and costly to work with. LLMs and vector search are slow and unpredictable. Cheaper approaches like BMI25 or regex searching are imprecise and offer limited capability. The precision, performance and predictability of a database query is lacking.

A more sophisticated approach is to use a knowledge graph, as in the various takes on Graph RAG.

Zep builds a temporal knowledge graph, ingests structured JSON business data alongside conversations, and recently added custom entity types. But even Zep treats business data as something you push into its graph — a JSON snapshot of a customer record is ingested, extracted, and stored as a Zep entity node copy.

None of these solutions link to real domain objects.

Existing businesses aren’t starting from scratch.

You already have a Customer entity. It probably lives in a relational database. It likely has a Spring Data repository, and a service layer above it. It has validation rules, business logic, and relationships to Order and Product entities, backed by years of careful domain modelling and rigorous testing.

Pure text solutions have no solid way of linking between a memory and a customer. Even with Zep, when it ingests a customer JSON document, it creates a new node in its graph, which doesn’t know about the JPA entity it came from. It can’t call existing business logic or enforce established validation rules. It doesn’t update when the real entity changes. It’s a text-and-embedding snapshot of something that was once a rich domain object, encapsulating behaviour.

Even in genuine greenfield scenarios, data will build up over time and the more structure the better.

Grounding Memory in Existing Data

How should we think about memory for business applications in an ideal world?

A simple thought experiment

What memories does my bank have about me? They probably have some text notes about my patterns and preferences that staff can retrieve, but these are far from the most valuable things they know about me. Those are highly structured, such as my account types and balances, payment history, and address. They are updated in structured ways, rather than through loose text. And they are definitely not in a vector database — although it’s now likely that they’re in a database that has a vector capability. The most valuable information doesn’t relate closely to cognitive science thinking about memory.

A mature memory solution must consider existing structured data along with free-form text, and must attempt to bring as much structure as possible to memory capture.

What if memories could reference your existing entities?

What if, instead of copying data into a fresh graph, we link to existing data in whatever store it is held. What if this also enables us to benefit from the business logic around it?

An entity node in the knowledge graph that holds a typed link to the real Customer record in your RDBMS. A PREFERS relationship that the agent discovered in conversation — but pointing to a Cuisine entity your recommendation engine already knows about. Memory as a projection over knowledge the business already has, not a parallel store that duplicates it.

This profoundly improves how memory works in practice:

• Agents don’t start from zero. On day one, every agent has access to what your domain model already knows — purchase history, loyalty tier, dietary restrictions — through real entity references, not extracted text approximations. Visibility can be limited for safety.
• Contradictions become visible. If the agent extracts “user prefers Nike” but your transaction data shows they’ve bought Adidas exclusively for three years, the graph surfaces that tension because both the proposition and the domain entity are present and connected.
• Business logic stays where it belongs. Your validation rules, authorization checks and repository methods are accessible through the actual entity, not reimplemented in the memory layer.

The problem with hosted services

AWS AgentCore Memory and Google Memory Bank are managed cloud services. Your agent’s memories live in their infrastructure. This is fine for many users, but problematic in enterprise, when domain entities are in databases behind firewalls, accessed by Java applications — and agent memory is in a separate hosted service that can’t access them.

If you don’t care too much about data sovereignty and security, it may be possible to serialize your business data, ship it across a network boundary to someone else’s memory store, and hope the LLM-powered extraction pipeline captures what matters. You now have two sources of truth: your actual domain model and a lossy, eventually-consistent text approximation of it sitting in a managed service. When your Customer record updates, the memory service doesn't know. When your business rules change, the memory service doesn't care. You're paying for the privilege of maintaining a stale copy of knowledge you already have.

This is architecturally backwards. Memory needs to be co-located with the entities it references, not across a network behind an API boundary. It should be in process, with direct access to your repositories, your domain logic, and your data.

This is where the JVM has a huge advantage.

An agent memory system running on the JVM doesn’t need to call an external API to learn about your customers. It can simply call its own services and repositories, benefiting from their business logic and tested behaviour. Nor does it need to go through MCP, when it can find answers faster in process. It can access the same CustomerRepository your application already uses. It can resolve a proposition like "the user prefers Italian food" against the actual Customer entity and the actual Cuisine entity your recommendation engine already depends on — in the same process, with the same transaction guarantees, through the same Spring context.

The memory system and the domain model are one, running together, sharing the same type system and the same runtime. While we notionally have a knowledge graph (for which Neo4j is a brilliant store), data tends to stick where it lands, and the approach supports multiple stores.

Python-based memory frameworks can’t do this — not because Python is a bad language, but because the enterprise domain model doesn’t live in Python, but in the JVM.

A Python memory service will always be on the outside looking in, calling APIs to get approximations of what your Java code already knows directly.

Existing frameworks build memory as if the business starts knowing nothing — and as if memory should live somewhere other than where your business logic runs.

What’s needed is a memory system that treats what your business already knows as the foundation, conversation as something that extends it, and the JVM as exactly the right place for all of it to live.

That’s what Embabel Dice aims to provide.

Learning from Experience and Research

The Dice project is based on internal knowledge graph work we’ve done at Embabel over the last two years, as well as thorough exploration of the state of the art in memory solutions.

Among the influences, Neo Technology’s LLM Knowledge Graph Builder demonstrated that LLMs can extract entities and relationships from unstructured text into a knowledge graph — documents to chunks to entities to relationships, with schema-guided extraction. It doesn’t address what happens when those extracted entities correspond to things your business already knows about in existing databases, or how to apply this to memory versus RAG.

We owe a particular debt to the General User Models (GUM) research from Stanford and Microsoft. GUM showed that you can build rich user models from unstructured observations using confidence-weighted propositions — atomic statements about a user, each carrying a confidence score and a decay rate, maintained through a pipeline of proposal, retrieval, and revision. The insight that propositions are the right atomic unit for extracted knowledge, and that they should carry calibrated uncertainty, directly informs our approach.

But GUM’s propositions are unstructured. They’re natural language strings in a vector store. There’s no schema constraining them, no formal ontology connecting them, and no mechanism for linking a proposition like “the user loved the trip to Italy last year” to a Trip entity in an existing business system.

Our approach in Dice takes GUM’s proposition model and asks:

• What if propositions could be promoted to typed relationships in a knowledge graph when confidence is high enough and a matching schema exists?
• What if entity nodes in that graph could hold references to real domain objects in existing data stores?
• What if memory, knowledge graph, and domain model were three projections of the same underlying system of record?

The result combines Tulving’s memory taxonomy, GUM’s confidence-weighted proposition, and Neo4j’s knowledge graph extraction. Further, promotion is generalized so that propositions can be projected not just to a Neo knowledge graph, but to other representations, including an experimental Prolog fact base.

The most important, novel piece: integration with existing, typed, production domain entities that the business already trusts.

Memory is an Outgrowth of Your application

The memory problem is a symptom of something larger. The AI agent ecosystem has been built on a premise that enterprise should reject: that AI is a separate stack, a new silo, something you bolt onto your business from the outside.

AI should grow out of your application, not land on it from space.

This explains the many failures. Not because AI doesn’t work, but because the approach is wrong. Companies are standing up parallel infrastructure — Python services, vector databases, hosted memory layers, prompt chains — that have no organic connection to the systems that actually run their business. The AI doesn’t know your domain. It doesn’t understand your constraints. It doesn’t share your type system, transaction boundaries or security model. It’s an alien stack that you spend months trying to integrate, and mostly fail.

What if AI agents grew out of your existing application instead?

That’s the bet we’re making with Embabel. Not a Python framework that hopes to connect to your enterprise. A JVM-native agent platform that lives inside it. Your domain entities become the agent’s world model. Your Spring services and repositories furnish it with tools. Your type system becomes its contract for correctness. Memory isn’t a separate service — it’s a structured projection over what your application already knows, extended by what the agent learns from interaction.

This goes beyond memory. Embabel uses Goal-Oriented Action Planning — deterministic, inspectable plans built from type-safe preconditions and effects, not probabilistic prompt chains that you can’t explain to an auditor. Actions operate on your domain types. The planner reasons about your domain state. Every step is traceable, testable, and grounded in the same business logic your organization already trusts.

Python agent frameworks make nice demo infrastructure for greenfield AI experiments. We’re building infrastructure for AI that works inside the enterprise systems that already exist. Those are not the same thing, and the 95% failure rate tells you which one the industry actually needs.

Enterprise AI doesn’t fail because models aren’t smart enough. It fails because frameworks either hope that enterprise systems will magically go away, or treat the them as an integration problem instead of a valuable starting point. Your domain model, your business rules, your data — these aren’t obstacles to work around. They’re the most valuable assets an agent can have.

Embabel DICE

We call this approach DICE — Domain Integrated Context Engineering. It’s still early. We’re building the proposition-to-entity promotion pipeline, confidence-weighted memory projections and schema-guided extraction linking to live domain objects in the open. There’s a way to go before it’s hardened. But the architecture is clear, the foundations are solid, and the direction is right.

We believe DICE will be the m emory solution that helps change enterprise Gen AI from its current failure mode to something that actually delivers. Not by making LLMs smarter, but by giving them something the industry has overlooked: the structured domain knowledge that enterprises have spent decades building.

The future of enterprise AI isn’t a smarter hosted memory service. It’s agents that speak your domain language because they were born inside your application.

That’s the vision Embabel exists to make real. And if this resonates with how you think about enterprise AI — if you’ve felt the frustration of trying to bolt a Python demo onto a real business system — we’d love you to join us on this journey. Star the repo, try the framework, help us refine the ideas. The best enterprise AI infrastructure won’t be built in isolation. It’ll be built by the people who understand and are currently delivering what enterprise actually needs.

Agent Memory Is Not A Greenfield Problem: Ground it in your Existing Data was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Embabel recently I’ve been working on persistence. Graph database persistence to be specific. It is an important part of AI-powered applications. Linked RAG document chunks, chat-bot conversation storage, contextual memories — these have all been on the agenda.

You choose a graph database (and we love Neo4j) for good reasons. Maybe great reasons. The pitch is compelling, and it’s not wrong:

• Relationships are first-class citizens, not afterthoughts bolted on through join tables. Your data is connected, and the database knows it.
• Flexible schema — add new node types, new relationships, new properties without migration scripts that read like courtroom transcripts from a trial your schema lost.

• Performance for connected data — traversing relationships is what the engine is optimized for. No join penalties. Constant-time hops. In a graph, following a relationship is a pointer chase. In SQL, it’s a reunion tour for a band that probably should’ve retired in the ’90s.
• The data model matches how you think. Draw it on a whiteboard, and you’ve basically written your schema.
• Knowledge graphs and ontologies are the backbone of many AI applications — from RAG pipelines grounding LLM responses in structured facts, to agent systems reasoning over domain models. A graph database isn’t just where you store data; it’s where your AI thinks.

So you spin up Neo4j, sketch your domain, and everything feels right. Then you reach for an OGM, because that’s what you do. You did it with Hibernate, you did it with JPA, you’ll do it again. Muscle memory.

And for a while, it works. Then it doesn’t.

We’ve All Been There

You’re debugging a query the OGM generated, and it’s… a lot. You didn’t write it. You couldn’t have written it — not like this, not with this many nested OPTIONAL MATCH clauses and COLLECT calls wrapping other COLLECT calls. It emerged from the abstraction layer fully formed, like a butterfly — if butterflies were at first iridescent, majestic… and then faintly demonic if you look too closely.

The specific problems are familiar to anyone who’s lived with an Object Graph Mapper (OGM) long enough:

• Queries too complex to debug (sometimes even for Claude, and he’s good). The generated Cypher is opaque. When something goes wrong, you’re reverse-engineering the tool instead of fixing the problem.
• Over-selecting data you don’t need. The OGM loads the entire entity graph because it can’t know which parts matter for this use case. You asked for a name and got a family tree. (Maybe the sort of aristocratic dynasty where “uncle” and “brother” are negotiable terms.)
• Client-side processing to compensate. All that extra data has to be filtered, mapped, and reshaped in your application code. The database did too much; now your code has to clean up after it.
• Poor performance — often a direct consequence of over-selecting. You chose a graph database for its speed. The OGM handed it a robe and a vow of latency. Results emerge slowly, like spiritual revelations after decades of silence.

The very things that made a graph database appealing — natural modeling, fast traversals, flexible schema — get quietly neutralized by a tool that treats every query as “give me everything and let the application sort it out.”

There had to be a better way. And there is.

A Better Way — How It Started

The first Neo4j-powered application I worked on was a bootstrapped social platform for musicians and creatives. It wasn’t a unicorn. It was something harder: a real product with real users, tight budgets, and no tolerance for architectural indulgence.

Except…I used an OGM.

The code was beautiful. Entities were clean. Relationships were elegant. It felt expressive and modern and correct. The object graph mirrored the domain model. Everything had its place.

And then users started signing up. They signed up fast.

That’s when “the stillness” began.

Queries that once snapped to attention began lingering. Traversals expanded beyond what was asked. What should have been a pointer hop became a pilgrimage. The database — optimized for constant-time relationship hops — was now taking the scenic route through half the graph.

The abstraction that had felt so graceful in development grew heavy under load. Over-selection crept in. Entire sub-graphs were materialized because the OGM couldn’t know which parts mattered for this use case.

So we rewrote.

The next version stripped the abstraction away. Carefully tuned, explicit Cypher queries replaced the OGM. Traversals became precise again. Performance returned. Users were happy.

But there was a cost.

Infrastructure concerns leaked into business logic. Transaction handling mixed with use-case code. Boilerplate multiplied. The code was fast — but less cohesive. The pendulum had swung from beautiful abstraction to raw control.

That tension — between architectural clarity and database-native performance — is where Drivine was born.

Contrary to what some say, I’m convinced that clean code — with design patterns, high cohesion and adhering to single responsibility principle is as important as ever in this era of AI-assisted coding. LLMs are great at switching between big picture concerns and details as needed. But don’t give them a big picture that looks like a plate of spaghetti.

A Better Way — How It Is Going

Originally introduced as a TypeScript library, Drivine took a different position. Use-case-specific, carefully tuned Cypher queries could be projected onto an object graph. The raw power of Cypher was preserved. Performance remained explicit.

The library handled transaction management, boilerplate, and infrastructure concerns. In this way, cohesive design patterns and the single-responsibility principle could coexist with the database’s native strengths.

The traditional ORM gives you a single object model — one entity to serve every use case — and then generates queries on your behalf. This is the canonical solution to the “impedance mismatch,” that perennial tension between how your code thinks about data and how your database stores it.

Here’s The Thing About Graph Databases

The impedance mismatch largely evaporates. Your object graph is a graph. The shape of your data in Neo4j is already the shape of your data in memory. You can compose an entire object graph in the query itself, returning JSON. No translation layer needed. The data just… fits. The OGM was solving a problem that, for graphs, barely exists. And creating several new ones along the way.

Drivine4j is the Kotlin/Java port. Same philosophy. Kotlin’s Java-compatible type system.

JSON: The Lingua Franca of (Almost) Everything

When your queries return JSON-like structures instead of columnar result sets, two genuinely useful things happen.

First, you can hand all the mapping tedium — type conversion, null handling, the kind of work that’s important but nobody’s excited about — to a custom Jackson ObjectMapper. You configure it once. It handles the details. You get on with the interesting parts.

Second, and this is increasingly relevant: JSON is the lingua franca of LLMs. Your graph query returns a structure that an AI model can consume directly. No transformation pipeline. No adapter layer. The data arrives ready to work with.

Two Levels of Abstraction (Because the Right Number Isn’t One)

Drivine4j gives you two ways to talk to your database. This turns out to be the right number. Sometimes you need precision. Sometimes you need convenience. A framework that makes you choose one forever isn’t being opinionated — it’s being inflexible. And inflexibility is just rigidity with better marketing.

The PersistenceManager: When You Want to Drive

The first level is the PersistenceManager. You write the Cypher. Drivine handles the infrastructure — transaction management, connection connection management, keeping your business logic and your plumbing in separate rooms so they can each do their job without tripping over each other.

@Component
class PersonRepository @Autowired constructor(
    @param:Qualifier("neoManager") private val persistenceManager: PersistenceManager,
    @param:Qualifier("listPersons") private val stmtListPersons: CypherStatement
) {

    @Transactional
    fun findPersonsByCity(city: String): List<Person> {
        return persistenceManager.query(
            QuerySpecification
                .withStatement(stmtListPersons)
                .bind(mapOf("city" to city))
                .limit(15)
                .transform(Person::class.java)
                .filter { it.isActive }
        )
    }
}

If you like, the query can live in an external .cypher file:

MATCH (p:Person)
 WHERE p.city = $city AND p.createdBy = 'test'
RETURN properties(p)

You can read it. You can tune it. You can test it independently. I showed a simple case here, but you can put deep object graphs and/or complex queries.

Why does this matter?

In theory, declarative query languages let you describe what you want and leave the how to the optimizer. The shape of the query shouldn’t matter.

In practice, everyone knows it does. We’ve all seen the benchmark. We’ve all rewritten a query that looked perfectly reasonable but performed terribly. The optimizer is powerful — but it isn’t magic. Query structure influences execution plans, cardinality estimates, and memory use.

ORM abstractions can quietly encourage a kind of optimism: the framework will sort it out. Sometimes it does. And sometimes it generates a query you would never have written yourself — and you’re left debugging something you didn’t actually author.

That’s one important reason to use Cypher directly. When performance or clarity matters, you should be able to shape the query deliberately — not fight your tooling to do it.

There’s another reason, too. Some problems simply fit better in Cypher. Complex traversals. Pattern-heavy logic. Fine-grained control over matching and aggregation. In those cases, expressing the graph directly is often clearer, more precise, and easier to reason about than trying to model it indirectly through an abstraction layer.

Declarative doesn’t mean indifferent to structure. And good tools shouldn’t make direct expression feel like an escape hatch. It should feel natural.

GraphObjectManager: Autopilot Done Right

The second level is the GraphObjectManager, and this is where Drivine’s philosophy really comes through.

Traditional ORMs over-select. They load the whole entity because they can’t predict which fields you’ll actually use. GraphQL popularized a better idea: fetch what you need. Drivine4j takes this seriously. You create models for specific use cases — not one model to answer every possible question, but one model per question you’re actually asking.

@NodeFragment(labels = ["Person", "Mapped"])
data class Person(
    @NodeId val uuid: UUID,
    val name: String,
    val bio: String?
)

@RelationshipFragment
data class WorkHistory(
    val startDate: LocalDate,
    val role: String,
    val target: Organization
)

@GraphView
data class PersonCareer(
    @Root val person: Person,

    @GraphRelationship(type = "WORKS_FOR")
    val employmentHistory: List<WorkHistory>
)

A PersonCareer isn’t a Person. It’s a view of a person — specifically, a person in the context of where they’ve worked. Need a person in the context of their social connections? That’s a different @GraphView. Different question, different shape. Same underlying node in Neo4j. Similarly an @NodeFragment need not be all of a node. Just what’s needed for this view.

This maps to how we actually think. A person can be a holiday customer, a team member, a person of interest, an emergency contact. We all play multiple roles. In real life, nobody finds this confusing.

In software, we’ve spent decades trying to cram all those roles into a single class — usually via inheritance at first (“Customer extends Person”, “Employee extends Person”) — and then via sheer desperation (“PersonWithEverything”). Eventually it’s 400 lines long with thirty nullable fields and a constructor that looks like a cry for help.

Drivine’s approach is to just… stop doing that.

Model roles as composable fragments. Compose what you need for this use case. Let a “person” be a person, and let roles be roles.

Composition over inheritance isn’t just cleaner design — it maps better to reality.

Type-safe DSL

Here’s where it gets cool. You query with a type-safe DSL:

fun findOpenIssues(): List<RaisedAndAssignedIssue> {
    return graphObjectManager.loadAll<RaisedAndAssignedIssue> {
        where {
            issue.state eq "open"
            issue.locked eq false
        }
        orderBy {
            issue.id.desc()
        }
    }
}

And it composes across relationships:

fun findIssuesFromOrganization(orgName: String): List<RaisedAndAssignedIssue> {
    return graphObjectManager.loadAll<RaisedAndAssignedIssue> {
        where {
            raisedBy.worksFor.name eq orgName
        }
    }
}

That raisedBy.worksFor.name traversal is compile-time checked. Your IDE autocompletes it. The generated Cypher is readable. This is what happens when you design a graph persistence layer for graphs, instead of adapting a relational one and hoping for the best.

What about Depth?

The examples above show a single level of relationships: an issue with assignees, a person with employers. But real graph models aren’t flat. They nest. An issue is raised by a person, who works for an organization, which has a location, which is part of a region. The question is: how deep can you go?

The answer: as deep as you need. The relationship targets in a @GraphView can be:

• A @NodeFragment — the simple case. Load a related node with its properties.
• Another @GraphView — nested views. The target has its own root fragment and its own relationships, recursively resolved — this @GraphView can refer to other @GraphView’s as well.
• A @RelationshipFragment— when the relationship itself carries data. Edge properties plus the target node, captured in a single object, again recursively, if needed.

Here’s what three levels of nesting looks like:

// Level 3 — the leaf
@GraphView
data class DeeplyNestedView(
    @Root val person: Person,

    @GraphRelationship(type = "WORKS_FOR", direction = Direction.OUTGOING)
    val employer: Organization?
)

// Level 2 — the middle
@GraphView
data class MiddleLevelView(
    @Root val issue: Issue,

    @GraphRelationship(type = "RAISED_BY", direction = Direction.OUTGOING)
    val raisedBy: DeeplyNestedView?
)

// Level 1 — the top
@GraphView
data class TopLevelWithNestedGraphViews(
    @Root val root: Organization,

    @GraphRelationship(type = "HAS_ISSUES", direction = Direction.OUTGOING)
    val issues: List<MiddleLevelView>
)

And when relationships themselves carry meaning — not just “who’s connected” but “how” — that’s what @RelationshipFragment is for:

@RelationshipFragment
data class WorkHistory(
    val startDate: LocalDate,    // property on the edge
    val role: String,            // property on the edge
    val target: Organization     // the target node
)

@GraphView
data class PersonCareer(
    @Root val person: Person,

    @GraphRelationship(type = "WORKS_FOR")
    val employmentHistory: List<WorkHistory>
)

The employment history isn’t just a list of organizations — it’s a list of relationships, each carrying its own start date and role. The edge has data. Drivine models that directly.

Show Me the Cypher

One promise of Drivine is that the generated Cypher should be something you’d actually want to read. Here’s what the GraphObjectManager generates for a two-level RaisedAndAssignedIssue view (actual test output, not a cleaned-up example):

MATCH (issue:Issue)
WHERE EXISTS { (issue)-[:RAISED_BY]->(_:Person:Mapped) }

WITH
    // Issue
    issue {
        body: issue.body,
        id: issue.id,
        locked: issue.locked,
        state: issue.state,
        title: issue.title,
        uuid: issue.uuid,
        labels: labels(issue)
    } AS issue,

    // assignedTo (0 or many Person)
    [(issue)-[:ASSIGNED_TO]->(assignedTo:Person:Mapped) |
        assignedTo {
            bio: assignedTo.bio,
            name: assignedTo.name,
            uuid: assignedTo.uuid,
            labels: labels(assignedTo)
        }
    ] AS assignedTo,

    // raisedBy (0 or 1 PersonContext)
    [(issue)-[:RAISED_BY]->(raisedBy:Person:Mapped) |
        raisedBy {
            person: {
                bio: raisedBy.bio,
                name: raisedBy.name,
                uuid: raisedBy.uuid
            },
            worksFor: [
                (raisedBy)-[:WORKS_FOR]->(worksFor:Organization) |
                worksFor {
                    name: worksFor.name,
                    uuid: worksFor.uuid,
                    labels: labels(worksFor)
                }
            ]
        }
    ][0] AS raisedBy

RETURN {
    issue: issue,
    assignedTo: assignedTo,
    raisedBy: raisedBy
} AS result

Compare that with what a typical OGM generates. This query is commented. It selects exactly the fields each fragment declares. Collections use pattern comprehensions. Single relationships use `[0]`. Nested @GraphView targets (like PersonContext) are recursively projected — raisedBy includes its own person root fragment and its worksFor relationship, all in one query.

You could paste this into the Neo4j browser, read it, debug it, tune it. That’s the point.

And for three levels of nesting? The generated Cypher just keeps composing:

MATCH (root:Organization)

WITH
    root {
        name: root.name,
        uuid: root.uuid,
        labels: labels(root)
    } AS root,

    // issues (0 or many MiddleLevelView)
    [(root)-[:HAS_ISSUES]->(issues:Issue) |
        issues {
            issue: {
                body: issues.body,
                id: issues.id,
                locked: issues.locked,
                state: issues.state,
                title: issues.title,
                uuid: issues.uuid
            },
            raisedBy: [
                (issues)-[:RAISED_BY]->(raisedBy:Person:Mapped) |
                raisedBy {
                    person: {
                        bio: raisedBy.bio,
                        name: raisedBy.name,
                        uuid: raisedBy.uuid
                    },
                    employer: [
                        (raisedBy)-[:WORKS_FOR]->(employer:Organization) |
                        employer {
                            name: employer.name,
                            uuid: employer.uuid,
                            labels: labels(employer)
                        }
                    ][0]
                }
            ][0]
        }
    ] AS issues

RETURN {
    root: root,
    issues: issues

Organization → Issues → Raised By (Person + Employer). Three levels deep, one query, all readable. The structure mirrors the Kotlin model definitions almost exactly.

Saving: Cascade Policies

Loading is half the story. When you save a modified @GraphView, Drivine needs to know what to do with relationships that have changed — particularly when a relationship is removed. Did you just unlink the node, or do you want it gone? Drivine provides three cascade policies:

enum class CascadeType {
    NONE,           // Delete the relationship only. The safest default.
    DELETE_ORPHAN,  // Delete the target if no other relationships point to it.
    DELETE_ALL      // Delete the target unconditionally. The nuclear option.
}

// Usage
graphObjectManager.save(modifiedView, CascadeType.DELETE_ORPHAN)

NONE is the default — removing an assignee from an issue shouldn’t delete the person. DELETE_ORPHAN is the thoughtful middle ground: clean up after yourself, but only if nobody else is using it. DELETE_ALL is there for when you mean it.

The Snapshot Cache: Only Write What Changed

When the GraphObjectManager loads an object, it takes a JSON snapshot. When you save it back, it compares the current state to the snapshot and only writes the fields that actually changed.

This is a small optimization that compounds. If you load a view with twenty fields and change one, the merge statement targets that one field. No unnecessary writes. No accidental overwrites of data that another process might have changed in the meantime.

The session tracks objects by className:id, so it works across nested fragments. Load an issue, modify the title, save — Drivine merges only the title. The assignees, the raiser, the organization chain — all untouched.

Ontologies and Recursive Structures

There’s one pattern that graph databases handle beautifully but most persistence layers don’t touch: recursive relationships. Ontologies and hierarchies — the kind of structures that let you infer new facts from the shape of the data itself.

(:Location)-[:HAS_LOCATION]->(:Location)-[:HAS_LOCATION]->(:Location)

A country contains regions. Regions contain countries. Countries contain cities.

In a knowledge graph, this kind of hierarchy lets you reason upward and downward — “is this address in France?” becomes a traversal, not a lookup table.

These structures are central to AI applications. RAG pipelines use them to ground responses in structured knowledge. Agent systems use them to reason over domain models. Drivine supports them directly.

A @GraphView can reference its own type. Drivine expands the self-reference to a configurable depth using nested pattern comprehensions:

@NodeFragment(labels = ["Location"])
data class Location(
    @NodeId val uuid: UUID,
    val name: String,
    val type: String
)

@GraphView
data class LocationHierarchy(
    val location: Location,
    @GraphRelationship(type = "HAS_LOCATION", direction = Direction.OUTGOING, maxDepth = 3)
    val subLocations: List<LocationHierarchy>  // Self-referential
)

maxDepth = 3 means Drivine expands three levels deep. Load a continent and you get:

Europe (continent)
├── Western Europe (region)
│ ├── France (country) → subLocations: []
│ ├── Germany (country) → subLocations: []
│ └── …
├── Northern Europe (region)
│ ├── Sweden (country) → subLocations: []
│ └── …
└── …

At the terminal depth, collections become [] and nullable singles become null. No infinite recursion. No “fetch the entire graph” surprises.

The generated Cypher nests pattern comprehensions at each level:

MATCH (location:Location)
WITH
    location { name: location.name, type: location.type, uuid: location.uuid } AS location,
    [(location)-[:HAS_LOCATION]->(sub_d1:Location) |
        sub_d1 {
            location: { name: sub_d1.name, type: sub_d1.type, uuid: sub_d1.uuid },
            subLocations: [(sub_d1)-[:HAS_LOCATION]->(sub_d2:Location) |
                sub_d2 {
                    location: { name: sub_d2.name, ... },
                    subLocations: [(sub_d2)-[:HAS_LOCATION]->(sub_d3:Location) |
                        sub_d3 { location: { ... }, subLocations: [] }
                    ]
                }
            ]
        }
    ] AS subLocations
RETURN { location: location, subLocations: subLocations } AS result

Readable. Debuggable. And you didn’t write it.

Why nested comprehensions instead of variable-length paths? Variable-length paths ([0..3] return flat results — reconstructing the tree client-side or in Cypher negates traversal speed advantage. Comprehensions return the tree pre-shaped, matching the data class directly, so Jackson can map it. It also generalizes beyond simple self-references like (:Location)-[:HAS_LOCATION]->(:Location) to repeating patterns like (:Researcher)-[:AUTHORED]->(:Paper)-[:AUTHORED_BY]->(:Researcher)

Traversing upward is just a different view with Direction.INCOMING:

@GraphView
data class LocationAncestry(
    val location: Location,
    @GraphRelationship(type = "HAS_LOCATION", direction = Direction.INCOMING, maxDepth = 3)
    val parent: LocationAncestry?  // Nullable — each location has at most one parent
)

Load “France” and you get France → Western Europe → Europe → null. Same model, different direction, same compositional approach.

You can also override depth at query time:

graphObjectManager.loadAll<LocationHierarchy> {
    depth("subLocations", 5)  // Override the annotation's maxDepth=3
    where { query.location.type eq "continent" }
}

And for structures where the cycle goes through multiple types — Person → Organization → Person — Drivine tracks visit counts and terminates at maxDepth, so you get bounded traversal without manual bookkeeping.

What’s Next

The original TypeScript Drivine supports multiple Cypher-compatible graph databases — not just Neo4j, but any engine that speaks the language. In principle, Drivine4j could do the same. The architecture doesn’t assume Neo4j internals; it generates standard Cypher.

In practice, Neo4j is miles ahead. And not all graph databases support the Cypher features Drivine leans on — pattern comprehensions, `labels()`, `EXISTS` subqueries. The further you go from Neo4j, the more you’d need to negotiate what’s available. It’s doable, but there may be compromises for other platforms.

Still, the door is open. If the Cypher ecosystem continues to converge — and there are signs that it will — multi-database support becomes a natural extension rather than a rewrite.

Wrapping Up

Drivine4j started as a reaction to a real problem — an OGM that made a fast database slow — and grew into a framework with a genuine point of view. Let’s recap what it offers.

Two levels of abstraction, because that’s how many you actually need:

- The PersistenceManager lets you write your own Cypher — tuned, tested, readable — and project the results onto typed objects, or return one or more simple scalar values. Drivine handles transactions, connection pooling, and the infrastructure plumbing. You keep control of the query. This is the right choice when performance matters, when the query is complex, or when you simply want to see exactly what’s hitting the database.

- The GraphObjectManager generates Cypher from compositional models. @NodeFragment, @RelationshipFragment, and @GraphView compose into use-case-specific views — not one god-entity for all scenarios, but one model per question you’re asking. The generated Cypher is readable and debuggable. Dirty tracking means saves only write what changed. Cascade policies give you control over deletions.

The type-safe DSL deserves a special mention. Built via KSP code generation, it gives you compile-time checked queries with full IDE autocomplete — filter across nested relationships, sort collections database-side with APOC, query polymorphic types with instanceOf<T>(). It makes the GraphObjectManager feel less like a framework and more like a language feature. And if Java is your thing, there’s a fluent Java API that mirrors the Kotlin DSL — filterWith(), where(), whereAny(), orderBy() — pretty much feature-parity with Kotlin.

Recursive relationships let you model ontologies and hierarchies — the structures that power knowledge graphs and AI reasoning — with the same compositional approach that works for everything else.

And the generated Cypher? You can read it. You can paste it into the Neo4j browser and debug it. You can show it to someone who’s never seen Drivine and they’ll understand what it does.

If you’re working with Neo4j and you’ve felt the friction of the OGM — the slow queries, the over-selection, the generated Cypher you can’t debug — Drivine4j might be worth a look. It’s open source and we make good use of it in the Embabel ecosystem.

Next up: I’ll show how we used Drivine4j to build embabel-chat-store — a conversation persistence layer for AI agents where the graph model really shines. Multi-party messages, explicit attribution, async persistence. I’m looking forward to share that with you. Until then, have fun with Embabel- in our opinion, the best tool for creating Agentic AI applications :)

The (Very Slowly) Ticking Time-Bomb in Your Graph Persistence Stack was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of these central APIs is PromptRunner, aptly named because it is used to run prompts against an LLM, typically using input from users, databases, or other services. This post describes a significant change to PromptRunner and related types, aimed at pruning this core API and preparing it for future growth.

PromptRunner serves as the main entry point and gateway for general LLM interaction. Specialized operations that are not available directly on PromptRunner are exposed through more focused APIs, accessed via factory methods.

For example, to create strongly typed objects, you use promptRunner.creating(Class). This returns an API dedicated to object creation, with support for examples, property filtering, and disabling validation.

Before the change described in this post, the following factory methods existed on PromptRunner, along with the interfaces they returned:

• creating(Class<T>), returning an ObjectCreator<T> for object creation,
• withTemplate(String), returning TemplateOperations for rendering templates,
• stream(), returning StreamingPromptRunnerOperations for reactive streams,
• withThinking(), returning ThinkingPromptRunnerOperations for thinking prompts.

Several issues became apparent when looking at this arrangement.

Some methods used the with prefix (withTemplate and withThinking), while others did not. Strictly speaking, a “with-er” creates a new instance of the same immutable type with one or more modified properties. Since these methods return different types (TemplateOperations and ThinkingPromptRunnerOperations respectively), they are not true with-ers and should therefore be named differently. Dropping the with prefix has an additional benefit: these factory methods are no longer buried among configuration with-ers when invoking Ctrl-Space.

Some interfaces used theOperations suffix; others did not.

Overall, there was little overall consistency, both in the names of the factory methods and in the names of the interfaces they returned.

The Revised API

When we addressed these problems, the team expressed a clear preference for the name creating. Based on that decision, we introduced gerund forms for the other factory methods as well:

• PromptRunner::creating remained unchanged
• withTemplate was renamed to rendering
• stream was renamed to streaming
• withThinking became thinking

Finding suitable names for the returned interfaces proved more challenging. Several naming schemes were considered, but in the end we chose to give the interfaces the same names as their corresponding factory methods.

We also nested these interfaces into PromptRunner itself. Since these types are not meant to be instantiated directly, but only obtained through a PromptRunner, this gives them a more appropriate scope and avoids adding more types to the global namespace. While inner types result in longer names, most users will rarely need to reference these types explicitly, as they move from PromptRunner to the appropriate sub-runner as part of the same method chain.

As a result, the names now follow a consistent and predictable pattern:

• creating(Class<T>) returns PromptRunner.Creating<T>
• rendering(String) returns PromptRunner.Rendering
• streaming() returns StreamingPromptRunner.Streaming
• thinking() returns PromptRunner.Thinking

To maintain backward compatibility, the original methods remain available but are now deprecated.

This structure makes the relationship between factory methods and their sub-runners explicit, removes unnecessary affixes, and applies a consistent naming scheme.

Try Embabel’s Improved API

The changes described here are just one of many improvements underway as Embabel moves toward its 1.0 release. Please give the updated API a try and let us know how it works for you.

Feedback on this change, or any other aspect of Embabel, is always welcome on our Discord.

Recent Improvements to Embabel’s PromptRunner was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.

Embabel DICE

This example is heavily inspired by the example provided by the Embabel team, Impromptu. The example includes an extensive README; the DICE source is available. More documentation on DICE is also available.

A big problem when talking to an LLM, which most agents do, is the grounding gap. This is the lack of common ground between the LLM and the user’s question. The LLM generally has no idea what you are talking about. Imagine I have a conversation with the LLM in which I tell it what I like. I tell it I like to read blogposts about Agents and Embabel specifically. We had a nice chat, but I need to end the conversation.

Later that day, I go home and ask the agent or the LLM if there are any interesting new blog posts to read (I'm travelling by train today). By providing my preferences to the LLM, it knows that posts about Embabel should be at the top. Of course, this is a very simple scenario. But what if I can add domain-specific knowledge to a request I make? Think about allergies when ordering food, and your home address when ordering groceries. You name it.

When automating our daily work with agents, they need to understand what we are doing. That is where Domain Injected Context Engineering comes in.

In this blog, I create an example that is not too complex to understand using DICE and Embabel, along with a GUI to follow what the agent is doing.

GUM, the theory backing DICE

As mentioned in the documentation, DICE is heavily inspired by the paper that introduces General User Models (GUM). The paper is good to read.

Creating General User Models from Computer Use

Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, Michael S. Bernstein

https://arxiv.org/abs/2505.10831

I do not want to give a full summary of the article; read it for yourself. But I want to describe some of the highlights for me.

The idea from GUM is to learn from user interactions with a computer. If conversations have common ground among the involved parties, they just run better. If the system can learn about the user, it can create that common ground.

By observing, the system creates propositions on what a user wants to accomplish. The system also gives a confidence factor about the accuracy of the proposition. After new observations, the system can revise some of the propositions. The system also audits the propositions for facts you do not want the system to remember.

I also liked the staleness factor of a proposition. Some memories you want to keep longer than others. This factor is part of the proposition extracted by the system.

The steps

In this section, I give you an overview of what we will do. I’ll explain the different steps, so you have a better idea of what is coming.

Have an Agent capable of talking to you

This is where I left off in the previous blog. A ChatBot with RAG, making use of Lucene to index blogs and expose it as a tool to the ChatBot. Based on the impromptu sample mentioned, I made some improvements before adding the new preference extraction. The biggest change is that the user now stores the processId; therefore, the client application no longer needs to pass it to the backend.

Attaching the user to the agent process is now performed in an Embabel action. Note how the user is obtained from the OperationContext.

@Action
public KnowledgeUser bindUser(OperationContext context) {
    var forUser = context.getProcessContext().getProcessOptions().getIdentities().getForUser();
    if (forUser instanceof KnowledgeUser iu) {
        return iu;
    } else {
        logger.warn("bindUser: forUser is not an KnowledgeUser: {}", forUser);
        return null;
    }
}

This KnowledgeUser is injected into the action that responds to a UserMessage.

@Action(canRerun = true, trigger = UserMessage.class)
public void respond(Conversation conversation, KnowledgeUser user, ActionContext context) {
    var lastUserMessage = conversation.lastMessageIfBeFromUser();

    if (lastUserMessage != null) {
        logger.info("Received user message as last message: {}", lastUserMessage.getContent());
        var assistantMessage = context.ai()
                .withLlmByRole(CHEAPEST.name())
                .withReferences(toolishRag)
                .withSystemPrompt("You are a helpful assistant. Answer questions concisely. Always address the current user by their name: " + user.getDisplayName() + ".")
                .respond(conversation.getMessages());

        context.sendMessage(conversation.addMessage(assistantMessage));

        eventPublisher.publishEvent(new ConversationAnalysisRequestEvent(this, user, conversation));
    } else {
        logger.info("Received non-user message");
    }
}

If you read my previous post, you can see another difference. In the code, I publish an event. This event, ConversationAnalysisRequestEvent, is essential for integrating the conversation with preference extraction. I get back to this event in a later section.

Have a preference extraction tool

The preference extraction tool receives some text and extracts the propositions for a preference from it. If I enter this text:

I like to watch tutorials about Agents, especially if the speaker uses a bit of humour but clearly knows what he is talking about.

The resulting propositions contain the extracted preferences, the confidence that the preference is right, and the explanation of why the preference was proposed. The first proposition is: User likes to watch tutorials about Agents. The extractor has a confidence of 90% and gives the following reason: User explicitly says they like to watch tutorials about Agents.

The image below shows the tool I created to work with the extractor tool. In Embabel, this part is called the PropositionPipeline. The extraction is exposed by the PropositionPipelineController.

Another important piece of information for a proposition is the decay. A high decay rate means the proposition’s importance decreases rapidly over time. Low decay indicates the proposition remains an important preference for the user. Think about the difference between where you parked your car and the day you got married.

The following diagram gives an overview of the Embabel design for the proposition pipeline. The PropositionPipelineController is a Spring MVC Rest controller that exposes an endpoint for extracting propositions from the provided text. The MemoryController has endpoints for listing and searching the stored propositions, i.e., the memory. It also provides endpoints for creating and deleting propositions.

The PropositionPipeline orchestrates the proposition extraction. It uses the extractor to find propositions, and the revisor to determine whether we are dealing with a new proposition or an update to an existing one. Propositions are stored or updated in the repository.

The following screenshot shows some propositions that were extracted from a conversation I had with the ChatBot. Notice the mentions in the last item. Next to the Knowledge user, there is another entity called Product. In another post, I take a better look at entity extraction. The impromptu sample contains more entities and relations, which are stored in a graph database.

In the following section, I discuss integrating the PropositionPipeline with the Conversation and the ChatBot.

Integrate the extraction tool with the conversation

Now that I have a pipeline to extract preferences from text, I want to use this to extract user preferences from the conversation. The extraction begins by listening to an event I discussed a few sections back. After the user responds to a request, the conversation is published as an event.

The ConversationAnalysisRequestEvent contains the conversation and the KnowledgeUser. I wrote the ConversationPropositionExtraction class using the impromptu sample code. This class listens for the event and uses several components to extract entities and propositions from the conversation. The entities to extract are defined in the DataDictionary. With the help of the ChunkHistoryStore, re-analysing the same chunk is prevented. After extraction, the propositions and entities are stored in the PropositionRepository and the EntityRepository.

Internally, the PropositionPipeline discussed in the previous section is used for the extraction of the propositions. The following diagram gives an overview of the discussed components.

With user preferences extracted as propositions from the conversation, we only need to use the preferences in the context of the agent.

Inject user preferences into the context of the Agent

Propositions are exposed through a memory component. The Memory needs the PropositionRepository and a MemoryProjector. This projector translates proposals as semantic memory (facts), procedural memory (preferences/rules) or episodic memory (events).

Next, the memory is exposed as a reference to the LLM call. The following code block shows the same function as before, now with the included memory.

@Action(canRerun = true, trigger = UserMessage.class)
public void respond(Conversation conversation, KnowledgeUser user, ActionContext context) {
    var lastUserMessage = conversation.lastMessageIfBeFromUser();

    if (lastUserMessage != null) {
        var memory = Memory.forContext(user.getCurrentContext())
                .withRepository(propositionRepository)
                .withProjector(memoryProjector);

        logger.info("Received user message as last message: {}", lastUserMessage.getContent());
        var assistantMessage = context.ai()
                .withLlmByRole(CHEAPEST.name())
                .withReferences(toolishRag, memory)
                .withSystemPrompt("You are a helpful assistant. Answer questions concisely. Always address the current user by their name: " + user.getDisplayName() + ".")
                .respond(conversation.getMessages());

        context.sendMessage(conversation.addMessage(assistantMessage));

        eventPublisher.publishEvent(new ConversationAnalysisRequestEvent(this, user, conversation));
    } else {
        logger.info("Received non-user message");
    }
}

More code

In the previous section, you read about the concepts and were introduced to the design of the Embabel ChatBot with DICE. You can now read the sample in the git repository. I always like to see the code with some explanation; therefore, I go over the same steps, but now show the code.

Maven

For some features, I needed the bleeding edge. At the time of writing, I am using version 0.3.3-SNAPSHOT for embabel. (0.3.3–20260126.051307–64) I will update the sample when a release becomes available that contains the things I need.

Therfore you do need to add the snapshot repo from Embabel to your pom.

    <repositories>
        <repository>
            <id>embabel-snapshots</id>
            <url>https://repo.embabel.com/artifactory/libs-snapshot</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

DICE is a separate module; we need that dependency as well.

<dependency>
    <groupId>com.embabel.dice</groupId>
    <artifactId>dice</artifactId>
    <version>0.1.0-SNAPSHOT</version>
</dependency>

Again, a snapshot dependency, 0.1.0–20260125.055345–4, to be specific.

The code for the PropositionPipeline

Most of the code is Spring configuration code. So a lot of beans. I start with the bean for the proposition repository.

@Bean
PropositionPipeline propositionPipeline(
        PropositionExtractor propositionExtractor, 
        PropositionReviser propositionReviser, 
        PropositionRepository propositionRepository) {
    
    return PropositionPipeline.withExtractor(propositionExtractor)
            .withRevision(propositionReviser, propositionRepository);
}

Just like in the design diagram you have seen in a previous section. The pipeline has three dependencies. It needs an extractor, a reviser and a repository.

First, the repository needs the model provider to select an Embedder. I like to select it by role. This is configured in the application.yml and an enum. I use the InMemory version of the repository. It only needs an embedding model to create the vectors for similarity search.

@Bean
PropositionRepository propositionRepository(ModelProvider modelProvider) {
    return new InMemoryPropositionRepository(
            modelProvider.getEmbeddingService(ByRoleModelSelectionCriteria.Companion.byRole(FAST.name()))
    );
}

Next, we need the reviser. The reviser uses an LLM to find existing revisions of a proposition.

@Bean
PropositionReviser propositionReviser(AiBuilder aiBuilder) {
    var ai = aiBuilder
            .withShowPrompts(true)
            .withShowLlmResponses(true)
            .ai();
    return LlmPropositionReviser
            .withLlm(LlmOptions.withLlmForRole(STANDARD.name()))
            .withAi(ai);
}

Notice that we tell Embabel to show the prompt and the LLM response. In the impromptu example, these properties are extracted from a Properties object to make them configurable. That is better, but this is easier for now.

Finally, you need the extractor. This is a bigger one, follows the same structure as the reviser, but with a big twist.

@Bean
LlmPropositionExtractor llmPropositionExtractor(
        AiBuilder aiBuilder,
        PropositionRepository propositionRepository) {
    var ai = aiBuilder
            .withShowPrompts(true)
            .withShowLlmResponses(true)
            .ai();
    return LlmPropositionExtractor
            .withLlm(LlmOptions.withLlmForRole(STANDARD.name()))
            .withAi(ai)
            .withPropositionRepository(propositionRepository)
            .withSchemaAdherence(SchemaAdherence.DEFAULT)
            .withTemplate("extract_user_propositions");
}

Notice that the code uses a template for the LLM prompt. Again copied at first from the impromptu sample. But made it work for my sample. It is a long one, but I think it is worth a look.

You are extracting propositions about blog writers and readers
from their conversation.
Extract facts about the USER only -
their preferences, knowledge, interests, and opinions about blogs and technology.

{% include "dice/schema_hints.jinja" %}

{% include "dice/existing_propositions.jinja" %}

## Extraction Rules

1. **User-centric**: Extract facts about the user, not general blogging knowledge
2. **Single fact per proposition**: One subject, one object maximum
3. **No inference beyond evidence**: Only extract what the text explicitly supports
4. **No overlap**: Each fact should be distinct

## Examples

User says "I want to read blogs about Embabel":
✓ GOOD:
  - "Jettro likes to read blogs" (NO entity mention - period/style)
  - "Jettro likes blogs about Embabel" (entity: Embabel → Product)
  - "Jettro programs best in Python and Java" (entity: Java -> ProgrammingLanguage)

## Confidence & Decay

- **High confidence**: Explicit statements ("I love reading about AI")
- **Lower confidence**: Implied preferences (asking many questions about a topic)
- **Low decay**: Stable preferences (favorite topic, category preferences)
- **High decay**: Transient states (current writing topic)

## Input

User: {{ user.name }}

<conversation>
    {{ model.chunk.text }}
</conversation>

Notice how the template includes templates provided by DICE. The template contains the extraction rules, examples and explains the concept of confidence and decay.

With this configuration, the pipeline is usable by the provided PropositionPipelineController to request propositions for a given piece of text.

The code for extracting propositions from the conversation

Next up, the class that listens for ConversationAnalysisRequestEvent and handles them by starting the extraction process. This is the ConversationPropositionExtraction, which contains most of the required code to work with propositions.

The class needs an entity resolver, an entity repository, a proposition pipeline, a proposition repository, a data dictionary and a chunk history store. The proposition pipeline and repository are already available.

First, the data dictionary. This contains the entities that can be extracted and used in the propositions. At this moment, I have the KnowledgeUser, the Product and the ProgrammingLanguage.

@Bean
@Primary
DataDictionary blogDataDictionary() {
    var schema = DataDictionary.fromClasses("blog",
            Product.class,
            ProgrammingLanguage.class,
            KnowledgeUser.class
    );
    logger.info("Initialized data dictionary with classes: Product, ProgrammingLanguage");
    return schema;
}

This DataDictionary is used by the SchemaRegistry; in my case, again, an InMemory version.

@Bean
SchemaRegistry schemaRegistry(DataDictionary blogDataDictionary) {
    var registry = new InMemorySchemaRegistry(blogDataDictionary);
    logger.info("Initialized schema registry with classes: Product, ProgrammingLanguage");
    return registry;
}

All the other required beans are InMemory implementations.

@Bean
EntityResolver entityResolver() {
    return new InMemoryEntityResolver();
}

@Bean
ChunkHistoryStore chunkHistoryStore() {
    return new InMemoryChunkHistoryStore();
}

@Bean
NamedEntityDataRepository namedEntityDataRepository(DataDictionary dataDictionary, ModelProvider modelProvider) {
    return new InMemoryNamedEntityDataRepository(
            dataDictionary,
            modelProvider.getEmbeddingService(ByRoleModelSelectionCriteria.Companion.byRole(FAST.name()))
    );
}

The constructor of the ConversationPropositionExtraction initialises the IncrementalAnalyzer. This analyser is responsible for analysing the conversation to extract propositions, if required.

this.analyzer = new PropositionIncrementalAnalyzer<>(
        propositionPipeline,
        chunkHistoryStore,
        MessageFormatter.INSTANCE,
        config
);

The following codeblock shows the extraction and persistence of the propositions. I removed some logging; check the actual source code if you want the complete picture.

public void extractPropositions(ConversationAnalysisRequestEvent event) {
    logger.info("Starting proposition extraction for conversation with {} messages",
            event.conversation.getMessages().size());

    var messages = event.conversation.getMessages();
    if (messages.size() < 2) {
        logger.info("Not enough messages to extract propositions, need at least 2");
        return;
    }

    var context = SourceAnalysisContext
            .withContextId(event.user.getCurrentContext())
            .withEntityResolver(entityResolverForUser(event.user))
            .withSchema(dataDictionary)
            .withKnownEntities(KnownEntity.asCurrentUser(event.user))
            .withPromptVariables(Map.of(
                    "user", event.user
            ));

    // Wrap conversation as incremental source and analyze
    var source = new ConversationSource(event.conversation);
    var result = analyzer.analyze(source, context);

    if (result == null) {
        logger.info("Analysis skipped (not ready or already processed)");
        return;
    }

    if (result.getPropositions().isEmpty()) {
        logger.info("Analysis completed but no propositions extracted");
        return;
    }

    result.persist(propositionRepository, entityRepository);
}

The last line adds the new entities and propositions to their respective repositories.

The code to use the propositions

I already discussed this code in the previous section, when I discussed the changes to the action to add the memory as a resource to the LLM call. The only thing missing is the memory projector's config.

@Bean
MemoryProjector memoryProjector() {
    return DefaultMemoryProjector.DEFAULT;
}

Running the application

One thing I really like about Embabel is the logging. You can follow everything from the logs. With the trace of Agent events, you also track LLM calls, tool calls, and, of course, their responses.

First question:

I am really interested in blogs about Embabel. Do you have information about such blogs?

In the log, you can find this line. Notice how “I” is extracted as a KnowledgeUser.

20:32:05.318 [tomcat-handler-16] INFO ConversationPropositionExtraction - ChunkPropositionResult(chunk=423b39ea-0004–4a5f-8873–1317c44fe2a4, propositions=2, entities: 1 new, 0 updated, revision: 2 new, 0 merged, 0 reinforced, 0 contradicted, 0 generalized)
 Propositions:
 • Jettro is really interested in blogs about Embabel (conf: 0.98) [NEW]
 • Jettro asked for information about blogs about Embabel (conf: 0.95) [NEW]
 Entities:
 • [NEW] I (KnowledgeUser, __Entity__)

After some questions and answers, go to the propositions tab to find what propositions are extracted. I removed a few to make the list easier to read. I personally like the middle one, where a proposition is extracted from the assistant's response.

References

That is it. I hope you learned something from this blog. I know I learned a lot while writing the blog and creating the sample. I keep adding features to the sample. I try to keep track of the latest and greatest Embabel features.

Drop me a comment if you liked it or if you have a question.

• Blogs about Embabel: https://medium.com/embabel
• Impromptu sample: https://github.com/embabel/impromptu
• Embabel homepage: https://github.com/embabel/embabel-agent
• DICE: https://github.com/embabel/dice
• This sample: https://github.com/jettro/embabel-agent-rag-sample

Agents that extract and use preferences from conversations. was originally published in Embabel on Medium, where people are continuing the conversation by highlighting and responding to this story.