This article was originally published here. Sign up for free to never miss out and join 200K weekly readers getting the most important insights in AI delivered to their inbox.

A quick note before we begin: This article was largely written before Jensen Huang’s departure from NVIDIA on November 12. The circumstances surrounding his exit have received plenty of coverage elsewhere, so I won’t be revisiting them here. What interests me more is the remarkable business and engineering strategy he spent decades building, and how Nvidia has set themselves up to continue their position as the worlds most valuable company.

Today, we will be exploring one of the most misunderstood companies in the AI Industry right now. Despite their position at the top of the stock market, NVIDIA’s deeply technical business and unique customer dynamics make it hard for investors and analysts to put them in a box. NVIDIA also tends to make very intelligent long-term plays that are hard to understand for an industry typically obsessed with quarters. In this article we will be breaking down their strategy in great detail covering what they’re doing, why, how it can go wrong, and what would have be the case for things to work out. Keep reading if that interests you.

NVIDIA made $89 billion from data centers last quarter, growing 117% YoY, with company-wide gross margins of 75%. This is why their substantial investments in open-source AI confuse a lot of people:

• Nemotron: Its new Bolt model squeezes 74 billion parameters into 24GB of memory, while an even smaller model fits into just 4GB. Both are designed to make capable AI accessible on local hardware.
• Dynamo: NVIDIA’s open-source inference software promises up to 7× better inference performance on Blackwell GPUs, allowing customers to process far more requests with the same hardware.
• Vera Rubin: NVIDIA reports up to 45× lower inference costs than Blackwell Ultra on a selected benchmark, continuing its push to make each generation substantially cheaper to operate.

Why would a company making most of its money from selling expensive computing hardware spend so much effort helping customers need less of it? Wouldn’t that eventually undermine its own business by letting people use less hardware for the same level of intelligence?

There’s actually a really smart reason for this, and it’s the kind of far-sighted strategic thinking that led NVIDIA to become the most valuable company in the world to begin with (remember that in the beginning the media and Wall Street relentlessly mocked CUDA and failed to understand NVIDIA’s ecosystem play). Presently, most of NVIDIA’s leather jacket money and growth is tied to a few “whale” companies that drop billions on AI infrastructure (one direct customer alone accounted for 16% of its revenue last quarter). These whales mostly pay the high margins to NVIDIA to ensure that their training runs proceed smoothly since the cost of paying for premium hardware is much lower than the cost of having your expensive engineering teams debug new hardware on frameworks they aren’t familiar with —

But this creates another issue. If a company can afford to spend billions buying GPUs, it can also (… cue the drum rolls) afford to spend billions finding ways to build their own baby formula instead of paying top dollar to suckle on Jensen Huang’s plentiful teats. And so you have all of NVIDIA’s biggest customers making their own chips: Google has its TPUs, Amazon has Trainium, OpenAI recently broke the market with their Jalapeno chips, all while AMD is competing for a share of the same market.

Essentially, it’s a war on all fronts for NVIDIA’s empire.

(If you’re interested, we broke down this dynamic in great detail in our article How AI will Change in 2026 predictions. It contains deep dives on how hardware is changing with so many new entrants, deep technical splits bw inference and training chips, and other detailed insights into how the AI industry is changing. Read it here) —

Back to our main article, this revenue concentration + incentive structure is why, we’ve seen so much capital and attention flow into the AI Inference Market. Unlike training frontier models, inference is something almost every business could eventually need. And reducing its cost doesn’t just save money on existing applications. It can make applications that were previously too expensive worth deploying in the first place. Consequently, in just 2026, six inference infrastructure providers have announced over $7 billion in financing, with NVIDIA itself participating in several rounds.

This sets up interesting question numero dos — why is NVIDIA investing in people that (at least superficially) bet against (their) GPUs?

The simple answer here is market expansion (also known as Jevons Paradox) — Jensen is betting that if the cost of inference goes sharply enough, it will open up whole new use cases for AI that will dramatically expand the demand for AI (we’ve seen this happen w/ tokens where the fall in token prices has made AI companies much more valuable since they’ve unlocked much more usage; the bet now is that we will unlock the efficiency involved with economies of scale to eventually make tokens more profitable as it did for other infra buildouts/commodities).

But does that work? What would make this successful, and when would this fail? In this article, we will be exploring 3 questions —

1. How (on a technical level) NVIDIA is making AI much cheaper from everyone.
2. The market expansion opportunities this creates.
3. What needs to happen for this to be a game-winning play instead of NVIDIA undercutting its own business.

If you’re unsure about what the long term play for NVIDIA is, this company map will help you understand one of tech’s most misunderstood companies.

Executive Highlights (TL;DR)

The core thesis: NVIDIA is deliberately making AI cheaper to expand the market for its hardware. But Jevons Paradox doesn’t guarantee that NVIDIA gets richer. For this strategy to work, customers must spend more on AI, that spending must require additional infrastructure, and NVIDIA must capture a profitable share of it.

• The engineering is impressive, and it’s happening across the entire stack. LatentMoE offers up to 3.5× higher modeled throughput, Nemotron Bolt compresses 74B parameters into 24GB, and Dynamo improves utilization of existing clusters. Rubin and Groq LPX take this further by redesigning how inference runs across entire racks. NVIDIA isn’t just building faster GPUs; it’s trying to reduce the cost of intelligence at practically every level.

• There is real evidence that cheaper AI creates more valuable work. Anthropic’s revenue grew 13× while compute expenses grew 4.3×, with over 11,900 customers spending $100K+ annually. But the opportunity varies by application. Routine workloads eventually become cheap commodities, while coding, research, and autonomous agents can justify increasingly large reasoning budgets. I suspect the latter will drive much of the industry’s profitable compute demand.
• NVIDIA’s biggest customers are also its most dangerous competitors. Google’s TPU v7 beats GB300 by 1.85× on tokens per dollar in one FP8 comparison, but NVIDIA reverses that advantage through FP4 and software optimizations. Google’s next TPU introduces native FP4 support. NVIDIA’s moat increasingly depends on staying ahead in full-stack engineering, not simply producing better silicon.

• The infrastructure buildout introduces a different kind of risk. CoreWeave’s depreciation and amortization already represent 54% of quarterly revenue, while power and construction delays restrict deployments. NVIDIA has committed $279B to manufacturing and supply capacity and provided up to $105B in contingent guarantees for OpenAI-linked infrastructure. These commitments can secure future demand, but become dangerous if paying workloads fail to absorb the increasingly productive hardware.
• My current bet is that the market splits. Routine, predictable inference increasingly moves toward cheaper accelerators and depreciated GPUs, while frontier reasoning and complex agents remain hungry for NVIDIA’s latest systems. Three things will tell us whether this holds: how quickly enterprise AI spending grows, whether competing chips close NVIDIA’s software-performance gap, and what happens to older GPU rental prices as new data centers come online in 2027.

Hey there. I keep this newsletter ad-free, w/o sponsors, and mostly w/o paywalls to ensure that everyone has access to high-quality deep-dives on the most important ideas in the AI Industry without any bias. B/w my research, independent verification, and the chocolate milk needed to produce these works, these deep-dives take a lot of time, resources, and infrastructure. For just 10 USD/month or 100 USD/year, you can help me continue my vision of democratising access to the most important ideas in AI by becoming a paid subscriber over here —

Help me buy chocolate milk

(If that is too much for you, we follow a pay-what-you-can plan over here, and it goes down all the way to 1 USD/month or 10 USD/year. No matter what plan you pick, you will get all of our articles without paywalls and have access to all of our articles. Check it out here).

PS — Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

How is NVIDIA Making AI Cheaper

When it comes to making AI Cheaper, there are actually several ways you can eat that horse —

• First, you can reduce the amount of raw math required to generate each token. Refer here to our exploration of How Much an AI Token Costs to get the full details.

• Second, you might choose to cut down the amount of memory you have to buy just to hold the model’s weights and conversation history.
• Third, you may reduce the standard autoregressive generation so that people aren’t wasting your high end GPUs on token by token generation. The most extreme version of this are diffusion models, which are I believe will be the future of Language/Multimodal Models for the reasons mentioned here.

• Fourth, there’s cluster utilization, because a data center full of expensive GPUs still loses money if half the chips sit idle waiting on network transfers. When we studied WEKA a few months back, we discovered that this problem isn’t something that can be solved by simply spamming more hardware, and there since then I have come across a ton of interesting engineering focused on cluster utilziation. I think this spac is generally a bit overlooked by analysts studying it (it is a somewhat crowded market now, but there is a lot of moving dynamics in there constantly).
• And fifth — often the biggest bill of all — is the months of human engineering required to clean data, wire up pipelines, and build the application before you ever serve a single request.

As we will learn, NVIDIA has shipped a wave of models, software schedulers, and rack-scale chips to attack these bottlenecks. We’re going to go through each layer, one by one, starting with how Nemotron 3.5 rewires the Mixture-of-Experts architecture to drive more efficiency.

(Fair warning, this will be a long section since the goal is to give you a really good understanding of the engineering fundamentals of their solutions and how they impact various industry focal points. If you’re busy and not interested in the engineering, the TLDR + this image map is a good enough to skip ahead, but I always recommending building your intuition into the platforms so you can build your own assessments).

How Does LatentMoE Fix the Hidden Data-Movement Costs in Nemotron 3.5?

NVIDIA’s Nemotron 3.5 release dropped two different models to match two operating budgets:

• Nemotron 3.5 Super: A 120-billion-parameter multimodal model that activates roughly 12 billion parameters per token, built for heavy reasoning, long contexts, and multi-step agentic workflows.
• Nemotron 3.5 Lightning: A 30-billion-parameter text model that activates roughly 3 billion parameters per token, stripped down for high-speed, low-cost inference.

Both models combine hybrid Mamba-Transformer layers, native speculative decoding, and Mixture-of-Experts (MoE) routing, the last of which comes with an interesting seasoning. Traditionally, MoE cuts your costs by routing each token to only a small fraction of the network’s experts; helping you get the knowledge capacity of a massive model while paying for the inference of a smaller one. That’s why MoE has become a mainstay in all modern LLMs (to the point that it was actually surprising that Meta refused to MoE for as long as they did for their Llama models).

Source

However, even when you cut the FLOPs with MoE, you have to pay for the freeloading inactive experts in memory (AI models got free housing before you did), read their weights into the compute units, and — when the model is split across multiple GPUs — ship token activations back and forth across the network. That data movement creates two nasty bottlenecks depending on your traffic regime:

• When concurrency is low (say, a few interactive users), the GPU spends most of its time waiting on high-bandwidth memory (HBM) wires just to stream in the selected expert weights.
• When concurrency is high across a multi-GPU cluster, the interconnect saturates as the system dispatches full-width token vectors to whichever remote GPUs hold the chosen experts. While MoE reduces the total number of calculations for you but you’re still streaming lots of body positive bytes across your network, which adds a cost of it’s own.

This is what LatentMoE was built to solve. Simply put, it reduces the dimensions of your input vectors, making processing much cheaper (look at the image above). More mathametically, given an input representation h_in of width d the layer multiplies h_in by a shared down-projection matrix W_down to compress it into a much smaller latent dimension l (where l << d). All of the routed experts do their math inside that tiny l-dimensional space, and a shared up-projection W_up restores the result back to dimension d at the end:

h_out = W_up * Sum_i( g_i(h_in) * Expert_i(W_down * h_in) )

I gave you the longer math b/c I want you to look closely at the router gating function g_i(h_in) in that equation. While the experts operate on the compressed vector W_down * h_in, the router itself still looks at the uncompressed hidden state in full dimension d. This makes sure your routing decisions don’t lose important fine-grained features that get wiped out during compression.

(As an aside, notice how more and more companies are embracing reasoning in the latent space?? When this becomes mainstream, don’t forget that I was the first one to call it)

With these cost savings, each latent expert becomes much cheaper to read from HBM much lighter on the network; so Nemotron can pack in far more total experts and activate a much higher top-k combination of experts per token without spiking serving costs. This allows them to hit the perfect top-bins half volley of intelligence per dollar, the metric that the entire AI Industry is now chasing when they realized that AI models eventually need to make money AND that perhaps you don’t astra level intelligence to edit your spreadsheets based on your macros. The results speak for themselves: conventional MoE would need roughly 350 billion extra parameters just to match LatentMoE’s accuracy. Under modeled distributed serving assumptions, LatentMoE delivers up to 3.5x higher throughput at matched accuracy.

From NVIDIA: Standard MoE versus LatentMoE architecture

Naturally, when model calls get cheaper, man is tempted to spam them in autonomous agents. So we gotta look at the the next thing that breaks when you deploy Nemotron as an autonomous agent?

The answer is Context.

How NVIDIA reduces the cost of long context reasoning.

Imagine a software-engineering agent that has been running for four hours — reading repository trees, inspecting pull requests, running compilers, parsing build logs, and calling tools. As one might imagine, you are going to end up burning through a lot of tokens.

As we’ve discussed at length, long context reasoning in vanilla transformers has two major costs:

• During prefill, full attention scales quadratically as O(N²) in sequence length N.
• During token generation, caching past keys and values avoids recomputing the whole sequence, but that KV cache still grows linearly as O(N) in HBM — eating up gigabytes of memory per active agent and choking memory bandwidth on every single decode step.

This is why the entire AI industry has been moving away from vanilla transformers and instead putting all their attention on building kinky transformers (this isn’t the scientific term yet, but I’m doing my best to get it to catch on).

NVIDIA’s version of kinky transformers involves interleaving two different techniques together (generally a good approach to life) —

1. Mamba models fold each new token into a fixed-size O(1) recurrent state S_t = A_t * S_(t-1) + B_t * x_t. Because that state never grows, processing token 800,000 takes the exact same memory footprint as processing token 100. This raised a lot of eyebrows when it was released, but it never caught on as much since this recurrent is very lossy, especially when precise recall over time is needed.
2. Vanilla transformer layers rely on full self-attention on the input. These have perfect recall within their context window (technically a very dubious claim, but we will let it slide for now). So Nemotron takes a page out of newer edge models like Gemma 4 and Liquid AI to interleave full Transformer attention layers at periodic intervals — 4 Mamba layers to full self attention layer so that it can cut down the cost of long context inference.

Source

Naturally, accommodating such a big change also requires them to rework their training pipeline a bit to make sure they can still work over a million tokens—

• NVIDIA started by distilling Nemotron 3.5 Super across roughly one trillion tokens from a larger teacher model, then ran 65.4 billion tokens of specialized software-engineering mid-training over code repositories, pull requests, build logs, and agent trajectories.
• Hammering the model on dense software-engineering trajectories made it a much better coder, but caused catastrophic forgetting on long-range retrieval.
• To fix that damage, NVIDIA ran an extra nine-billion-token long-context recovery stage, which pulled Super’s RULER-1M long-context score back from a degraded 33.5 to 81.0.

Their acquisition of Gretel and subsequent investments into Synthetic Data (something we flagged in our March Market Report for 2025) probably helped them get the extra billions of training tokens relatively cheaply. This extra training (cheap) lets them cut the cost of inference dramatically —

Nemotron 3.5 Super hits 51.78% on SWE-bench Pro, beating the previous Nemotron 3 Ultra while activating 4.6x fewer parameters per token.

Okay, so LatentMoE and hybrid Mamba-attention make a 120-billion-parameter model much cheaper to run per token. Are we done? Not if you want to run it on a single GPU. Storing 120 billion parameters in standard 16-bit BF16 still takes 240GB of memory just for the static weights — before you allocate a single byte for the KV cache or runtime activations.

To get frontier-class models onto local workstations and single accelerators, NVIDIA had to figure out how to crush the weights themselves without lobotomizing the model.

How Nvidia uses Star Elastic and Two-Bit Quantization to Fit a 74B Model into 24GB?

In October, NVIDIA released Nemotron 3.5 Bolt, a model with 74 billion total parameters and 9 billion active parameters, which they somehow fit inside a 24GB memory budget (for context, storing 74 billion params in standard BF16 takes roughly 148GB of VRAM, so they squeezed them params 6x). Let’s understand how they pulled off this black magic.

First we got a lil something they call Star Elastic. Instead of chopping of layers randomly (which tanks your performance b/c some layers carry more importance than others), Star Elastic runs a router-guided neural architecture search constrained by your target hardware budget. It looks at how the router actually uses different layers, then varies both the total number of experts and their intermediate widths layer by layer to keep maximum capacity where it counts. SE pruned Nemotron 3.5 Super down from 120B total parameters (12B active, 22 routed experts per token) to the 74B total parameters (9B active, 9 routed experts per token) in Bolt, followed by recovery training to restore vision, reasoning, and agentic tool use.

Step two is where things get aggressive. Rather than quantizing the whole 74B checkpoint to a uniform bit width, NVIDIA matches the bit precision of each layer family to how badly it reacts to quantization noise:

• Major feed-forward expert matrices (which eat up the vast majority of total weight storage) are crushed down to IQ2_XXS (this was a new one for me, crazy how many detailed there are in AI these days) at an average of 2.0625 bits per element.
• Mamba linear layers are stored in four-bit NVFP4.
• Attention weights, token embeddings, the language-model output head, and LatentMoE projection/router weights are kept at eight-bit Q8_0, protecting the sensitive pathways where rounding errors would otherwise snowball across layers.

Combining mixed-precision assignments with quantization-aware distillation reveals a non-obvious rule about memory-constrained AI: if you only have 24GB of VRAM, a massive 74B model crushed down to ~2 bits can actually beat a smaller model stored cleanly at 4 bits. The larger network learns high-level reasoning circuits during pretraining that survive aggressive weight compression better than a smaller network can ever learn from scratch.

It’s worth noting that unpacking ultra-low-bit weights and running 9B active parameters means that Bolt runs at roughly 84 tokens per second, vs smaller four-bit competitors like Qwen3.6 (~97 tok/s) and Gemma 4 (101 tok/s). You’re trading raw token velocity for heavyweight reasoning inside a single 24GB card.

Running two-bit lookup tables works well on a local workstation, but in a massive data center, you don’t want compute units wasting cycles unpacking software quantization formats. This is where our next round of optimizations becomes very handy.

How Does NVFP4 Preserve BF16 Accuracy at Four-Bit Precision?

Shrinking weights from 16 bits to 4 bits only gives you throughput benefits if the GPU’s Tensor Cores can multiply four-bit numbers directly without destroying model accuracy. Unfortunately, neural network tensors are full of massive outlier values; try to stretch a single scaling factor across a whole tensor (to accomodate the outliers), and all the normal values get squashed into the rounding bucket.

With its Blackwell architecture, NVIDIA baked a hardware fix for outliers directly into the silicon via NVFP4. Each weight uses a tiny 4-bit E2M1 floating-point format (1 sign bit, 2 exponent bits, 1 mantissa bit), backed by a two-level hierarchical scaling scheme:

1. Every tiny microblock of 16 contiguous values gets its own dedicated 8-bit FP8 scale factor.
2. The entire tensor shares a second, higher-precision global scale factor.

Because every 16-number block gets its own local scale, an outlier in position 4 only stretches the scale for its 15 immediate neighbors, leaving the rest of the matrix untouched.

NVFP4’s two-level scaling mechanism

Storing those extra microblock scales eats up a bit of memory (adding 0.5 bits per weight of scale metadata, so you don’t get a literal 4.0x footprint cut from BF16). This means that on Nemotron 3.5 Super, the NVFP4 checkpoint holds 99.62% of median BF16 accuracy across 36 benchmarks while running at 2.43x higher interactive serving throughput.

If you want to go even lower, NVIDIA pushed the same compression playbook down to the edge with Nemotron 3.5 Bolt E4B by taking a Gemma 4 E4B base and running NVIDIA post-training, to make Bolt E4B fits inside a strict 4GB memory budget. This geodude build model packs a 64K context window, tool calling, and toggleable reasoning, beating NVIDIA’s previous small model by roughly 40% across conversational-agent benchmarks and letting developers run continuous local intelligence.

Read more: Bolt E4B benchmarks and local-agent demonstrations

At this point, we’ve shrunk active FLOPs with LatentMoE, tamed the KV cache with Mamba, and compressed the weights with Star Elastic and NVFP4. At this point, there’s very little you can do to speed up the generation of tokens without reducing their quality. So can we finally go home now?

Lol, no. Remember that individual tokens are largely worthless. Valuable outputs require many tokens. So if we can’t speed up one token, perhaps the next frontier is to explore how we can speed up the generation of groups of tokens.

Read more about Spec Decoding and other Inference Optimization here.

How Do LiLiCorr and Dory Trade Extra Compute for Faster Tokens and Deeper Reasoning?

Even if a single forward pass is fast, generating a 400-token response requires 400 back-to-back forward passes. Speculative decoding breaks that serial chain by using a tiny, fast “drafter” model to guess several future tokens at once, then letting the big target model verify the whole batch in a single parallel pass. Spec Decoding relies on the assumption that, on average, a good portion of the draft model’s speculations are correct. This means that a good portion of the inference is done by a much smaller and cheaper model. Instead of paying for k target model passes, we pay for 1 target model pass and k (much cheaper) draft model passes. Since the draft model is significantly smaller and faster, the overall cost per generated token is substantially reduced.

There is an interesting dilemna with Spec Decoding:

• If the drafter predicts tokens sequentially, drafting itself becomes slow.
• If the drafter predicts multiple future positions in parallel, it runs fast — but because each position is guessed independently, the words often clash with each other. For example, imagine that position t+1 assigns high probability to both dog and cat, while position t+2 assigns high probability to both barks and meows. A parallel drafter picking the top token at each slot independently can easily output dog meows — a nonsensical pair that the target model immediately rejects, wasting the draft.

LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding — NVIDIA Nemotron fixes parallel incoherence without going back to slow step-by-step neural drafting. Instead of picking one winner at each draft position right away, LiLiCorr keeps a small shortlist of top candidate tokens at every position, forming a candidate lattice. Next, a tiny two-layer Transformer sweeps across that entire lattice in a single parallel pass. Each candidate token gets a representation enriched by its neighbors in the lattice, the drafter’s confidence, and the target model’s last verified hidden state. That single pass outputs compatibility vectors for every candidate, allowing the system to score how well neighboring words fit together in parallel, followed by a dirt-cheap sequential pass over those precomputed numbers to pick the best path.

On Nemotron 3.5 Bolt, a 1.13-billion-parameter LiLiCorr drafter (excluding shared embeddings and output head) proposing 7 tokens at a time averages 4.68 accepted tokens under greedy decoding and 4.25 under probabilistic sampling — a massive latency win when serving interactive users at low concurrency.

While LiLiCorr spends a bit of extra compute horizontally to draft future tokens faster, Nemotron Labs Dory 1.0 spends extra compute vertically to make a small model punch above its weight class. Dory is a 4-billion-parameter NVFP4 model built with input layers, a shared recurrent middle block, and output layers. Instead of passing through the middle layers once, Dory can loop the same hidden state through that shared block multiple times before emitting a token.

Why loop a 4B model instead of just loading a 12B model? Looping lets you keep a tiny 4B footprint in memory while you can unlock a lot of the intelligence in small models by exploring their internal representations better (which perfectly aligns with our latent space reasoning thesis that most models don’t fail due to a lack of intelligence but from inadequate exploration of their internal representations, more details here)

Notice that going from 1 loop to 2 loops adds over 9 points on SWE-bench Verified and 7.6 points on SciCode, while doubling to 4 loops hits diminishing returns on those two benchmarks (though Terminal-Bench 2.1 keeps climbing from 18.1% to 22.2%). To keep 4 loops from taking 4x the KV cache memory, Dory uses last-loop KV caching, overwriting the recurrent block’s cached states on each pass so memory stays completely flat.

Now step back and look at what happens when you take these fast, compressed models and deploy them across a thousand GPUs in a production data center. Even the most efficient model checkpoint will bleed money if the cluster software schedules it poorly.

Dynamo: Getting More Out of Existing GPUs

Even the fastest GPUs waste money when workloads are poorly scheduled. LLM inference has two very different phases: prefill, which processes prompts in parallel and is compute-intensive, and decode, which generates tokens sequentially and is largely constrained by memory bandwidth. Running both on the same GPUs creates unnecessary contention. NVIDIA’s Dynamo separates these workloads into specialized GPU pools, uses NIXL to transfer KV caches between them, and routes requests toward machines that already hold relevant context. This becomes particularly valuable for AI agents making dozens of tool calls, where repeatedly processing the same conversation history wastes substantial compute.

Thanks to their optimizations NVIDIA reports up to 4x faster time-to-first-token and 1.5x greater throughput in agentic workloads. Combined with TensorRT-LLM’s kernel-level optimizations, Dynamo effectively increases the useful capacity of existing GPU clusters without requiring more hardware.

Once Dynamo and TensorRT-LLM squeeze the software bubbles out of the cluster, you hit the hard physical limits of the hardware itself: how fast wires can move activations between chips, and how many megawatts of electricity the local utility will let you pull from the grid. This is where our final optimization comes in.

Why Is NVIDIA Pairing Vera Rubin GPUs with Groq 3 LPX Accelerators?

Vera Rubin takes this system-level optimization into hardware. Its NVL72 rack connects 72 GPUs through NVLink 6, delivering up to 3.6 TB/s of GPU-to-GPU bandwidth to reduce communication bottlenecks in large Mixture-of-Experts models. Rack-level power smoothing also helps operators use more of their available electrical capacity.

NVIDIA reports roughly 10x higher DeepSeek-R1 inference throughput per megawatt than Grace Blackwell, although this doesn’t translate directly into a 10x reduction in total costs. The more interesting development is its integration with Groq 3 LPX, a specialized accelerator system offering 128GB of on-chip SRAM and 40 PB/s of aggregate bandwidth. Rather than forcing GPUs to handle everything, NVIDIA proposes using Rubin for compute-intensive prefill and attention, while Groq handles latency-sensitive feed-forward and expert operations. The combined architecture reportedly delivers up to 35x higher throughput per megawatt than GB200 NVL72 under NVIDIA’s benchmark conditions. The strategic implication is more important than the benchmark: NVIDIA is preparing for a world where GPUs aren’t necessarily the best processors for every inference operation. By supplying the interconnect, specialized hardware, and orchestration software, it can retain control of the computing system even as workloads become increasingly heterogeneous.

There are several other engineering optimizations they make to really drive down the cost of intelligence, but those are lot more serving/topic specific (or are just not that major/novel), so I don’t think we should spend a lot of time covering them. Instead, we must now take all of this is a setup for the question that most of you are likely here for: why should any of this actually make NVIDIA more money?

When Does Making AI Cheaper Actually Make NVIDIA Richer?

Whenever you ask bulls this question, they immediately point to Jevons Paradox: Make a resource cheaper, people use way more of it (and find completely new uses for it), and thus the total spending goes up around the new industries that spawned out of this new prosperity.

I think that before we really get into the whole AI buildout discussion, it’s important to study a bit of history and see why infrastructure operators bet the farm on that dynamic. When we study how heavy infrastructure (think railroads, fiber-optic networks, and hopefully AI data centers) have made money in the past, a surprisingly consistent pattern emerges —

In the beginning your capital heavy investment (say railway system) is used by a few people. This means that every individual person has to also carry the cost of paying for the RnD, building, operations etc. To keep trains profitable, you might have to pay 100 USD on a 99 USD cost, giving the railway line a measly 1% margin. At this cost, people would rather Uber or drive.

Now imagine that the government drops the cost by 10x to promote more train usage. This isn’t the financial suicide it seems like on the surface:

• First, fixed-cost amortization across elastic volume: Because tickets are 10x cheaper, people use the rails for everyday trips they never took at $100. So instead of growing 10x, your volume might grow by 15x (pushing top-line revenue from $100 to $150) while also spreading your fixed infrastructure (railway lines, ticketing systems, train cars etc, all of which would be very expensive one time payments) and customer-acquisition costs across 15x more rides.
• Second, scale-driven systems innovation: Running at 15x volume exposes every inefficiency in your operations and forces you to innovate on the underlying plumbing (the exact role of LatentMoE, NVFP4, and Dynamo; this research would not have been worth it in a world that wasn’t processing hundreds of trillions of tokens regularly).

Between high utilization and engineering optimizations, your fully loaded cost per unit craters from $99/head down to $7/head. This means that your profits go up 45x (15x usage spike * 3x profit per head), even though your price 10x.

Tie this back to Nvidia and the parrallel becomes clear: crater the cost of intelligence (optimizations), ensure that the intelligence flows through your system (the CUDA ecosystem; in our rail example, worth noting that it’s not the train track maker that profits from Jevons, it’s the railways ), and you will ensure a long lasting dynasty.

So is that all? Not quite. To assume that this play will work out for Nvidia, 3 things have to go right: customers will have to spend more total dollars on AI, this spending will require net-new computing hardware, and that NVIDIA will be the one cashing the hardware checks.

Depending on which of those three links holds or breaks, we get three very different futures (all of which can play out simultaneously across different parts of the market):

1. NVIDIA creates new demand and captures the value: Cheaper inference unlocks high-value applications, customers buy more compute, and NVIDIA keeps its crown as the default infrastructure supplier of an expanded empire.
2. NVIDIA creates the demand, but someone else captures the margin: AI usage explodes, but custom hyperscaler ASICs, alternative accelerators, and cloud landlords pocket a growing share of the industry’s profit pool.
3. NVIDIA creates demand, but efficiency compounds even faster: The increased demand isn’t enough to offset the falling costs, essentially dropping profit margins.

For the rest of this article, we will study the three outcomes which ones might play out (in what contexts since all 3 can happen at once in different degrees and different subspaces) to help us identify where the market is headed and how we can ahead of.

Outcome 1: NVIDIA Creates New Demand and Captures the Value

One of the biggest mistakes analysts make when modeling cheaper inference is assuming all AI workloads have the same price elasticity. In reality, enterprise AI splits into two completely different economic buckets:

• Capped-value workloads (like a customer-support chatbot): Once a bot is smart enough to resolve routine refund tickets, making the model twice as smart doesn’t justify spending twice as much. When inference gets 50% cheaper, the enterprise simply pockets the savings and moves on.
• Uncapped-value workloads (like an autonomous coding agent): Give a coding agent 10x more test-time compute to inspect repository dependencies, draft candidate patches, run unit tests, and verify its own pull requests, and it starts solving engineering tickets it previously failed on. Here, extra tokens aren’t an overhead cost; they directly manufacture higher-value labor.

Given the conversations around Swarms breaking into systems, I think we might even see a future subgroup of uncapped workloads, which essentially operate on the insurance model — did you really invest resources into checking for vulnerabilities in your base, with certification agencies for the security measures conducted (so even if you get hacked, insurance would cover you). The reason I’m calling this out differently is because unlike standard uncapped value workloads (which are economically beneficial), these are more insurance/downside protection coded ( can you really afford to get hacked and possibly lose billions from reputational loss; and if you do get hacked wouldn’t you want to be able to say that you did what was industry best so you’re not completely ruined by bad PR). In a way, this kind of growth would be the AI industry monetizing a problem it helped create (although I’m sure some will argue that the problem was the human written bugs from older times).

We don’t have to guess whether these uncapped workloads are driving real hardware demand; Anthropic’s November S-1 amendment gives us the receipts:

• In the first nine months of 2026, enterprise demand for coding and agentic workflows pushed Anthropic’s revenue up 13x to 33.24 billion USD (driven by 11,900+ enterprises spending over 100K USD a year, including 250 whales spending over 10M USD).
• To feed that usage, Anthropic had to grow its compute bill 4.3x to 18.81 billion USD. In fact, demand outran their contracted clusters so badly in Q3 that Anthropic had to drop 5 billion USD on overpriced, short-term spot compute just to keep the lights on (temporarily swinging them from a 974M USD Q2 operating profit to a 567M USD Q3 loss).

At one level, Ant eating that 5 Billion Penalty to get energized racks online ASAP is a HUGE positive signal. However, there is a less flattering alternative interpretation here. Because Anthropic’s revenue grew 13x while its compute bill only grew 4.3x, the revenue Anthropic extracts from every dollar of compute jumped from 0.58 USD to 1.77 USD (even with training and R&D baked in). When NVIDIA’s engineering makes clusters more productive during a hardware shortage, model providers and clouds don’t hand 100% of that efficiency back to NVIDIA in new chip orders — they keep prices steady and pocket a big slice of the efficiency delta as their own margin expansion.

Now here, one might argue that Nvidia’s ecosystem that lets it engineer efficiency at scale will create a premium for Nvidia products in the short term, similar to their existing premium in Cuda, but I think the overall trend is toward customers pocketing efficiency gains, simply b/ of the value chain involved and how far back Nvidia has to sit:

Useful AI work → Customer spending → Net-new infrastructure required → NVIDIA’s share of hardware orders → NVIDIA’s gross profit

Once Anthropic and its 11,900 enterprise accounts (who generate 74% of its revenue) stare at an 18.81 billion USD annual compute bill, their number one financial priority becomes breaking that fourth link in the chain — which brings us directly to Outcome 2.

Outcome 2: NVIDIA Creates the Market, but Someone Else Gets the Money

Every billion dollars a hyperscaler or frontier lab sends to Santa Clara gives them another billion reasons to build an escape hatch ( even Anthropic, which has historically only been laser focused on models and crying wolf to not have to compete against open models, explictly laid out a 4-hardware plan across NVIDIA GPUs, AWS Trainium, Google TPUs, and AMD accelerators to maximize their intelligence per dollar).

Unfortunately, this investment in cost reduction, might not trickle down to the customers. When it comes down to it, if a model/agent company can drop their costs of intelligence 10x, what stops them from keeping the margins for themselves. Especially in a world where competitors will not undercut them (compute is constrained, so there is an upper limit to the amount of demand you can capture there). Too see this phenomenon in action, let’s take a look at some of the research done by the Gs at SemiAnalysis and their breakdown of the cost of Google’s TPU v7 —

Developing the chip internally costs Google roughly 1.03 USD per chip-hour (or 1.21 USD on an external ownership basis), which Google then rents out on Google Cloud for ~2.00 USD per chip-hour on a three-year commitment. Google can essentially price just below the GPU rental cost, drive demand to themselves, and pocket the margins for themselves.

So why hasn’t everyone dumped NVIDIA for cheaper ASICs already? This is where Nvidia’s software ecosystem becomes clutch. Thanks to the ubiquity of CUDA and Nvidia GPUs, a lot of effort has been invested in improving the utilization of Nvidia’s hardware (both by internal and external folk), which lets them overcome the raw hardware differences. This is shown very clearly in the October SemiAnalysis InferenceX benchmarks on Qwen3.5 (at 200 tok/s/user):

• On a pure hardware basis (both chips running standard FP8): Google’s cheaper TPU v7 beats NVIDIA’s GB300 NVL72 by 1.85x in tokens per dollar.
• With NVIDIA’s full-stack optimizations turned on (NVFP4 + Dynamo + TensorRT-LLM): Because TPU v7 lacks native four-bit arithmetic, cutting the byte payload in half lets GB300 flip the scoreboard and beat TPU v7 by 1.84x in tokens per dollar (assuming your workload tolerates any unmeasured quality gap between FP4 and FP8; this is also why we spent so much time understanding the engineering efforts Nvidia is taking to keep intelligence high at much lower costs).

This inversion when software comes into play means that NVIDIA it’s relatively protected from costly wars of attrition, since it wins on quality, not cheaper bare metal. This has let them keep the margins for now, even as the ASIC boys have spawned off a whole zoo of competitors, and why I think that too much money is invested in chip designs and not enough in reshaping the ecosystem around it.

Back to Nvidia, this inversion and software moat does have it’s own kinds of risks however. Generally speaking software is easier to produce than hardware (especially as agents produce more and more of it), and unlike previous times, heavy weights have a massive incentive to attack the said moats. This means that Nvidia is in a perpetual arms race to keep their software ahead of the curve. We’re already seeing this in real time as Google’s next inference chip, TPU 8i, adds native FP4 support, erasing that specific advantage, which forces NVIDIA to invent the next system primitive (like Rubin + Groq LPX disaggregation) just to protect its pricing power.

Assuming worst case here, what happens if hyperscalers eventually close that systems gap? This leaves Nvidia with 3 options:

• Taxing competitor racks (NVLink Fusion): By letting third-party chipmakers plug into NVIDIA’s rack fabric (cc it’s deal with Rebellions into NVLink Fusion for SK Telecom )NVIDIA can still sell switches and NICs when it loses the GPU socket. This will unfortunately be a pure volume play since networking plumbing doesn’t replace 75% flagship GPU margins. They will also lose some major customers since Google and AWS use their own proprietary fabrics anyway.
• Charging for enterprise software: NVIDIA AI Enterprise lists at 4,500 USD per GPU per year (plus commercial NIM licensing). There is a huge market to address with enterprise software, and Nvidia’s brands will make it a preferred provider compared to the many other software grunts and new gen Palantir’s teeming in the market. However, to replace just 40 billion USD of annual hardware gross profit at an illustrative 90% software margin, NVIDIA would need nearly 10 million GPUs paying full list price (or ~20 million GPUs at a 50% paid attach rate before discounts).
• Selling turnkey Sovereign AI factories: National buildouts in Japan and Korea love NVIDIA’s integrated racks and free open-weight models b/c governments lack internal ASIC teams. This can be a huge long term play as sovereign countries will want their ai to be protected in their own controls (so that they aren’t reliant on another countries policies; look at what happened when Fable was banned in other countries), but this will be a much longer buildout, and there is a chance that the other countries will eventually want to move their supply chain dependency off Nvidia onto local manufacturers.

Putting all this together tells us that whether Nvidia capturing margins, instead of becoming one of many, ultimately boils down to whether they can keep their full stack efficiency higher than everyone else. I parallel it almost to higher education — in an ecosystem teeming with colleges and online certifications, a Harvard/Stanford degree becomes more valuable since it acts as a signal (regardless whether it works or not). Similarly, as long as Nvidia keeps a strong full stack lead, I think the fragmentation in the AI hardware market will cause decision paralysis and make companies instinctively reach for Nvidia (even if there is a premium).

With all this discussed, let’s look at the weirdest risk of all: what happens if NVIDIA’s engineers are too good at their jobs?

Outcome 3: NVIDIA Creates Demand, but Efficiency Improves Even Faster

When a cloud provider installs Dynamo, switches to NVFP4, and makes its existing GPU cluster 30% more productive in the middle of a compute shortage, everybody celebrates. Waiting customers immediately soak up the extra capacity, and the cloud provider still buys every new GPU it can find.

Run that exact same 30% software upgrade when the cluster already has enough capacity to serve paying customers, however, and the math flips. Instead of buying more racks, the cloud provider just delays its next hardware order.

If effective compute supply starts growing faster than paying demand, NVIDIA runs into a competitor that is much harder to beat than AMD or Google: the millions of GPUs NVIDIA already sold two years ago (a brand-new Rubin rack has to generate enough revenue to pay back tens of thousands of dollars in fresh capital expenditure, while a depreciated H100 sitting in an already-wired rack only has to cover electricity, cooling, and basic maintenance to stay cash-flow positive +every time NVIDIA ships a free software update that speeds up inference, that paid-off H100 gets more productive for free).

We haven’t hit that saturation point yet. Silicon Data’s H100 rental index tells us that even the ancient H100s rent at around 3–3.1 USD per GPU-hour. As older fleets pay off their upfront capex, the productivity hurdle NVIDIA’s newest chips have to clear just to justify rip-and-replace upgrades gets steeper every year.

There’s another overlooked wrinkle that makes this analysis messy: software moves on internet time, while the electrical grid moves on utility time (this is also why so many neoclouds are hitting credit crunches, since their investors anchored their expectations to software lifecyccles (they were tech investors), and completely overlooked that the short-medium term constraints would all be blocked by physical problems that the tech industry had never dealt with, due to which growth hasn’t been as smooth as anticipated).

This means that while you can drop a 2x model optimization overnight, you can’t summon high-voltage transformers or substation hookups on command (the IEA estimates grid bottlenecks could delay roughly 20% of planned data-center capacity through 2030). That physical bottleneck pulls operators in two completely opposite directions depending on their balance sheet:

• If you already own an energized building with a fixed megawatt cap: Power efficiency is king. Swapping older GPUs for Rubin racks that squeeze 10x more tokens out of the same electrical allocation lets you serve way more customers without waiting three years for the utility company — assuming you have paying customers lined up for those extra tokens.
• If you are borrowing billions to build new sites from scratch: NVIDIA’s frantic annual hardware cycle and power delays put you in a brutal vise. Look at CoreWeave’s Q2 2026 numbers: the company brought in 2.575 billion USD in quarterly revenue, paid 640 million USD in net interest, and watched 1.393 billion USD (54% of its entire top-line revenue) vanish straight into depreciation and amortization — all while waiting on 748MW of undelivered power across two leased site groups phasing in through 2028.

Pause on that 54% depreciation number for a second. When more than half of your revenue goes toward hardware losing value on the shelf, timing can kill you. Buy GPUs too early before your power is hooked up, and NVIDIA launches a 2x faster architecture before your chips earn their first dollar; wait until the substation is live, and you risk hardware shortages or blown customer deadlines.

This is where NVIDIA’s incentives clash directly with its infrastructure partners: NVIDIA wants to make its own hardware obsolete every 12 months to force new purchases, while its debt-financed cloud customers desperately need older generations to stay economically relevant long enough to pay back their lenders (this is also why Coreweave and co have very generous depreciation schedules).

In 2027, that timing gamble will reach a boiling point. SemiAnalysis forecasts a massive wave of delayed data-center shells finally coming online in 2027 —

If empty buildings suddenly outpace chip supply — especially with the IEA’s November 2026 analysis projecting high-bandwidth memory (HBM) shortages through at least the end of 2027 — the bottleneck swings from power right back to silicon. NVIDIA has already positioned its balance sheet for that exact flip, locking in roughly 279 billion USD in supply and manufacturing commitments (per its latest 10-Q) so it controls the memory and packaging when customers finally have rooms ready to fill.

But what happens if all those new 2027 buildings come online at the exact same time as 10x hardware jumps, 3.5x LatentMoE gains, and 4x Dynamo scheduling speedups? Effective computing capacity could multiply far faster than raw megawatts or GPU shipment counts suggest — even while total token usage keeps climbing.

And if effective supply overshoots paying demand, NVIDIA can’t simply tee hee and walk away, because it has been using its own balance sheet to finance the buildout. In August, NVIDIA agreed to provide up to 105 billion USD in contingent guarantees (disclosed in its 10-Q) backing land, power, and building shells for OpenAI-linked infrastructure in Ohio, phasing in starting fiscal 2029.

On one hand, guaranteeing 20-year power shells is a smart strategic play:

• Securing the Ohio campus locks in sockets for several future generations of NVIDIA racks.
• Funding independent neoclouds expands NVIDIAs customer base so it isn’t held hostage by three hyperscalers.

But at the same time, it’s undeniable that those neoclouds are currently pass-through landlords leasing compute back to the exact same handful of frontier labs (like Anthropic, which carries roughly 895 billion USD in multi-year compute commitments across hardware suppliers, with 69.8 billion USD due in the next 12 months). So Nvidia is still left to deal with the reality that while they can help finance their customers’ infrastructure to speed up adoption, this will only work until their end-user revenue fails to cover the rent. This is another reason why Nvidia is so frantic about market expansion, since the Neoclouds it’s investing into can only survive if they see a lot of customer growth outside of the valley.

Where Does All of This Leave Us?

The biggest takeaway from studying NVIDIA’s engineering alongside its balance sheet is that “AI demand” and “GPU demand” are no longer the same line on a chart. Every optimization we covered in Part 1 drives a wedge between the two, meaning AI usage can 10x from here while hardware profits go through a massive regime shift.

If I had to bet on how our three outcomes play out, they won’t happen as separate timelines — they’ll split the market in two:

• Routine, capped-value AI (basic enterprise workflows, standard RAG, customer support) will slide rapidly into Outcomes 2 and 3. Once a workload stabilizes, hyperscaler ASICs and paid-off H100s are more than good enough to run it, squeezing NVIDIA out of the commodity tier.
• Frontier, uncapped-value AI (autonomous coding, continuous cyber-defense swarms, physical robotics) will stay locked in Outcome 1, because those workloads are so compute-starved that they need NVIDIA’s bleeding-edge system tricks just to be economically viable.

In other words, NVIDIA is quietly trading away the commodity floor of the AI market to keep a monopoly on the ceiling. Whether that trade makes NVIDIA richer over the next 24 months comes down to three signals I’ll be tracking:

1. Enterprise cohort graduation: Are 100K USD customer accounts graduating into the 1M+ USD tier fast enough to pay off the industry’s nearly 1 trillion USD in forward compute commitments?
2. Real-world ASIC software parity: How quickly do chips like TPU 8i close the production TCO gap once NVIDIA’s software lead is factored in?
3. Used H100 rental rates in 2027: When the delayed wave of 2027 data-center power finally flips on, spot rental prices on depreciated GPUs will be the first canary in the coal mine telling us if software efficiency has finally outrun paying demand.

Fun, fun times ahead.

Thank you for being here, and I hope you have a wonderful day,

Dev ❤

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction.

Help me buy chocolate milk

PS- We follow a “pay what you can” model, which allows you to support within your means. Check out this post for more details and to find a plan that works for you.

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

This article was originally published here. Sign up for free to never miss out and join 200K weekly readers getting the most important insights in AI delivered to their inbox.

Databricks paid $1.3 billion for MosaicML. Harvey raised $550 million at a $15.5 billion valuation. Gartner expects the specialized AI model market to reach $4.9 billion in 2026.

Several multi-billion dollar Silicon Valley backed companies are trying to sell you a similar philosophy: Your company has unique data. General AI doesn’t know that data. If you spend a fortune training on your own data, you can get much better returns compared to generic models.

Once you get into this cope, every road leads to becoming a training-cel. Have more data? Train more. Your information changes? Train again. The model still makes mistakes? Clearly, it needs more training. There is a loss function Shangri La after which all your business problems a prompt away from completion, and you should give joyfully give your AI Cloud providers all your operating margins to reach this happy ending.

However, there is a slight issue here… once you start getting into the details, a growing body of research suggests that training language models on proprietary knowledge is probably the most inefficient way to make that knowledge useful.

Now here you might stop and ask me about the impressive benchmark results. And how it makes intuitive sense that a model trained on the specific data/workflows will outperform general models. This is where we’re going to split training into two different outcomes —

1. Training to teach a model new facts is very expensive and mostly doesn’t work. We will go deep into the science of learning by exploring what pre-training, post-training, and continual learning can (and can’t) teach a model — and how much each approach will cost you.
2. Training to teach a model workflows (for eg how to navigate a filesystem to perform a certain task) is more grounded in science, but still doesn’t have the ROI that AI companies pretend it does.

Most importantly, I will show you with evidence how so many of the companies peddling this nonsense skip over the fact that many of the impressive improvements attributed to training can also come from building better reasoning, retrieval, memory and data infrastructure surrounding the model with more reliability and safety and at a much lower cost.

Now these are big claims, and I’m challenging some industry “best” practices (best for the investors and companies, for sure). I also understand that the people who peddle the post-training myth often struggle with reading the soo many big words that are found in research papers (which is why they are still peddling beliefs from the BERT era of LLMs). So in the spirit of good science and true intellectual inclusivity (literacy should not be a barrier to public participation), I am issuing a simple open challenge to anyone claiming the superiority of specialized training and can’t back it up by pointing to third party research (which doesn’t exist):

1. Use a real baseline. Compare your trained system against a capable model with proper reasoning, retrieval, memory, and tools. Beating a naked base model proves very little.
2. Show us your harness and evidence. Publish enough of your architecture, evaluations, and results to establish that training — not a weaker baseline — is responsible for the improvement.
3. Prove the economics. Compare the full cost of training against investing the same resources in better inference infrastructure. Include engineers, failed experiments, inference, refreshes, and vendor fees. Which investment delivers more?
4. Beat our general-purpose reasoning engine. We’ll compare your domain-trained system against Irys’s domain-agnostic reasoning infrastructure (you can sign up for a free Irys account in minutes, so there’s no setup complexity there; API also available) on identical inputs, with the same evaluation and transparent costs. Let’s see us what your specialization actually adds.

Join me as we do a Ateba Gautier level fraud-check on one of the most hyped trends in AI right now.

Executive Highlights (TL;DR)

• Pre-training is where models learn most of their factual knowledge. However this is extremely expensive, which is why most vendors use the language of pretraining (“train on your own data”, “models that remember your facts”) but in reality try to sell post training.
• However, post-training is much better at teaching behaviors than reliably injecting new facts. So the mechanism they’re trying to sell doesn’t actually meet do what their marketing departments promise. While not the focus of this article, this is also why companies like Hippocratic AI, which imply that they have special ways to train domain specialized LLMs, are misleading customers. There just is no way to reliably inject knowledge in the weights of already trained LLMs.
• This is clear when we look at the benchmark marketing. Vendors show impressive gains from training, but these amazing numbrs are often compared against weak baselines or unreproducable reasoning infrastructure. The latter is especially important when you consider that multiple bits of research have shown us that you can get really impressive results by improving the harness, w/ no investments in model training required.
• This means that for most people investing in your own training pipeline (even the relatively cheaper post training) doesn’t manifest much ROI. This is especially true for systems that draft a lot (environments where files are changed constantly, tool specs are rebuilt etc). This drift is increasingly common with agentic systems (which tend to change a lot of things), hence we have seen the rise of continual learning. However…
• Continual learning doesn’t magically solve the problem. Your knowledge, software, and workflows keep changing. Training-dependent improvements need ongoing updates and testing. Better retrieval, memory, and reasoning infrastructure can instead be reused across successive generations of models.
• All in all, most of the conversation around training your models is smokes and mirrors that is meant for headlines but has almost no merit in actual research and products.

Hey there. I keep this newsletter ad-free, w/o sponsors, and mostly w/o paywalls to ensure that everyone has access to high-quality deep-dives on the most important ideas in the AI Industry without any bias. B/w my research, independent verification, and the chocolate milk needed to produce these works, these deep-dives take a lot of time, resources, and infrastructure. For just 10 USD/month or 100 USD/year, you can help me continue my vision of democratising access to the most important ideas in AI by becoming a paid subscriber over here —

Help me buy chocolate milk

(If that is too much for you, we follow a pay-what-you-can plan over here, and it goes down all the way to 1 USD/month or 10 USD/year. No matter what plan you pick, you will get all of our articles without paywalls and have access to all of our articles. Check it out here).

PS — Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

1. Pre-training: Where AI actually learns its knowledge

We’ll keep this section short, because almost nobody seriously recommends that enterprises train their own foundation models from scratch. But ironically enough, most people are ignorant of the fact that if your goal is to teach a model new facts and associations, pre-training is where most of that learning actually happens. This was shown quite conclusively studying Gekhman et al. (EMNLP 2024) reached an even more interesting conclusion:

“large language models mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently.”

(In case you’re skeptical of forming your worldview on the basis of one research paper, I’ve complied a list of papers that will help you really understand the knwoledge acquisition process in language models —)

Unfortunately, training a foundation model from scratch requires enormous amounts of data, computing power, and specialized expertise. This is why new pretrains are relatively rare, and most model releases tend to work on taking the knowledge base and utilizing it better through post training. To their credit, even Harvey and the other “custom train” data companies are not stupid enough to tell you that you need to post train the data.

This leads us to what they do sell you. In a nutshell, their proposition can be summarized as following

• Take an existing model
• Train it on your company’s data (either directly on the documents/procedures so it “learns your company data” or indirectly by learning to mimic your workflows/the right behaviors.
• Tehee.
• ????
• Endless Money Machine, no more expensive employees.

Sounds much cheaper. But does it actually work? Pitch 1 (train on data) does not (cue the reasearch list mentioned earlier about why pretraining is the only real place to inject knowledge, or refer to the image below for a more distilled understanding of why post training/later stages are hostile to knowledge acquisition).

What about the second one? Can you leverage the fact that post training can be used to teach models behaviors to best teach an AI system how to work within your corporate environment? Technically yes, but let’s see why this approach still makes no sense.

FineTuneBench, a Stanford study examining commercial fine-tuning services in 2024, found that models achieved just 37% average generalization accuracy when learning new facts, and 19% when updating existing knowledge. In other words, even when models learned their training examples, they struggled to apply that information to questions they hadn’t seen before. Another study, Ovadia et al. (EMNLP 2024), directly compared teaching models factual knowledge through unsupervised fine-tuning versus giving them access to the relevant information through retrieval. Retrieval consistently won in their experiments.

2. Why Post-training (very) is Overrated

Training models to follow specialized workflows can produce impressive results. The problem is that theres a huge difference b/w saying something can boost your performance and that something is the best (or even a good/economical) way to spend your resources. Let’s take Engram’s collaboration with Harvey. They trained an agent on 108 million tokens of synthetic law-firm data, improving closed-book factual accuracy from 4.7% to 72.6%. Their systems also hit inference costs of $0.13 per query versus $1.32 for Claude Opus 4.8.

Very cool, and it makes sense why this announcement was paraded heavily on social media. But let’s start digging into the details a little bit more. Firstly, we must note that their complete system also combined structured notes, search, and agentic execution. This immediately leads to the question — how much of the useful performance came from training, and how much came from the system surrounding the model? What did the training cost? Could a better reasoning system have achieved comparable results without it?

This is a recurring problem with the vendor benchmarks used to justify their specialized training: Companies either compare their trained models against relatively bare base models, or publish results using proprietary harnesses that we can’t properly inspect. There’s no way for us to know whether the harnesses where the post trained models follow good engineering practices, or if it’s vibe coded slop.

You might think I’m being nitpicky here, but comparing against multiple simple reproducible baselines is the very basis of scientific research. Without them, the publications parading around all the amazing results are just research cosplay, a kind of 0 nutrition junk food science science.

Let’s go one layer deeper than the lack of baselines. When we contextualize the training discussions within the larger AI ecosystem, one thing to note is that we consistently see no additional model training:

• Harvey improved average criteria pass rates from 23.3% to 62.4% across seven models simply by replacing its standard harness with a recursive reasoning architecture.
• OpenAI improved ARC-AGI-3 scores from 13.3% to 38.3% by changing two inference settings, while reducing output tokens sixfold.
• NVIDIA improved agent correctness scores from 46 to 87 by adding verified skills, without retraining the underlying models.
• Our own general-purpose reasoning infrastructure (stateful swarms) got really good results on the public Harvey LAB and the DELTA benchmarks without domain-specific training using an out of the box Gemini 3.7 Flash (non thinking mode), simply by using the base model much better.
• Similarly, our fractal embeddings research demonstrated how improving the structure of information representations can improve performance without training proprietary knowledge into a foundation model. When it was independenlty implemented by Rob Currie, former CTO of the UCSC Genomics Institute, he found that it “works well improving accuracy over the original embeddings while learning a hierarchical structure based on biology.” The system even outperformed the Genentech baseline, even though it wasn’t designed or tested on the domain when we released it(Robs replication can be found here).

All that to say, there is a strong, and constantly growing field of research that clearly shows us that we can hit enormous improvements by making better use of models we already have. Why are we spending millions retraining them before exhausting these alternatives?

But fuck it, let’s for the sake of argument assume that your infrastructure is perfect. You’ve optimized your retrieval, memory, reasoning, and agentic execution. There’s absolutely nothing left to improve. You now want to squeeze out the remaining performance by training your own specialist model.

I don’t think you realize how much that costs, so let me show in a apinful amount of detail. Harvey Tenet was developed using approximately 150 NVIDIA B300 GPUs over two months. Renting that amount of compute continuously at roughly $7 per GPU-hour comes to about $1.5 million. Let’s take that as our baseline.

To train your new model, you’ll need a few people. Let’s build a pretty lean engineering team: two AI researchers at $250K each, two data engineers at $200K each, a model-training engineer at $240K, and an MLOps engineer at $220K. That’s $1.36 million in annual personnel costs. Add another $120K for evaluations, security, and supporting operations.

Assuming that you have a cracked team that needs only three months to 0–1 the whole thing, you’re looking at a bill of $370K on top of the training compute.

This brings your number to $1.87 million. To stay here, you’re making a few giant assumptions —

• You’ve one shot hired the perfect team and not had to pay any costs in recruitment (standard industry fees are 20–30% that we’re cutting out; + the time required to hire but double your actual time requirements).
• Every experiment works so you can get the perfect design without failures.
• You’re completely overlooking the opportunity costs of putting all this effort and resources.

To amp up your savings, let’s say you’re willing to fire everyone in your perfect team after model deployment and keep just your training and MLOps engineers, with $60K annually for evaluations and monitoring. That’s another $520K a year.

So w/ these assumptions, how efficient is this decision? Assuming that your firm is filled with a 1000 power users making 50 queries daily and if your newly trained system is 0.50 cents cheaper/query, that gets you $25,000 in daily savings.

While that sounds impressive, that means you’ll need roughly 80 days after deployment to break even. Add the three months spent developing the system, and you’re approaching six months before recovering your investment.

And remember, this is a fantasy scenario. You have 1,000 active users from day one, making 50 queries every single day, including weekends. Every team-member is hyper-competent (but somehow still not accpeting millions from foundation model labs or through their own startup), no experiments fail, and all the other operational wiggles are completely non-existent.

You STILL need ~6 months before your system produces ROI. Want to guess how often a new frontier (or close to frontier but more efficient) model is dropped?

It’s every 5 days. Best of luck of staying ahead of the most talent dense and well-funded ecosystem in tech across that many iterations.

You might think that this is the end of your woes, but life is rarely that kind to ones it chooses to beat down. Even if your newly post trained is somehow still coming out on top, that’s not going to last very long thanks to a nagging lil phenomenon in software called environment draft. Software changes. APIs change. Foundation models improve. And as you deploy more agents, those agents begin changing the workflows and software environments they’re operating within. So if your performance comes from training models to navigate a particular environment, you’re learning behaviors tied to a moving target. New tools, changed interfaces, and different workflows can force you to revisit that training.

Which means you start again. And again. And again.

And again.

And again.

Until you and Totenham Fans are the only people left at the heat death of the universe, both waiting to reach the promised land that never came

Now compare that against spending the same money improving your retrieval, memory, and execution infrastructure. Better infrastructure can compound across generations of foundation models. When a stronger model arrives, you can plug it into the existing system and immediately benefit from the engineering you’ve already paid for. You maintain that infrastructure, of course, but you aren’t rebuilding specialized model weights every time the underlying technology advances.

To reiterate, training can absolutely improve performance. But that isn’t the question that people actually care about. The question is whether that improvement justifies the money, maintenance, and opportunity cost. And that answer is almost always a no, especially in the many non-tech domains where the solution is sold. Even Cursor, which has some major structural advantages —

• Operates in Coding, a more verifiable domain much closer to an AI researchers native domain expertise
• Has a lot of money
• Has a LOT of usage and traction.
• Doesn’t have the AGI mandate and can focus on coding

still failed to push Composer to any meaningful difference compared to general models. So what secret do the other players in the space have?

This is a realization that most of the companies selling post training are slowly realizing. Which is why you’re seeing some goalposts being shifted, new terms being sold, and a whole new savior that will finally make domain specialized LLMs a reality.

Enter continous learning.

(Before I tear it apart, I want to differentiate between two things. As a researcher, I think CL is very interesting and has the potential to do some great things eventually. But for this article, our goal is to evaluate whether it’s ready to be sold in the short term, and that is a HELL NO).

3. Continual Learning: The Next Promised Land

The obvious problem with training models on company knowledge is that this knowledge keeps changing. Contracts get amended, regulations change, and workflows evolve. Constatly retraining models in the outdated ways is not sustainable. This is the impetus behind continual learning (CL): the field that studies how to keep updating the model with new information without destroying what it previously learned.

We would be very gung-ho about this field, except that updating knowledge is considerably harder than it sounds. A 2025 paper, The Mirage of Model Editing, found that knowledge-editing techniques reporting 96.8% success achieved just 38.5% under more realistic evaluations. Some earlier benchmarks had effectively leaked parts of the correct answers to models during testing. The methods also deteriorated dramatically after just 1,000 sequential edits.

And that’s before dealing with how corporate information actually works. Change a contract’s payment terms, and you’ve potentially changed its obligations, financial exposure, and everything that depends on those terms. Getting the model to remember the new number doesn’t mean it’ll update all the conclusions built on the old one.

A 2026 ACL study examined exactly this problem with evolving real-world knowledge. Continual fine-tuning, knowledge editing, and ordinary RAG all struggled with temporal consistency. The researchers improved results by organizing external evidence into a time-aware knowledge graph.

Notice what’s not needed? More training.

For the sake of argument, let’s say your vendor has the secret of continual learning (again a huge leap). Is there any utility to engaging with it then?

Each major refresh can mean preparing new training data, running experiments, evaluating what the model learned, checking what it forgot, and deploying another version. And you’re still paying the engineering team responsible for all of this. Let’s assume refreshing the model costs just 10% of the original $1.5 million GPU bill (this is really low if you look at the actual vendor claims, but we’re being generous here).

That’s $150,000 per update, or $1.8 million annually if you refresh monthly.

And you’re still quite susceptible to the generalization bias + environmental drift double whammy that hits normal training teams.

Compare that with maintaining knowledge externally. A contract changes? Update the record. A new regulation arrives? Update the knowledge base. Your reasoning system can access the latest information without another training cycle. Signals can be propogated more reliably (and most importantly you can detect every time they are not).

You still maintain and evaluate that infrastructure, but the investment can carry over to newer foundation models instead of remaining tied to one set of weights.

And once again, if any CL vendors have a problem with what I’m saying, let’s run a head to head publicly.

Conclusion: Don’t Train Your Own Model

There’s a rather beautiful business model hiding underneath all this. AI vendors get paid to train your models. Cloud providers get paid for the compute. Then everyone gets paid again when your data changes, your workflows evolve, or a better model comes along.

And somehow, you’re the one who’s supposed to be excited about this arrangement.

To reiterate what will undoubtely be deliberately misinterpreted about this article — I’m not saying that doesn’t work. I’m saying that for most (99%+) enterprise use cases, there are better and more efficient solutions out there. While commiting millions to your “own AI model” is a great way to get headlines and impress some board members, there’s honestly no good engineering or performance related reason to do so.

Your company’s knowledge doesn’t need to live inside a model’s weights. Your AI needs to know how to use it. That gets much easier with proper skills, documentation, and infra setup, especially as models get better with complex instruction following.

Perhaps we should try solving that problem before financing another GPU cluster.

Thank you for being here, and I hope you have a wonderful day,

Dev ❤

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction.

Help me buy chocolate milk

PS- We follow a “pay what you can” model, which allows you to support within your means. Check out this post for more details and to find a plan that works for you.

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

This article was originally published here. Sign up for free to never miss out and join 200K weekly readers getting the most important insights in AI delivered to their inbox.

“The fallacy is to believe that under a dictatorial government you can be free inside. Quite a number of people console themselves with this thought, now that totalitarianism in one form or another is visibly on the up-grade in every part of the world. Out in the street the loudspeakers bellow, the flags flutter from the rooftops, the police with their tommy-guns prowl to and fro, the face of the Leader, four feet wide, glares from every hoarding; but up in the attics the secret enemies of the régime can record their thoughts in perfect freedom — that is the idea, more or less. And many people are under the impression that this is going on now in Germany and other dictatorial countries.

Why is this idea false? I pass over the fact that modern dictatorships don’t, in fact, leave the loopholes that the old-fashioned despotisms did; and also the probable weakening of the desire for intellectual liberty owing to totalitarian methods of education. The greatest mistake is to imagine that the human being is an autonomous individual.”

—George Orwell, “As I Please”

I’ve been noticing a pattern in the United States and India, two countries that take up most of my time and attention. In the US, Flock’s network of over 120,000 cameras has sparked lawsuits and public backlash over mass surveillance and potential misuse. Here in India, the Delhi Police’s use of facial recognition to monitor protesters at Jantar Mantar has faced legal challenges of its own.

There are legitimate uses for these technologies, of course. But more often than not, new capabilities that come in with good intentions (but lacking the appropriate checks and balances) often become pillars for surveillance capitalism, institutional overreach, and totalitarian oppression. That makes it all the more important that people retain the ability to question, challenge, and resist abuses of that power. That’s all well and good, but the question is, how do we actually do that?

George Orwell is an obvious person to turn to here. He gets plenty of credit for anticipating surveillance, propaganda, and the machinery of authoritarian control. But in the same case as Harry Kane’s elite goalscoring drawing away attention from his also elite playmaking skills, I think people often overlook some of his most interesting insights; beyond explaining how institutions control people, Orwell also gives us something much more useful to think about: what makes people capable of resisting them in the first place?

To see what I mean, consider a rather odd detail at the end of 1984. The Party has already captured Winston and Julia. It knows everything about their affair, has extracted their confessions, and can imprison or execute them whenever it wants. Yet it goes to extraordinary lengths to make them genuinely betray one another, before eventually letting them go.

Why bother? What does destroying their relationship accomplish that all the Party’s surveillance, prisons, and torture couldn’t? That, my loves, is imo the real takeaway worth looking into from 1984.

Executive Highlights (TL;DR)

• Orwell’s most interesting warning goes beyond surveillance and propaganda. In 1984, the Party isn’t satisfied with controlling Winston and Julia; it needs them to betray each other. The point is to destroy loyalties and relationships that exist independently of the Party, leaving people with fewer ways to question or resist its authority.
• Our understanding of reality is surprisingly dependent on other people. In Animal Farm, Napoleon abolishes debate and rewrites history, making it difficult for the animals to compare their experiences or organize against him. Several recognize that something is wrong, but individual skepticism is of little use when people cannot establish a common understanding and act together.
• Relationships also develop the very abilities that make solidarity possible. Disagreements, compromises, rejection, and the general inconvenience of dealing with other people teach us to understand perspectives beyond our own. This matters politically because recognizing injustice is one thing; caring enough to inconvenience ourselves for somebody else is quite another.
• AI introduces an interesting complication. We can increasingly get companionship, validation, and emotional support without much of that interpersonal friction. A study published in Science found that AI models affirmed users’ actions 49% more often than humans, while overly agreeable responses made people less willing to repair conflicts. Useful technology, certainly, but what happens if we gradually become less practised at navigating realities that don’t accommodate our own?

• The political consequence is what interests me most. Surveillance becomes particularly dangerous when people lose the trust, shared reality, and willingness to stand up for one another that make resistance possible. I’m not arguing that AI companionship leads to totalitarianism. But while we’re busy worrying about institutions becoming powerful enough to watch everybody, it’s worth asking whether we’re becoming less capable of looking out for each other.

Hey there. I keep this newsletter ad-free, w/o sponsors, and mostly w/o paywalls to ensure that everyone has access to high-quality deep-dives on the most important ideas in the AI Industry without any bias. B/w my research, independent verification, and the chocolate milk needed to produce these works, these deep-dives take a lot of time, resources, and infrastructure. For just 10 USD/month or 100 USD/year, you can help me continue my vision of democratising access to the most important ideas in AI by becoming a paid subscriber over here —

Help me buy chocolate milk

(If that is too much for you, we follow a pay-what-you-can plan over here, and it goes down all the way to 1 USD/month or 10 USD/year. No matter what plan you pick, you will get all of our articles without paywalls and have access to all of our articles. Check it out here).

PS — Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

The Inconvenience of Other People

“In our world there will be no emotions except fear, rage, triumph, and selfabasement. Everything else we shall destroy — everything…We have cut the links between child and parent, and between man and man, and between man and woman. No one dares trust a wife or a child or a friend any longer. But in the future there will be no wives and no friends. Children will be taken from their mothers at birth, as one takes eggs from a hen. The sex instinct will be eradicated. Procreation will be an annual formality like the renewal of a ration card. We shall abolish the orgasm. Our neurologists are at work upon it now. There will be no loyalty, except loyalty towards the Party. There will be no love, except the love of Big Brother. There will be no laughter, except the laugh of triumph over a defeated enemy. There will be no art, no literature, no science….Always, at every moment, there will be the thrill of victory, the sensation of trampling on an enemy who is helpless. If you want a picture of the future, imagine a boot stamping on a human face — for ever.”

— George Orwell, 1984

Let’s consider two things Orwell does across his most famous books.

• 1984: The Party already has Winston and Julia completely at its mercy, yet it goes to extraordinary lengths to make them betray each other. Why bother? Their relationship represents something the Party cannot control: an independent source of loyalty, trust, and shared experience. By destroying it, the Party undermines the very relationships through which people might question its authority.
• Animal Farm: Napoleon achieves something similar by abolishing the animals’ meetings, rewriting their history, and making himself the authority through which they understand events. Several animals suspect something is wrong. Benjamin understands far more than most. But without the means to compare experiences and act together, their private understanding accomplishes remarkably little.

What interests me here is how much we rely on other people to understand the world. Friends expose us to experiences we haven’t had, colleagues help us recognize problems we thought were unique to us, and even ordinary conversations can challenge something we’ve believed for years. Through these interactions, we build a shared view of the world, a critical component for bringing together collaboration. Without that common ground, a population can contain plenty of intelligent, skeptical people who remain individually powerless.

“Napoleon, with the dogs following him, now mounted on to the raised portion of the floor where Major had previously stood to deliver his speech. He announced that from now on the Sunday-morning Meetings would come to an end. They were unnecessary, he said, and wasted time. In future all questions relating to the working of the farm would be settled by a special committee of pigs, presided over by himself. These would meet in private and afterwards communicate their decisions to the others. The animals would still assemble on Sunday mornings to salute the flag, sing Beasts of England, and receive their orders for the week; but there would be no more debates.

In spite of the shock that Snowball’s expulsion had given them, the animals were dismayed by this announcement. Several of them would have protested if they could have found the right arguments…Then the sheep broke out into a tremendous bleating of “Four legs good, two legs bad!” which went on for nearly a quarter of an hour and put an end to any chance of discussion.”

— George Orwell, Animal Farm, Chapter V

But relationships do something else that I think is equally important. The process of building them changes us.

Think about how many people you interact with before finding a handful of friends or somebody you love. You deal with disagreements, rejection, awkward conversations, and people whose priorities make absolutely no sense to you. Even the relationships that survive require work. You learn to accommodate other people’s needs, reconsider your judgments, and occasionally accept that someone else’s understanding of a situation is just as reasonable as yours. Not every interaction makes you a better person, obviously, but navigating other people’s realities is a large part of how we develop our own. And losing this will likely compromise something we haven’t adequately valued.

A study published in Science found that AI models affirmed users’ actions 49% more often than human respondents. Participants interacting with overly agreeable AI became more convinced they were right and less willing to repair interpersonal conflicts. They also preferred the agreeable responses. Even if one is careful about “sycophancy” and tries to share their situation with AI more objectively (and the AI itself is trying to be the best judge; very big assumption here), our own cognitive biases mean that we will fail to remember less flattering things and articulate the other party's views properly.

Let’s play the implications of that out. What happens when we become accustomed to environments that adapt to our understanding of the world rather than requiring us to engage with somebody else’s? Much of our ability to recognize another person’s perspective develops through these encounters, including the inconvenient ones. Removing that friction may make companionship easier while leaving us less practised at dealing with people who aren’t like us. Ultimately, that takes away a key pillar of for us developing empathy, the capacity to care for another person, that might move us to bear discomfort for the sake of the other.

Now bring this into our earlier conversation about political consequences. Most people would be appalled at watching someone being unjustly dragged off the street. However, it’s much less striking to hear that an algorithm flagged someone and thus they were denied something essential. The consequences may be equally serious, but the latter is easier to experience as a statistical failure of the system rather than something happening to another person. If our social experiences increasingly revolve around individually tailored environments, what happens to our ability to recognize the lives affected by institutions operating far outside our immediate circles? What happens to our capacity to agree that we should care for them, that we should inconvenience ourselves for the ones beyond our noses on the faith that they would do the same for us?

“A British and a German historian would disagree deeply on many things, even on fundamentals, but there would still be that body of, as it were, neutral fact on which neither would seriously challenge the other. It is just this common basis of agreement, with its implication that human beings are all one species of animal, that totalitarianism destroys. Nazi theory indeed specifically denies that such a thing as ‘the truth’ exists. There is, for instance, no such thing as ‘science’. There is only ‘German science’, ‘Jewish science’ etc. The implied objective of this line of thought is a nightmare world in which the Leader, or some ruling clique, controls not only the future but the past. If the Leader says of such and such an event, ‘It never happened’ — well, it never happened. If he says that two and two are five — well, two and two are five. This prospect frightens me much more than bombs — and after our experiences of the last few years that is not a frivolous statement.”

— George Orwell, Looking Back on the Spanish War

Now it’s not my job (or interest) to shame you for relying on your very understanding AI Boyfriend to get the emotional support you need. Nor will I say that this is the path to you becoming a drone of totalitarianism, a worker grunt of the oligarchs. But I do think it is worth thinking about how the act might be stopping you from engaging with the world on your own terms, ultimately limiting your exposure to it. Because w/o, fighting for your freedom becomes an uphill battle.

What Makes Resistance Possible?

“If there was hope, it must lie in the proles, because only there in those swarming disregarded masses, 85 per cent of the population of Oceania, could the force to destroy the Party ever be generated. The Party could not be overthrown from within. Its enemies, if it had any enemies, had no way of coming together or even of identifying one another. Even if the legendary Brotherhood existed, as just possibly it might, it was inconceivable that its members could ever assemble in larger numbers than twos and threes. Rebellion meant a look in the eyes, an inflexion of the voice, at the most, an occasional whispered word. But the proles, if only they could somehow become conscious of their own strength. would have no need to conspire. They needed only to rise up and shake themselves like a horse shaking off flies. If they chose they could blow the Party to pieces tomorrow morning. Surely sooner or later it must occur to them to do it? And yet — !”

— George Orwell, 1984, Part One, Chapter VII

Coming back to the surveillance systems we discussed earlier, I think this is what makes Orwell particularly interesting. The problem isn’t simply whether a government can identify protesters or monitor its citizens. It’s also whether those citizens can recognize abuses, establish what happened, and find enough common ground to challenge them.

That requires more than information. Someone needs to care when a stranger is mistreated, believe their account, and sometimes accept personal inconvenience or risk to help them. Courts, journalists, civil liberties organizations, and ordinary communities all depend to some extent on people being willing and able to do this.

And that’s why I’m particularly interested in what we’re doing with our social lives. We can increasingly obtain companionship, validation, and personalized experiences without having to navigate much of the difficulty of understanding other people. Some of these technologies are genuinely helpful, but we’re not entirely sure what disappears when we remove the experiences that traditionally helped us develop those abilities.

“Actually, however, the avoidance of reality is much the same everywhere, and has much the same consequences. The Russian people were taught for years that they were better off than everybody else, and propaganda posters showed Russian families sitting down to abundant meal while the proletariat of other countries starved in the gutter. Meanwhile the workers in the western countries were so much better off than those of the U.S.S.R. that non-contact between Soviet citizens and outsiders had to be a guiding principle of policy…

To see what is in front of one’s nose needs a constant struggle.”

— George Orwell, In Front of Your Nose, 1946

I don’t think this means we’re heading toward 1984, nor that the inconvenience of other people is always worth preserving. But I do wonder whether our growing ability to make life comfortable within our own little worlds might leave us less equipped to understand the world outside them.

After all, much of what allows us to recognize injustice, particularly when it’s happening to somebody else, comes from the relationships and shared experiences through which we’ve learned to understand people beyond ourselves. And if those begin to disappear, I wonder how effectively we’ll be able to stand up for one another when it matters.

As a closing thought — we spend so much time discussing what happens when institutions become powerful enough to watch everybody. Orwell also makes me wonder what happens when the people being watched become less capable of trusting, understanding, or standing beside one another.

Thank you for being here, and I hope you have a wonderful day,

Dev ❤

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction.

Help me buy chocolate milk

PS- We follow a “pay what you can” model, which allows you to support within your means. Check out this post for more details and to find a plan that works for you.

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

For single-GPU serving of 70B-class models, long context windows, or memory-bound inference, the H200 is the right call. For short-context, compute-bound workloads or tight capital budgets, the H100 remains the better value. Both chips share the same compute ceiling, but the H200’s extra memory bandwidth and capacity change what fits on one GPU and what it costs to run.

TL;DR:

• The H200’s larger memory capacity and bandwidth make it more suitable for long-context, memory-bound workloads, often justifying its higher initial cost.
• For models that fit on one GPU with 141GB of memory, the H200 can eliminate multi-GPU sharding and associated synchronization overhead.
• Although the H200’s list price exceeds the H100’s, it can deliver lower cost per million tokens in inference due to higher throughput from increased bandwidth.
• Power consumption is higher for the H200 during memory-intensive inference, requiring upgraded cooling infrastructure and careful power budget planning.
• Hardware selection depends heavily on actual model size, context length, and workload type, with memory capacity and bandwidth being key decision factors over raw compute power.

Table of Contents

• How memory bandwidth and capacity change throughput and latency
• Memory capacity, NVLink topology, and node design
• What TCO and cost per token actually look like
• A decision checklist for choosing between them
• Our technical verdict and a procurement framework
• Power draw and thermal management differences
• What the memory-first era means for procurement
• Choose research that treats memory, not just compute, as the real constraint
• FAQ
• Sources

How memory bandwidth and capacity change throughput and latency

The H100 and H200 deliver essentially the same FP8 compute, so comparing them on TFLOPS alone misses the point. The real divide is memory: the H200 packs 141GB of HBM3e running at 4.8 TB/s, against the H100’s 80GB of HBM3 at 3.35 TB/s.

For inference, bandwidth often determines throughput more than raw math does, because generating each token means pulling the full set of weights and the growing key-value cache through memory. NVIDIA reports up to roughly 1.7 times faster inference on some LLM workloads when comparing the two chips in selected configurations, a gap that traces almost entirely to bandwidth rather than compute.

A production detail that changes the math: KV cache and sequence length overhead typically add 10 to 30% more effective memory traffic beyond the theoretical weight-loading requirement, which means real-world throughput ceilings sit below the clean benchmark numbers vendors publish.

A few consequences follow directly from this:

• Longer context windows push more KV cache through memory per token, so bandwidth headroom matters more as prompts and outputs grow.
• Multi-GPU setups add NCCL all-reduce communication overhead every time GPUs synchronize activations across tensor-parallel shards.
• A model that fits on a single GPU skips that synchronization tax entirely, which is one reason the H200’s extra headroom pays for itself in some deployments.
• Short-context, high-throughput batch jobs that are compute-bound rather than memory-bound see much smaller gains from the extra bandwidth.

Memory capacity, NVLink topology, and node design

Capacity determines whether a model needs one GPU or several, and that decision cascades into everything else: software complexity, latency, and cost. A 70B parameter model in FP8 needs roughly 70GB for weights alone, before KV cache. On an 80GB H100, that leaves little room for context, often forcing two-way tensor parallelism. On a 141GB H200, the same model fits with substantial headroom for cache and longer sequences.

Larger models shift the math further. A 405B parameter model in FP8 needs around 405GB of weight storage, which exceeds a single GPU of either generation and requires multi-GPU sharding regardless of which chip you pick. The difference is how many GPUs each sharding scheme requires and how much communication overhead that sharding introduces.

Node-level design matters as much as per-GPU specs. The HGX reference architecture shows an 8-GPU H100 node aggregating up to 640GB of GPU memory, while an 8-GPU H200 node reaches roughly 1.1 terabytes.

• An H200 node’s larger pooled memory lets bigger models span fewer GPUs, cutting the number of NVLink hops needed for each forward pass.
• NVSwitch bandwidth within a node governs how expensive all-reduce operations become as GPU count rises, so fewer required GPUs directly means less synchronization latency.
• Teams planning around a fixed node budget get more usable model capacity per dollar of node infrastructure with H200, independent of the per-GPU price difference.

What TCO and cost per token actually look like

Hardware list price is a minority of the real bill. Lifetime total cost of ownership includes the GPU itself, power draw, cooling, rack space, networking, software operations, and depreciation over a multi-year window, and ops and power frequently dominate that sum over three or more years.

• Hardware acquisition cost, including node, networking, and storage infrastructure.
• Power and cooling, which scale with utilization and rack density.
• Operations and engineering time spent managing deployment complexity, including tensor-parallel setups.
• Depreciation schedule, typically three to five years, which spreads the upfront premium across the chip’s useful life.

A third-party TCO analysis modeling a 70B FP8 inference workload found the H100 delivers an effective throughput around several dozen tokens per second at about one dollar plus per million tokens, while the H200 reaches higher throughput and lower cost per million tokens under certain assumptions. The H200’s higher sticker price is offset by lower cost per token once throughput gains from bandwidth are factored in, at least under those stated assumptions.

That result is sensitive to utilization rate, average sequence length, and the depreciation window chosen. A team running short, low-utilization batch jobs will see a smaller gap, while one running long-context, high-utilization serving will see the H200’s advantage widen. The same analysis frames the H200 purchase as buying simplified software architecture as much as buying memory: a single-GPU fit removes the tensor-parallel all-reduce overhead that a two-GPU H100 deployment carries, which is a real but easy-to-overlook operational savings. For a deeper look at how utilization assumptions change these numbers, see our earlier breakdown of throughput per dollar.

A decision checklist for choosing between them

Run through these signals before committing a procurement budget:

1. Does your model fit on a single GPU with H200’s 141GB but require two or more H100s? That alone often justifies the premium.
2. Are you serving long context windows where KV cache dominates memory traffic? Bandwidth headroom pays off directly.
3. Is your workload compute-bound (short context, heavy batch math) rather than memory-bound? The H100 captures most of the value at lower cost.
4. Does horizontal scaling across many H100s fit your rack power and cooling budget without excessive NVLink overhead?
5. Can your cloud provider or procurement pipeline deliver H200 capacity on your timeline, or does H100 availability make it the practical near-term choice?

Pro Tip: Measure your model’s actual memory footprint, including KV cache at your target context length, before comparing GPU prices. A capacity gap that looks marginal on paper can be the difference between one GPU and two.

Our technical verdict and a procurement framework

The clearest way to choose between these chips is a three-step framework: measure your model’s real memory footprint including KV cache at production context length, map that footprint to single-GPU fit, then compute total cost of ownership against the operational complexity of whatever sharding scheme the alternative requires.

The most common mistake we see is comparing hourly cloud prices without accounting for throughput differences, which inverts the actual cost-per-token ranking. A close second is underestimating NVLink all-reduce overhead when a model barely overflows a single GPU’s capacity, where a small reduction in parallelism can remove a disproportionate amount of latency.

• Measure footprint first, not list price: a model that just barely needs two H100s may run comfortably on one H200.
• Account for KV cache growth at your actual production context length, not a short benchmark prompt.
• Model three-plus years of power and ops cost, not just the hardware quote, before ranking options.

For a broader look at how GPU supply dynamics shape these buying decisions, our analysis of GPU market share claims is worth reading alongside this one, as is our latency measurement playbook for teams quantifying NVLink overhead directly.

Power draw and thermal management differences

The H200 draws more power than the H100 in most configurations, largely because of the additional HBM3e stacks and the higher bandwidth those stacks sustain. Both chips share similar thermal design envelopes in comparable SXM and NVL form factors, but the H200’s higher memory bandwidth under sustained load tends to push power draw closer to the top of its rated envelope more often during memory-intensive inference.

Data center operators deploying H200 at scale generally need the same cooling infrastructure tier as H100 deployments, whether liquid or high-density air cooling, but should budget for higher sustained power draw per GPU during long-context or high-batch workloads. Rack-level power budgets that were sized for H100 density may need rechecking before an H200 refresh, particularly in facilities already running close to their power ceiling. This is less a redesign than a recalculation: the physical footprint and cooling approach carry over, but the sustained draw assumptions do not.

Teams planning multi-year deployments should treat power and cooling capacity as a gating factor alongside GPU availability, not an afterthought discovered during installation.

What the memory-first era means for procurement

GPU buying is shifting from a pure TFLOPS race to a capacity-planning exercise, and that changes who should own the purchase decision. Memory footprint modeling now belongs earlier in the procurement conversation, not as a footnote after price negotiations.

Teams that lock into one chip generation for a multi-year lease risk missing the next bandwidth jump. Favoring cloud flexibility or modular on-prem builds over long fixed contracts keeps options open as Blackwell-class capacity becomes more available. Watch memory bandwidth, not headline TFLOPS, as the number that predicts your next upgrade cycle.

— Devansh

Choose research that treats memory, not just compute, as the real constraint

We built AI Made Simple around exactly this kind of problem: the places where vendor benchmarks, TCO math, and real operational tradeoffs diverge, and where the honest answer takes more than a spec sheet comparison. If this GPU breakdown was useful, our deeper research on credibility checks for AI safety debates applies the same skepticism to a different corner of the industry, and it is one of the pieces our paying subscribers point to most often.

Our content aims to provide access to original frameworks, data-backed breakdowns, and detailed analyses to support deep understanding of such tradeoffs beyond vendor talking points. Start with the AI safety credibility checks piece and see whether the depth matches what you need for your own procurement and research decisions.

FAQ

Is the H200 the best GPU available right now?

The H200 is the strongest Hopper-generation chip for memory-bound inference, particularly long-context LLM serving, but Blackwell-generation chips like the B200 push capacity and bandwidth further still. Whether it is the right choice depends on whether your workload is memory-bound and whether Blackwell capacity is available on your timeline.

Is the H200 the same as Blackwell?

No. The H200 is built on the Hopper architecture, the same compute generation as the H100, with upgraded HBM3e memory. Blackwell is NVIDIA’s next architecture generation, represented by chips like the B200, and is a separate compute design rather than a memory refresh of Hopper.

Why does the H200 cost more than the H100?

The price premium reflects the 141GB of HBM3e memory and 4.8 TB/s bandwidth, both substantially higher than the H100’s 80GB and 3.35 TB/s, which together raise bill-of-materials cost. For memory-bound workloads, that premium often comes back as lower cost per million tokens through higher throughput.

What is the real cost difference between the H200 and H100?

Hardware list price for the H200 runs higher than the H100, but comparing them on sticker price alone misses the throughput difference. In one modeled TCO analysis of a 70B FP8 workload, the H200 reached a lower cost per million tokens than the H100 despite the higher upfront price, because its bandwidth advantage raised effective throughput enough to offset the cost gap.

Sources

• NVIDIA H200 Tensor Core GPU: AI Superchip for Data Centers | NVIDIA
• Components — NVIDIA HGX AI Factory (HGX H100, H200, B200)

Recommended

• Throughput Per Dollar, Not Percent: GPU Utilization for Engineers
• Engineers: Measure and Shrink Compute Overhang With Roofline Metrics
• Flash Attention for Engineers: Implementation Checklist, 2–7× Speedups
• Cut P99 Inference Latency for Engineers With a Measurement Playbook

Thank you for being here, and I hope you have a wonderful day,

Dev ❤

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction.

Help me buy chocolate milk

PS- We follow a “pay what you can” model, which allows you to support within your means. Check out this post for more details and to find a plan that works for you.

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

This article was originally published here. Sign up for free to never miss out and join 200K weekly readers getting the most important insights in AI delivered to their inbox.

The largest technology companies are expected to pour more than $1 trillion into AI across 2025 and 2026, while by 2027 Microsoft, Alphabet, Amazon, Meta, and Oracle are projected to spend about $1.57 in additional capex for every $1 of additional operating cash flow. Add in other big spenders like Nvidia + Chinese Labs + X, smaller players like IBM, Liquid AI, Netflix, Databricks, and an entire economy around RL Environments and post-training, and you are forced to ask: how will anyone justify all of this spend? How will these companies make money?

There are lots of anwers floating around the internet, from 30 Trillion Dollar Market value predictions to Ed Zitron + Gary Marcus style end of the bubble proclaimations. However, all of these wild predictions overlook the much more interesting and nuanced reality. The simple truth is that there is no “AI Model Industry” in the way it’s traditionally defined. While tons of companies might be building and selling models, what they’re doing with those models is extremely different. Different business models, different end games, all lead to very different definitions of what a successful rollout for a company is.

Analysis of these groups is further complicated by the fact that most of these companies playing in the model space are worth billions or trillions. So the traditional mapping used to analyze a company's role in the ecosystem (is it a model layer company, apps layer company, infra, or something else) is often unsatisfactory, as the same company seems to occupy multiple roles.

Therefore, in this article, I will be proposing a new map to analyze AI Model Makers. Instead of looking at crude classifications, we will specifically analyse how each model maker (minus Chinese labs since they have Government backing, so very different rules) is utilizing the models they create to monetize for a very specific vision of how AI will look in the future —

(Ofcourse, many of these companies might be trying to monetize in multiple ways, but if you study their sales and priorities, it becomes clear which bucket their primary focus falls into.)

In order to think about the ecosystem, we can further group 6 buckets of strategy into 3 sets:

1. Set 1 (Strat 1 and 2): are companies that got into intelligence before LLMs. They use models to augment their existing ecosystems and make them more valuable. Their main philosophical divergence is how they want customers to interact with their ecosystem. Strategy 1 tries to pull customers into their ecosystem. Strat 2 tries to work their ecosystem inside their customers' ecosystems.
2. Set 2 ( Buckets 3 and 4): Model native companies (where the model is the primary thing they’re known for). They diverge primarily in their focus (bucket 3 wants to be take over all the most expensive work, while bucket 4 wants diffuse intelligence in every interaction, and then monetize across the board). Since both depend heavily on models, this is the group that is most aggressive about lobbying, regulation etc.
3. Set 3 (Buckets 5 and 6): Don’t monetize models directly. Bucket 5 is focused on expanding the usage of models to increase their customer base, while Bucket 6 looks inwards and utilizes models to improve their own processes. This is the one set where you can invest in both buckets since their focus isn’t mutually exclusive.

By the end, you will understand how different model makers are thinking about intelligence, where each bucket is likely headed, and how you should be thinking about investing your resources into this space to make sure that you don’t get lost in the media hype and come out on top.

Keep reading if that interests you.

Executive Highlights (TL;DR)

• There is no single AI business model. The useful question is not who has the smartest model, but what each company wants to make cheap, what dependency it wants to create, and where it eventually expects to collect rent. The same-looking model release can serve completely different economic goals.
• IBM, Google, Microsoft, and Amazon use AI to deepen larger ecosystems. The historical model was never really “sell intelligence.” It was “use intelligence to pull customers into cloud, software, data, consulting, and infrastructure.” Their bet is that AI works best when the surrounding stack is increasingly centralized and integrated.
• Mistral, Cohere, and Databricks make almost the opposite bet: bring intelligence into the customer’s environment. Private deployment, air-gapped models, customization, and enterprise data integration become selling points because they assume the most valuable AI will be institutional — adapted to how each organization already works rather than forcing that organization into someone else’s stack.
• Anthropic is building AI as premium labor. Its subscriptions provide a revenue floor and get users dependent on Claude; power users then become both heavy metered consumers and internal advocates who can pull Claude into larger enterprise contracts. This explains the obsession with coding, long-running agents, and end-to-end execution: Anthropic needs tasks that are economically valuable, technically verifiable, and important enough that customers stop caring about token prices. The long-term vision is compelling; the problem is financing the journey while competitors attack the same market.
• OpenAI and xAI are betting on ubiquity instead. OpenAI’s apparent sprawl — consumer ChatGPT, Codex, enterprise, ads, commerce, apps, devices, chips — is the strategy. The assumption is that intelligence will create value across thousands of surfaces, so get embedded broadly and monetize each interaction differently. Ads already give free users economic value; commerce lets OpenAI capture transactions; future interfaces could let AI identify or even create demand before the user explicitly asks for anything. The real strategic asset is optionality + time. However, it must be noted that for now, Anthropic’s run rate passed $65B at the end of July against roughly $40B for OpenAI, and OpenAI reportedly ran a negative 122% operating margin on $5.7B of Q1 revenue. So the key will be to survive the next 2 years, so the ecosystem effects can kick in properly.
• This makes cheap intelligence strategically useful to OpenAI. Lower inference costs do not merely improve margins; they unlock entirely new categories of usage. If OpenAI can push itself into Jevons-paradox territory — cheaper intelligence causing massively more consumption — it gets more opportunities to monetize subscriptions, enterprise seats, referrals, transactions, ads, APIs, or whatever new surface emerges next.
• NVIDIA, AMD, and other compute players want everyone else to build more AI. Open models, datasets, synthetic data, recipes, and tooling lower the barrier to creating new workloads. More experiments create more training and inference, which creates more demand for the hardware and software underneath. Their optimal strategy is often not to capture every layer above them, but to make every layer above them bigger.
• Meta uses openness differently. It benefits when the model layer gets cheaper because its real economics sit in apps, ads, commerce, devices, and other downstream products. Open models also outsource experimentation and act as market sonar: thousands of external developers tell Meta which architectures, applications, and technical directions are gaining traction before traditional market research catches up.
• The easiest way to analyze future AI strategy is to follow the subsidy. If a company is giving away models, subsidizing subscriptions, offering unusually generous usage, or absorbing massive infrastructure costs, ask what dependency that generosity creates. The AI market increasingly looks like a fight over which layer gets commoditized, where scarcity moves next, and who owns the tollbooth when it gets there.

Hey there. I keep this newsletter ad-free, w/o sponsors, and mostly w/o paywalls to ensure that everyone has access to high-quality deep-dives on the most important ideas in the AI Industry without any bias. B/w my research, independent verification, and the chocolate milk needed to produce these works, these deep-dives take a lot of time, resources, and infrastructure. For just 10 USD/month or 100 USD/year, you can help me continue my vision of democratising access to the most important ideas in AI by becoming a paid subscriber over here —

Help me buy chocolate milk

(If that is too much for you, we follow a pay-what-you-can plan over here, and it goes down all the way to 1 USD/month or 10 USD/year. No matter what plan you pick, you will get all of our articles without paywalls and have access to all of our articles. Check it out here).

PS — Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

1. AI as Ecosystem: The Original AI Business Model. You have an existing ecosystem that Clients pay for. Use AI to improve it.

make the intelligence useful enough that customers eventually buy the ecosystem around it.

In the dark ages before ChatGPT, IBM was building machines that could beat humans at chess and Jeopardy!, OpenAI punted pro starcraft players, and DeepMind beat the world champion at Go. These aren’t the only examples of this type. So, why were these companies sinking so much money to play games?

Perhaps you fail to see the relevance, so let me be more practical. By 2021 Google was publicly demonstrating LaMDA for open-ended conversation alongside MUM, which could understand information across text and images. At the same time, Microsoft had released GitHub Copilot (writing code from comments) and had created a system that let you upload sketches, which their system would turn into websites.

So why did none of them build ChatGPT? Believe it or not, both of those answers have the same answer: Because, for most of the industry, AI was not supposed to be the product. It was supposed to make everything else easier to sell.

This has nothing to do with the “organizational lethargy” or ignorance that gets peddled commonly in this kind of analysis. Pre-ChatGPT, there wasn’t much of a market for direct model APIs; there was a lot of money in cloud and enterprise software services (seen another way, the model teams were the grunts of the cloud and service teams, who naturally bossed them around to strengthen their own positions instead of letting that be a stand-alone vector). So companies rationally optimized for GTM that unlocked this. Their “unit of value” (our idea exploring why some features can be copied while others can’t, explored here) out of AI was completely different than something like OpenAI, which had no such business team to rely on (from now on, I’m going to tell people that our company is not poor, we’re busy creating a new business model).

Let’s study IBM for a sec. Watson was not really designed as a standalone consumer business. Instead, IBM packaged AI into healthcare, analytics, IoT, financial services, and other enterprise products, while using it to pull customers deeper into IBM Cloud, software, and consulting. By 2016, IBM was explicitly organizing its strategy around becoming both a “cognitive solutions” and cloud platform company.

This legacy carries on today. IBM’s underrated model family Granite is not going to be hacking HuggingFace or stealing people’s Navier-Stokes solutions anytime soon. Instead, IBM has deliberately pushed smaller, more efficient models that are easier to deploy inside enterprise environments. The same goes for their flagship Docling: they open-sourced their top-tier document-processing stack to build credibility, then leveraged it to sell a managed Docling product inside watsonx.

That’s why IBM can open-source their models w/o compromising their business. The model is the bait to get you in the door. Then IBM can sell the harder parts: deployment, cloud, governance, Red Hat, integration, consulting.

Google spent years thinking about AI in much the same way. When Google introduced LaMDA in 2021, it talked about eventually bringing conversational capabilities into products like Search and Assistant. MUM was being built directly into Search. The assumption was not that users would leave Google’s products and spend their day talking to a model. The model was supposed to be the background entity making Google’s existing products better.

While Gemini is sold separately now, it’s also seen as a tool to give Google another way to pull users toward Workspace, Search, Android, and Google Cloud.

We have seen this directly at Irys. We started using Gemini heavily through the API. As the usage increased, Google started reaching out about moving more of our infrastructure onto Google Cloud. From Google’s perspective, that is the ideal outcome. Gemini does not just generate API revenue. It creates a relationship that can eventually expand into a much larger cloud account.

Microsoft and Amazon play variations of the same game. Microsoft can use AI to make Azure, GitHub, and Microsoft 365 harder to replace. Amazon’s Nova models give companies another reason to train, customize, and deploy inside AWS; with Nova Forge, customers can even start from Amazon’s model checkpoints, mix in their own data, and build customized models without leaving AWS.

2. AI Inside Your Ecosystem: Selling AI to Risk Aware Companies by Integrating in their Stack

The last group wants you to move into their stack. This group wants to move into yours.

This might seem like a trivial difference from the previous set (especially considering IBMs service heavy approach), but this philosophy actually represents a fairly deep disagreement about how enterprise AI will work.

Google would quite like you to use Gemini, then Vertex, then BigQuery, then more of Google Cloud. Even IBM ultimately tries to pull you deeper into their complete stack. The bet is that intelligence gets better as more of the surrounding infrastructure is centralized and tightly integrated.

Companies like Mistral, Cohere, and Databricks make the diametrically opposite bet. Asking a bank or government department to reorganize all of that around your AI stack is not always realistic, so they try to work into an org's existing data, security rules, workflows, infrastructure, permissions, and legacy infra.

That is a very different product philosophy from “please send everything to our API/host everything on our cloud.”

Let’s see how this manifests across the big players of this bucket:

• Mistral makes this explicit with their run the models on your infrastructure, under your rules philosophy. Customers can self-host, deploy on-prem, use private cloud, or run at the edge. With Forge, Mistral goes one step further and lets companies build models around their own proprietary knowledge, workflows, policies, and systems.
• Instead of worrying about frontier intelligence, Cohere focuses on limiting liability for risk-averse institutions by answering: where does the data go, can this run inside our environment, can security control it, and can we connect it to internal knowledge without accidentally emailing the crown jewels to California? Consequently, it’s given up on competing with GPT, and its models can instead run inside a customer’s VPC, on-premises, or air-gapped, while Command A was designed to run on just two A100 or H100 GPUs.
• In 2023, Databricks spent $1.3 billion buying MosaicML, because once a company has centralized years of transactions, documents, customer histories, logs, and internal data, the next obvious question is: how can you make any of this useful? Before LLMs, that meant dashboards, SQL, analysts, and BI tools. Now it can mean agents, document generation, internal search, workflow automation, and eventually systems that act against the company’s own data. The 12 Trillion Token, 3K H100 DBRX open release that confused a lot of analysts? The point wasn’t to sell a chatbot, but to prove that Databricks had the infra to let enterprises train models on their own data (while I think custom-trained LLMs are still a technically useless pitch, it was really good marketing, going by what I heard from DB insiders).

Taking a macro-level picture, this is why companies in this bucket are happy giving away their best models with open weights and private deployment. Stronger open models allow them to start the conversation on a more positive note, and since direct APIs were never going to be a huge part of their revenue, this tradeoff makes perfect sense. The airgapped, “in your environment " deployment is something that bucket 1 companies would never do (while all of them allow VPCs and custom deployments, they come with sizable minimum contract requirements, and they’ll always drag you into their tenants/clouds).

Taking an even further step back, I think this divergence b/w Bucket 1 and 2 is an interesting philosophical divergence b/w two thesis around what the interplay of work and intelligence at orgs will look like. The first model assumes intelligence will concentrate inside a few large integrated platforms, and so believes that efficiency and top-line performance will come out on top. The second assumes the really valuable intelligence will be institutional: embedded inside companies, adapted to their data, and increasingly shaped by how each organization already works. In contrast to the first, this is more of a bet that for orgs, limiting risk and playing things safe is a much stronger priority than a few efficiency percentage points.

I personally think that atleast for the next decade, Bucket 2 has a stronger alignment than B1.

3. AI as Premium Labor: Using AI to Handle Tasks that would Require Expensive Manual Labor

The biggest AI revenues will not come from maximizing the number of interactions. They will come from owning a set of interactions valuable enough that customers are willing to pay to delegate to the AI.

Claude does something very interesting with their pricing:

1. Their 20 USD plan doesn’t have much usage and taps out relatively quickly (but it can be used to get your basic jobs of responding to emails well).
2. Their 100 USD plan starts to be viable for heavier research and coding. But at that point, there’s no way they’re making money on their models.
3. They lose a lot of money on their 200 Max Plan (at least if you’re a dehati that drains the usage limits like I am).

So how does Anthropic become profitable (AFTER accounting for expenses, since most of us lack the math genius to understand the creative accounting of Dario-stotle)?

Concretely, there are a few different strategies, all of which circle around the same principles:

1. Most users will never consume enough Claude on the 20 USD plan so they make money here.
2. The 100 and 200 Personal plan people are both addicts and marketing. The former will be discussed in detail, but it’s worth giving a few words to the latter as well. When someone like me uses the 200 personal plan and tells my company that Claude is worth buying, my IT team buys their garbage enterprise plan which bills me at api rates (and Claude is both very token hungry and very expensive). This means that they might lose 100 Million serving power users, but they offset that by gettign 1B in enterprise contracts, where they can recover the margins through API.
3. The addicts end up buying extra credit (which is really good for fundraise docs, and is proof of sticky, price-insensitive demand) and give them their very juicy margins.

This is why Claude (web), Cowork, and Code all pull from the same pool, since the real market intel only comes after the end of the allocated usage limit. However, for this play to really work out, Anthropic needs to two important things: 1) find very high value work items where people would gladly spend a few hundred dollars to save a few hours of work daily; and 2) they need to take over work end to end so that they save time.

Seen from this lens, it makes sense on why Anthropic has largely had a single-minded obsession with coding while OpenAI was focusing on roasting your Instagram and turning you into 80s aesthetics(if you’re doing that shit, please just stop, there are better ways to use your time and electricity). Coding has some very strong advantages that satisfy the criteria we layed out. Obviously, it’s economically valuable, and Anthropic can iterate on it much easier since it’s their domain. Additionally, models can use enormous amount of code and documentation to learn from. Finally, as we’ve covered before, coding is unusually verifiable. Your agents can get a pretty strong signal about how effective their changes by looking at test case performance, latency, and other metrics. While this doesn’t cover all the aspects (and you still have to be very smart with how you design your eval criteria and test cases), it’s still a much cleaner signal than having to evaluate fuzzy attributes like, make this picture more beautiful or make this ppt more rigorous.

This is also why Anthropic has been so focused on training over long running agentic tasks (they were the pioneers of this), since it lets Claude actually finish tasks end to end. The more Claude behaves like a worker rather than a chatbot, the less companies will anchor it’s pricing against ChatGPT subscriptions (cheap, no money) and the more they will compare against the cost of the work being done (expensve, more money).

Seen in another way, Anthropic’s end game is to unlock a future where revenue can scale with the amount of work being delegated rather than the number of people paying for Claude subscriptions/API. The subs, API etc are all meant to be easy entrances into that ecosystem that ultimately leads entire chunks of work being allocated to Anthropic. Philosophically, Dario’s claim of being the only company ever isn’t as outlandish as it was made to me — the game of a setup like Ants is to become the default interface for work (you go to Claude to do X, Claude worries about deploying on cloud, setting up the CI/CD, creating files etc). The Cloud choice, infra etc will all be implmentations handled bts by Claude Agents.

This ambition comes with two drawbacks:

1. They’ll have to train across orchestration, computer use, and several very expensive (and uncertain) training topologies. I’m not convinced that the neural network + backprop+ RL paradigm will get them where they need to go, and a majority of their research seems to be around serving this as opposed to pushing a new direction.
2. To account for the above, Claude has to be pricier than other models. The only way they survive is by being good at individual tasks (rewrite email task better when user provides input), while also being meaningfully better at completing jobs (automatically read emails, draft replies, escalate for approval, and send).

Google/other ecosystem can focus on cost efficiency, while OpenAI, Zhipu and other model-first companies will inevitably converge around the second. This means that hinging on this as your primary differentiator is building a castle of sand in the middle of a battlefield. While I do think this is a fantastic 20 year vision, I don’t think you can focus on it here and now and survive to get to the 20 year mark. You need another “supply line” that can get you the firepower to fight towards the 20 year vision. Anthropic seems to have realized this by focusing on enterprise sales, drilling down on their core competencies, and investing in the FDE play but it’s my personal opinion that in uncertain times, it’s better to have options and mobility than it is to be king of a fortress. From that lens, I can’t see myself being very optimistic about their long-term future.

OpenAI and xAI have realized this, which is why they’re making a very different kind of play. Their strategy is messier, vastly more capital intensive, and probably creates several wonderful new ways to accidentally build Black Mirror. But if for no reason other than that it’s being executed by two of Silicon Valley’s biggest conmen who will rob your granny to raise their valuation, I think this will come out on top.

Anthropic believes intelligence becomes most valuable when you let it take over increasingly expensive work.

OpenAI and xAI believe intelligence becomes valuable in so many places that the winning strategy is to get embedded everywhere first, then discover where the money appears.

4. AI as Air: Embedding Intelligence across our Lives and Monetizing the Interactions

Anthropic wants to own the work. OpenAI and xAI are making a broader bet: intelligence will be useful everywhere, so be everywhere.

ChatGPT already has more than 1 billion weekly users. Codex attacks Anthropic directly on valuable delegated work. OpenAI also has enterprise software, APIs, ads, commerce, apps, devices, and is now building its own inference chips.

That looks scattered until you realize that the sprawl is the thesis.

For OpenAI, the assumption is that there will probably be thousands of places where intelligence creates economic value, and nobody actually knows which ones will produce the biggest businesses. So, their strategy is to get embedded across as many surfaces as possible, and eventually figure out how each one should be monetized.

Consumer distribution is the obvious starting point. ChatGPT Ads have already reached a $1 billion annualized revenue run rate in under 200 days. This might not make ChatGPT profitable, but it buys them some more time (imo the most important strategic asset), in order to take their shadiyantra a few steps further.

For instance, ChatGPT already lets users complete purchases inside the conversation while OpenAI takes a fee from merchants, and apps like Booking.com, Expedia, Zillow, Spotify, and Canva can surface directly when they become relevant to a conversation. Even if they never become top #1 e-commerce + booking destination, one can easily see them capturing a non-trivial part of this market, further deepening their value capture from the platform. In this case, the nature of the human-app interaction provides an interesting advantage to OpenAI since ChatGPT often sees intent before it becomes a search query. This is a bit abstract to say, so let’s see the utility by walking through an example.

I might tell it that I need dinner near a particular place, one person is vegetarian, I want somewhere quiet, and we are walking somewhere afterward. ChatGPT can turn a vague set of constraints turn into commercial intent, w/o my never searching “restaurant near me.”

Take that logic a little further, and things get funky.

Imagine a future OpenAI device knows that I train combat sports, has access to my health signals, and notices after my sparring day that I have probably gone through intense exercise and am unusually dehydrated. I haven’t yet asked for anything, but it tells me I should probably rehydrate and eat. Since it knows what I tend to like, it finds somewhere nearby that fits, and asks whether I want it ordered.

Slightly dystopian? Obviously. But for now (and always), let’s focus on the value we can create for the shareholders. This kind of interaction is quite different to the current platforms or search monetizations for a simple reason: OpenAI did not capture demand. It created the transaction before I had consciously formed the demand myself. This means they can plausibly expect higher cuts of the business.

The same thing could happen with services. Ask ChatGPT enough health questions and, at some point, the useful answer is no longer another paragraph — it is the right doctor. The same applies to lawyers, accountants, tutors, consultants, contractors, financial advisers, or niche experts.

This will likely have some interesting societal consequences. The internet era brought the attention economy. An AI-enabled marketplace with more sophisticated matching could enable a scaled up GLG model by scaling the expert economy (matching people to the exact experts they need). One can imagine an interaction where an AI assistant answers what it can, recognizes where human expertise becomes necessary, understands enough context to know which human(s since we’ll want options) you need, and takes a referral fee for making the connection. This interaction would be a great way to reduce hallucination/liability for the AI platform as well (which is why I’m surprised it hasn’t been a stronger priority so far, but probably the engineering and compliance for such a thing would require much forethought).

And from a business perspective, this would OpenAI eventually monetize intelligence that it doesn’t even provide itself.

This also contextualizes OpenAI’s chaotic push to build across interfaces since they don’t want intelligence trapped inside “open ChatGPT, type prompt.” This requires a much more holistic approach, hence why their compute strategy now explicitly spans chips, data centers, models, developer infrastructure, consumer products, enterprise products and AI-native devices. I expect them to continue to invest a lot of money in model compression, inference chips, heterogeneous compute, etc because ubiquity only works if intelligence gets dramatically cheaper. They’re essentially trying to bully their way into Jevons Paradox greatness, and if they pull it off, it will be an era-defining change in how businesses are seen and evaluated.

The numbers here make this seem more realistic than you’d think.

This has its pros and cons. OpenAI isn’t really a category leader the way Anthropic has become, and they’re that much less likely to hit that next breakthrough that unlocks next-gen models (or drops costs the way Chinese Labs are doing) since their attention is split into so many fronts. They have the money and resources to run Bell Labs-style R&D, but the split focus has led to a huge handicap in their bid to capture knowledge work. And much of their successful GTM plays (the Sora app, ghibli style pictrue etc) has not been useful for converting attention to actual revenue. This means that they’re bleeding more money than Anthropic, while failing to capture as much value as them.

However, OpenAI is making money, and they still have some investor dollars. This gives OpenAI something I think is underrated: optionality and time. If they can survive the next 2 years, they will have an endless number of cards they can play to monetize their cpative audience (many of whom are deeply dependent on ChatGPT specifically, and dont like Claude/Gemini). Then, OpenAI can make money from the subscription. Or the API. Or an enterprise seat. Or an ad. Or the merchant when you buy something. Or a referral when it sends you somewhere. Or perhaps one day the device sitting beside you when the demand is created in the first place.

I have my reasons to be optimistic for OpenAI’s strategy (if I had to make my pick for the best strategy, it would be this). They’re still fighting Anthropic’s war too. Codex has already reached more than 5 million weekly users, with roughly 20% now coming from non-developers doing research, spreadsheets, presentations, contracts, and other knowledge work. Amongst a lot of commentators (myself included), Codex is considered superior to Claude Code (and certainly much better value for money). Between that, the ads, their new FDE investments, their fundraising, and the ecommerce, one can see the temporary supply line we discussed earlier. And once they figure out how to monetize a single user, just a little but, Sama will be walking around in clothes stitched from 100 USD bills.

xAI is making a philosophically similar bet. X explicitly says its goal is an “Everything App” spanning information, communications, media, payments, banking and commerce, with Grok embedded throughout it. Intelligence is another layer through which all of those surfaces can become more useful — and more monetizable.

5. AI as Compute Expansion: Make it Easy for Everyone to Train and Use Models, Sell the Hardware/Platform they will need to do so

NVIDIA or AMD do not need its model to win. It needs everyone else to build enough AI that compute demand keeps exploding.

This is probably the simplest business model in the article.

NVIDIA publishes open models, datasets, training recipes, RL environments, evals, and tooling. Nemotron now ships with open weights, training data, and reproducible recipes, while NVIDIA has released 10 trillion+ language-training tokens alongside open robotics, autonomous-driving, and biomedical datasets.

Why go through this effort? Because every reduction in the difficulty of building AI creates more workloads

The logic is straightforward: make models easier to build → more companies build them → more training/fine-tuning/inference → more compute gets consumed.

AMD is playing the same game from the challenger position. Its Instella family comes with weights, datasets, training configurations, and code, while making the rather unsubtle point that the models were trained on Instinct GPUs using ROCm. Cerebras has done versions of this too: its open Cerebras-GPT family was explicitly built to demonstrate what its wafer-scale systems could train.

You might pause here and ask yourself why NVIDIA stops there. With its cash and position, why not move further up the stack and become the cloud, SaaS company, or end-user AI platform too? Capture more of the value chain, as the business schools teach you.

Probably b/c Lisa Su and Jensen Huang weren’t smart enough to pass the formidable challenge that is the MBA degree. Just imagine how well they would have done with a few more middle managers and Excel merchants.

There is also the fact that these are expensive businesses — and, more importantly, many of the companies NVIDIA would be competing with are its customers. Cloud providers, neoclouds, and model companies are major buyers of NVIDIA infrastructure. Rather than spending heavily to replace them, NVIDIA has generally preferred to help them expand: it invested $2 billion into CoreWeave, while DGX Cloud Lepton aggregates capacity from CoreWeave, Crusoe, Nebius, Nscale, Lambda and other cloud partners. That way, if they blow up, Nvidia has demand. If they suffer, Nvidia’s downside is limited.

The same applies higher up the value chain. NVIDIA could spend billions trying to build a dominant SaaS product, but then it starts competing with the exact model and software companies it wants standardizing around CUDA, NeMo, NIM, and NVIDIA hardware.

So its optimal move is often not to capture every layer.

It is to make every layer above it bigger.

Open models create developers. Open datasets create training runs. Synthetic data creates specialized models. Recipes make customization easier. Tooling makes deployment easier. Every successful experiment has a decent chance of eventually becoming another inference workload.

That is why compute companies are unusually happy to give away intelligence: they monetize the consumption of intelligence rather than the intelligence itself.

Anthropic needs Claude to win. OpenAI needs enough of its bets to win.

NVIDIA and AMD mostly need everyone to keep using more compute.

6. AI as Commoditizing the Complement: Give away an LLM for free, enjoy the benefits of community RnD flowing into your actual money makers.

If the model isn’t where you make your money, making models cheaper can be more valuable than selling them.

Meta spent billions building models like Llama, then handed the weights to everyone else. When they first started doing this, the enlightened folk over at Wall Street hated it. Since then, Meta has outperformed financial analysts almost every quarter.

Why? If Meta made a major chunk of its money from selling models (or had a massive ecosystem where Models can be gamechanging like Google, to which Meta is often compared), this would be financial suicide. But that’s not what Meta does, and once you understand the different dynamics of their space, everything will make sense.

(We did a deep dive on why Meta Keeps Beating Financial Analysts over here if you’re interested in their strategy in more detail)

Meta makes money from Instagram, Facebook, WhatsApp, ads, commerce, and devices. They spend a lot of money on coding, content review, and creating/reviewing a lot of documents that corporates have to deal with. If capable models become dramatically cheaper, all of those aspects get better while Meta becomes less dependent on OpenAI, Anthropic, Google, or whoever happens to control the best closed model that month.

This is basically the old commoditize your complement strategy: destroy pricing power in the layer you need to consume, then capture value somewhere else.

Open models also give Meta two useful side-effects:

• Outsourced R&D: researchers, startups, and random people with too many GPUs test fine-tunes, architectures, tools, agents, and applications Meta would never explore internally. Meta funds the core asset; the ecosystem helps pay the experimentation tax.
• Market intelligence: open-source activity becomes a kind of market sonar. If developers suddenly start pushing Llama into a particular workflow, architecture, or interface, Meta sees where demand and technical progress are moving before those trends necessarily show up in normal market data.

I explored this much more deeply in my earlier Meta piece, particularly how the company has historically used React, PyTorch, and Llama not just as software releases but as ways to shape and observe the ecosystem around them. But for our purposes we can distill the key idea into: open source doesn’t just distribute your technology. It lets thousands of outsiders tell you what your technology should become next.

Meta’s strategy is becoming more complicated now that it is also willing to monetize models directly, so this is no longer the entirety of its AI play. But that does not kill the logic behind open releases. Meta can monetize some high-end capability while still benefiting enormously from pushing the broader intelligence layer toward lower margins. since Meta doesn’t make a majority of the money from the model serving, it can generate a lot of value for itself, even if the margins on model serving are horrible (hence why Muse is consistently very low cost for agentic tasks).

Conclusion: Follow the Subsidy to predict the Future of AI

Follow what they are giving away. Then figure out what dependency they are trying to create — and where they expect to collect rent once that dependency exists.

Far too often, investors ask me what the best model is as a proxy for where they should invest their money. In my opinion, this is a rather incomplete way to evaluate a market where so much is uncertain and where paradigms can change on a dime.

Where certainty eludes our grasp, it is perhaps better to bet on strategy and direction. Instead of asking about what the “best” is right now, it’s better to focus on where the company is headed by asking ourselves: what is this company willing to make cheap?

Anthropic subsidizes subscriptions because it wants expensive delegated work later. NVIDIA gives away models and data because it wants more compute consumption. Meta is happy crushing model margins if cheaper intelligence strengthens everything sitting above them. OpenAI will happily burn money giving people more intelligence today if those interactions eventually become ads, transactions, enterprise seats, referrals, or something nobody has invented yet.

As we’ve seen, the entire AI industry is a bit identity fluid right now. Model companies are building chips. Chip companies are moving into cloud and software. Open-source companies are launching paid APIs. Consumer AI companies are invading enterprise work. Nobody knows exactly where the durable profit pool will settle, so everyone is quietly buying options on somebody else’s business model.

That is probably what the next phase of AI competition looks like. Model quality will still matter, obviously. But as intelligence gets cheaper, the more important fight will be over who can afford to commoditize what — and what they own when the value moves somewhere else.

Thank you for being here, and I hope you have a wonderful day,

Dev ❤

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction.

Help me buy chocolate milk

PS- We follow a “pay what you can” model, which allows you to support within your means. Check out this post for more details and to find a plan that works for you.

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

This article was originally published here. Sign up for free to never miss out and join 200K weekly readers getting the most important insights in AI delivered to their inbox.

Google acknowledged in the report that its “moonshot” climate goals, including its 24/7 carbon-free energy (CFE) ambition, are becoming harder to achieve, as its AI infrastructure buildout accelerates faster than the grid is decarbonizing.

— Source

AI data centers are becoming a major part of the electricity system. In the U.S., they could consume roughly 12% of national electricity by 2030. In Ireland, they already account for 23% of metered electricity, up from just 5% in 2015. Utilities are planning new generation, transmission and storage around this demand, while Google, Microsoft, Amazon and Meta all say they match their electricity use with renewable energy.

The problem is that “100% renewable” can describe several very different things. A company may buy enough renewable electricity or certificates to match its annual consumption while its data centers remain connected to grids that still use gas, coal, nuclear and renewables.

In case you think I’m making this up — in FY2025, Microsoft continued to allegedly match 100% of its annual electricity consumption with renewable energy. In that year, while its electricity consumption rose 24%, its reported electricity emissions 10xd from roughly 260,000 tonnes to 2.7 million tonnes. Most of that discrepancy came purely from the change in accounting used by Microsoft. This had me thinking about how trustworthy the claims of “100% clean data centers” are and how much energy, water and other resources data centers actually consumed.

The following is part 1 of a 2 part investigation into this topic. In this article, we will look into questions such as:

• The reality behind Big Tech’s “100% clean” data centers (and what is the accounting trick behind these claims).
• Are these data centers actually driving more fossil-fuel power, even while companies report cleaner emissions on paper?
• Are the massive clean-energy and nuclear deals creating genuinely new power, or mostly changing who gets to claim power that already existed?
• Who ultimately pays for AI’s electricity boom — the tech companies, utilities, or ordinary households and businesses?
• If AI is going to consume a meaningful share of national electricity, are we measuring its real impact properly at all?

Keep reading if you’re interested in this and more.

“This increase in demand will result in data centre electricity demand consumption increasing from 22% of national electricity demand in 2024 to 31% by 2034.”

— Ireland Commission for Regulation of Utilities — Data Centre Connection Policy. This Data Center Problem is global.

Executive Highlights (tl;dr of the article)

• AI data centers are becoming a major electricity load: they could consume roughly 12% of U.S. electricity by 2030, while Ireland is already at 23%.
• Big Tech’s “100% renewable” claims usually mean annual matching, not that data centers are physically running on clean power every hour.
• The accounting gap can be enormous: Google matched 100% annually but only about 65% hour by hour; Microsoft’s market-based electricity emissions jumped roughly 10× after it changed how it counted renewable certificates; Meta reported roughly 135 tonnes market-based versus 5.86 million tonnes location-based for its data centers — about 43,000×.
• Renewable procurement can still help, but the key question is whether the buyer actually causes new clean power to exist. A certificate from an existing plant is very different from financing a new solar farm or restarting a closed nuclear plant.
• Even genuine clean projects can arrive years after the data center. In the meantime, utilities may build gas, storage and transmission or keep fossil plants open longer; Georgia’s latest buildout is a clear example.
• The better test is not “is this 100% renewable?” but: what did the company buy, what actually powered the data center, and what changed in the grid because the data center existed?

Hey there. I keep this newsletter ad-free, w/o sponsors, and mostly w/o paywalls to ensure that everyone has access to high-quality deep-dives on the most important ideas in the AI Industry without any bias. B/w my research, independent verification, and the chocolate milk needed to produce these works, these deep-dives take a lot of time, resources, and infrastructure. For just 10 USD/month or 100 USD/year, you can help me continue my vision of democratising access to the most important ideas in AI by becoming a paid subscriber over here —

Help me buy chocolate milk

(If that is too much for you, we follow a pay what you can plan over here, and it goes down all the way to 1 USD/month or 10 USD/year. Check it out here).

PS — Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

1. What Does “100% Renewable” Actually Mean — and Does It Really Help?

Data centers don’t run on dedicated green power lines and are plugged into regional grids that mix solar, wind, nuclear, gas, and coal in real time. So, when a company says it is “100% renewable,” they’re describing an annual paper match: balancing gross megawatt-hours consumed over 8,760 hours against total megawatt-hours of clean generation purchased globally. An operator can draw 100 TWh from coal-fired grids during freezing winter nights, buy 100 TWh of midday solar in a different interconnection during July, and report net-zero power.

This means that all three of these scenarios can be true at once:

• A company consumes 100 TWh of electricity.
• It buys renewable energy covering 100 TWh.
• Some of its data centers still draw from grids using substantial gas or coal during many hours of the year.

This process is called annual matching. On the surface, it seems innocent enough, but you will see how this accounting trick compounds with others to muddy the waters and greenwash the conversation around the sustainability of data center operations.

How the same Data Center can produce two very different emissions numbers

Under the GHG Protocol’s Scope 2 rules, companies can report purchased-electricity emissions in two ways:

• Location-based: How carbon-intensive are the grids where the company actually consumes electricity?
• Market-based: What electricity and environmental attributes has the company contractually purchased?

Contrary to their name, using market-based emissions can be used by companies to report dramatically lower emissions w/o really reducing their emissions. This is where we must learn about a concept Renewable Energy Certificate, or REC. When a renewable generator produces 1 MWh of electricity, it can effectively sell two things:

• the physical electricity;
• the REC, which acts like “ownership” of the electricity.

So, a wind farm may sell its physical electricity to one group and the certificate to a different one, not even connected to the wind farm's grid (we call this an unbundled REC since the certificate and actual electricity are now separated).

This is how companies use accounting to “reduce” their paper-emissions, while still increasing their carbon emissions. A common counterpoint here is that this difference isn’t a huge deal since big companies buying RECs → more money for renewable generators → more incentive (+ resources) to build out renewable grid → we eventually enjoy a tree-hugger utopia.

Before we move on, I think it’s worth addressing two common points that are used to justify all in building of data centers w/ no regard for the consequences (under the assumption that Data Centers and AI will Save the World so any cost is bearable).

• Data centers are getting more efficient.” True. Uptime Institute’s latest survey puts average industry PUE at 1.52, falling to 1.36 when weighted by capacity, which suggests the larger and newer facilities driving much of the current buildout are meaningfully more efficient. But this is efficiency per unit of compute, not total environmental impact. If each unit gets cheaper while the number of units deployed grows much faster, total electricity use can still rise sharply. Our good friend Jevons has found us once again.
• “AI could reduce emissions elsewhere.” Also plausible, and probably true in some sectors. The IEA estimates that widespread adoption of existing AI applications could enable emissions reductions equivalent to roughly 5% of global energy-related emissions by 2035. But that is a modeled future benefit dependent on adoption, substitution, and rebound effects; the electricity consumed by data centers is measured today. So both should be counted, but they should not be casually netted against each other as if one were a current meter reading and the other were equally certain.

Now let’s get back to our discussion about RECs and if they actually reduce grid emissions. Spoiler alert: it’s complicated.

Do Renewable Energy Certificates Actually Reduce Grid Emissions?

Think of renewable procurement as a spectrum:

• Buy a REC from a renewable plant that was already operating: you acquire the renewable claim, but your purchase may do little to increase the amount of clean electricity on the grid.
• Match consumption with clean generation during the same hour and on the same grid: much closer to the physical electricity problem, but the generator may still have existed without you.
• Sign a long-term contract that allows a new wind or solar project to secure financing: your purchase can directly help new clean generation get built.
• Restart a closed clean power plant or keep one from shutting down: your contract can add or preserve clean electricity that otherwise would not be available.

The sorting principle around this ranking boils down to one question:

Would this clean electricity exist without the buyer?

This ensures that the green energy becomes a permanent upgrade, not simply a checklist off the corporate checklists. Energy researchers call this additionality. Keep this term in mind, since we will come back to this a few times, and this metric is at the start of many sustainability discussions, even outside data centers. Looking from the additionality lens, the picture starts to look a bit less rosy:

• A 2024 Joule study modeling voluntary clean-energy procurement in the western U.S. found that annual matching produced minimal long-run system-emissions reductions in the scenarios studied, while hourly matching produced substantially larger reductions.
• A 2022 Nature Climate Change study found that REC use had inflated the apparent progress of the companies it examined; if historical trends continued, the researchers estimated that 42% of their committed Scope 2 reductions would not translate into real-world mitigation.

While this doesn’t make RECs useless, it does add a bit asterisk to any major claims of sustainability. (We will dig into the numbers on whether the data center buildout is helping w/ the growth of renewable in much more detail later).

Before we fall into doom and gloom, it’s worth highlighting that the opposte is also true: corporate renewable procurement can drive new investment. Google, for example, is pretty active in signing massive renewable energy deals with purchase agreements, which encourages buildouts and expansions.

So the problem, like in many other cases, isn’t the lack of work (although I think we could be doing much more), but rather the way one loaded term can mean wildly different things, which confuses the conversation and makes meaningful climate audits and activism harder.

Reading this, you might be thinking about how problematic these word games actually are. After all, every writer on the internet suffers from afflictions where they are willing to declare holy war on whatever topic catches their fancy at that moment. So let’s run the numbers and see the spread when we put Google, Microsoft, Meta, Amazon and Oracle side by side.

“The pace at which new electricity demand is being sought by data centres is faster than the pace of network infrastructure delivery.”

— CRU — Final Large Energy Users Connection Policy, December 2025

2. So How Clean Are Big Tech’s Data Centers, Actually?

Once we move past the headline “100% renewable” claim, the meaningful question becomes simple: how much of a company’s actual electricity use is matched with clean power when and where it is using it?

Google is the clearest example because it reports both numbers.

It says it matched 100% of its annual electricity use with renewable purchases in 2025. But it also says only about 65% of its electricity use was matched with carbon-free energy hour by hour in 2025.

This tripped me out quite a bit, so we’ll spend some time clarifying what this means:

• If Google uses 100 units of electricity over a year, and buys or contracts for 100 units of renewable electricity over that year, it can say it has 100% annual matching.
• But if only 65 of those 100 units line up with carbon-free electricity in the same hour and region as Google’s actual consumption, then its hourly carbon-free energy score is 65%.
• This creates a surplus of 35 units of renewable certificates (I wonder if these will eventually be sold as commodities). Remember, this is not an excess of actual electricity since the certificates are unbundled from the physical current.

Google’s regional numbers show how large those differences can be. Its latest available regional data show:

• Stockholm: 100%
• Finland: 98%
• Mumbai: 9%
• Singapore: 4%
• Hong Kong: 1%

(from the Google regional carbon-free energy data)

I think it’s really cool that Google is actually reporting its hourly matching stats, since it reflects greater seriousness around the topic. It adds a bit more transparency to a space typically wrapped up in linguistic sleight of hand and sustainability theater.

Microsoft is a great example of linguistic theater/activism through definitions (the aforementioned 10x difference in emissions when you use a less friendly accounting metric), but they’re not the worst here. That honor goes to Meta, where their Meta Environmental Data Index reports roughly 135 tonnes of market-based Scope 2 emissions versus 5.86 million tonnes of location-based emissions. Any time tech is doing something shady, Meta is right there to lead the charge. Have all the moral problems with them you want, but at some point, you gotta appreciate their dedication to supervillainy and ruining this world for the rest of us.

To round out our discussion: Amazon reports 100% annual renewable matching but does not publish a directly comparable global location-based Scope 2 figure in its corporate reporting. Oracle reports about 92% renewable electricity in its cloud operations (gotta appreciate how they don’t even try for 100% unlike the new-age tech cos.), with location-based Scope 2 roughly 6.7× its market-based figure.

“Singapore’s electricity demand is expected to grow rapidly… driven by… energy-intensive sectors such as semiconductor manufacturing and data centres.”

— Singapore Energy Market Authority — Tuas Power 670 MW Gas Plant Award. Singapore is 93% Carbon Energy. so this will up their emissions a lot.

All this teaches us one important thing: we can’t treat company renewability claims as simplistically as their PR teams online pretend. Perhaps our Hobbesian tech overlords simply wish to protect our fragile and myopic minds from turning away from utopia.

Having studied the renewability claims extensively, we must move on to the next step in understanding the impact of data centers on our energy grids: even when these companies buy clean power, are they actually causing more clean electricity to exist?

3. What Happens Before the Clean Power Shows Up?

Sometimes, yes. But even this situation is more nuanced than you’d think. Even when a hyperscaler helps finance genuinely new clean generation, the data center and the power project may come online years apart, since new U.S. power projects generally have to enter an interconnection queue before connecting to the transmission grid (grid operators study whether the system needs new transmission equipment or upgrades before accepting them).

This process can be incredibly inefficient. Berkeley Lab’s latest data show:

• Only 13% of generation capacity that entered U.S. interconnection queues from 2000–2020 had reached commercial operation by the end of 2025.
• Projects that actually came online in 2025 had spent a median of more than five years between requesting interconnection and beginning commercial operation.
• Another 549 GW already had a draft or executed interconnection agreement but was still not operating.

This means that even when a new data center comes with its own source of clean energy, it might still take years before that clean energy goes live (this might be why some companies are pushing for special generators for only their data centers). In that interim, you would be increasing load on the existing grid, even as the companies buys credits and claim green operations.

Georgia and what Happens when Demand Outpaces Supply

Georgia Power’s 2025 planning process authorized procurement of up to 8.5 GW of new capacity. The Georgia Public Service Commission says approximately 80% of the new generation approved in December 2025 is expected to power data centers.

For scale, 8.5 GW running continuously would consume about 74.5 TWh a year. (For some context, using the EIA’s latest full-year average of 865 kWh per U.S. residential customer per month in 2024, that is roughly the annual electricity consumption of 7.2 million homes).

The final certified portfolio totals 9,885 MW:

• 3,692 MW of new combined-cycle gas generation
• 2,762 MW of standalone battery storage
• 350 MW of battery storage paired with solar
• 1,195 MW of PPAs from the original RFP
• another 1,626 MW of long-term PPAs and extensions
• 260 MW of additional battery storage paired with an existing solar PPA.
  Georgia Power SEC filing — full resource breakdown

So, in order to power the buildout of these supposedly clean data centers, Georgia will rely on new fossil fuel plants for almost 40% of their energy needs. But make sure you keep your air conditioner in check.

The same 2025 planning process also extended the expected life of existing coal generation. Georgia Power had previously indicated plans to retire Plant Scherer Unit 3 by the end of 2028. The 2025 IRP extended it through at least December 31, 2035. Plant Bowen Units 1 and 2, which Georgia Power had also previously indicated could retire by the end of 2028, are now assumed to operate through at least 2035.

It would be unfair to paint data centers as solely responsible for this expansion, but we can’t also overlook how much of the new energy demand is driven by them. This is a problem that will get much worse since the deregulation + buildouts for data centers will be much faster than the efforts to bring on more green energy.

To see how bad this can get, let’s look at a story playing out at this very moment.

The xAI lawsuit

In April 2026, the NAACP sued xAI and its subsidiary, alleging they were operating 27 gas turbines without the required air permit in Southaven, Mississippi to supply the nearby Colossus 2 data center. Court filings put those 27 turbines at a combined generating capacity of at least 495 MW. May 2026 preliminary-injunction filing

(from the lawsuit)

xAI claims these gas plants will be temporary but this will have meaningful long-term negative impacts on the climate and health of the people in the areas. So I guess if I want to stress one thing, it would be this: even if companies will eventually create a world where their data centers are fed through clean energy in the future, powering the data centers at this moment can still have severe consequences that should not be overlooked.

4. Who Pays for AI’s Power Boom?

If data centers are forcing utilities to build billions of dollars of new generation and grid infrastructure, who gets the bill? Contrary to the sensationalist reporting, the answer is not necessarily you.

Georgia’s approved projects and related transmission investments total about $16.7 billion, but Georgia Power has also committed to structuring its next rate case so that revenue from large-load customers creates at least $556 million a year of downward pressure on rates from 2029–2031, equivalent to about $8.50 a month for a typical household using 1,000 kWh. That is still a projection, not realized savings, but large data centers can be valuable utility customers because they buy enormous amounts of power consistently; the bigger risk is a utility building for demand that gets delayed or never appears.

A few states are offloading the risk back to the buyers of data centers. New AEP Ohio data centers face long-term capacity commitments, minimum charges even if they use less power than they reserved, and collateral requirements designed to keep more of the infrastructure risk with the customer creating it. This has not killed demand: by February 2026, developers had signed binding contracts for another 5,642 MW under the new tariff, on top of 12,219 MW contracted before it took effect.

Virginia has adopted the same principle. New large customers of 25 MW or more enter a separate rate class; qualifying data centers must pay at least 85% of their contracted transmission and distribution demand regardless of actual usage, commit to service for at least 14 years, and can be required to guarantee up to 60% of their minimum contract charges. Both of these arrangements work on the principle that utilities should not spend billions preparing for hyperscale demand and then dump the risk onto everyone else if the customer disappears.

We’ve gone quite deep into the accounting behind data centers and their renewability. All of this has led to one question —

Conclusion: So How Should We Actually Measure Whether a Data Center Is Clean?

At this point, I think we can retire “100% renewable” as a useful standalone measure of sustainability. It tells us something about what a company bought, but we have now seen that it can coexist with dirty local grids, huge differences in reported emissions, new gas plants, and years where the clean generation being financed does not even exist yet.

A better audit needs three separate answers:

• What did you buy? Did you buy cheap certificates, hourly clean energy, or finance genuinely new generation?
• What actually powered the data center? What was the carbon intensity of the local grid, ideally hour by hour rather than averaged across a year?
• What changed because you showed up? Did your demand cause new renewables, nuclear, gas, transmission or storage to be built? Did it keep a coal plant running longer? Would those things have happened without you?

We can take some comfort in the fact that the world seems to be moving in this direction. In July 2026, the GHG Protocol proposed a multi-statement framework separating physical emissions, market-based emissions, and the emissions impact of corporate actions and investments, rather than forcing all three ideas into one number. The new joint GHG Protocol/ISO corporate standard is expected in Q4 2028.

However, given the rapid rise in emissions and the increasing climate death, we need to be taking action much quicker. We must campaign for stricter metrics, more rapid actions, and much more transparency in the measurements.

We will be doing a follow-up on the complete resource needed for Data Centers. Stay tuned for that one.

Thank you for being here, and I hope you have a wonderful day,

Dev ❤

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction.

Help me buy chocolate milk

PS- We follow a “pay what you can” model, which allows you to support within your means. Check out this post for more details and to find a plan that works for you.

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

This article was originally published here. Sign up for free to never miss out and join 200K weekly readers getting the most important insights in AI delivered to their inbox.

Kimi K3 attracted a lot of attention for it’s insane performance, cheap cost, and for being an open weights model that performs at frontier model levels. With it come several numbers that have been catching everyones attention: the massive 2.8 trillion total parameters (104 B active), 1-million-token context window, and an insane 57 at launch on the Artificial Analysis Intelligence Index (they’ve now upgraded it to 60, putting it within touching distance of the top and above other open weight models).

Intelligence aside, this system really stands out with increasingly important intelligence per dollar, completing high value value tasks at a fraction of the cost of the othe frontier models.

While all of that is very cool, I think people have been overlooking a much more interesting story regarding how Moonshot (the makers of Kimi) attacked transformer scaling limits across three different (but related) bottlenecks that dilute important signals in transformers:

• Across Tokens: Instead of storing every past token’s key-value pair across every layer, it maintains a dynamic recurrent state that updates with new inputs. This keeps KDA’s recurrent-state cache fixed in size (overall more efficient), although K3’s 24 MLA layers still use a KV cache that grows with sequence length.
• Across Layers: Standard transformers pass a blunt running sum of hidden states down the stack, which dilutes earlier signals. Attention Residuals lets deeper layers selectively retrieve earlier block-level representations, preserving fine-grained context across deeper architectures.
• Across Capabilities: Traditional Mixture-of-Experts pushes high-dimensional token representations through every selected expert, creating a massive inter-device communication bottleneck. Stable LatentMoE routes compressed latent states to hundreds of fine-grained experts instead, scaling model capacity without saturating network bandwidth.

Combining dynamic state tracking with cheap routing and direct layer retrieval gives you a system that maintains high arithmetic throughput and coherent memory over long autonomous task runs. To really amp up that last ability, Moonshot trained this architecture inside execution environments across thousands of sequential tool calls, persistent application state tracking, and millions of context tokens.

More generally, Moonshot consistently pushes the boundaries on interesting model architecture changes (they’re the ones that pioneered the Muon -Clip optimizer) so I think this aspect is worth studying in depth to understand the future of how models will be designed going forward. Let’s get deeper into it.

A visual comparison of why MuonClip let AI converge faster.

Executive Highlights (tl;dr of the article)

Kimi K3 achieves its exceptional performance per dollar by attacking four places where Transformers either lose information or become expensive: across long sequences, deep networks, large expert pools, and long-running tasks.

Across tokens, the problem is memory. Standard attention stores information about every previous token, so memory usage grows with the conversation. Moonshot created Kimi Delta Attention to replace most of this growing history with a fixed-size recurrent state that continuously updates what it remembers. Since compression can lose details, K3 places one global-attention layer after every three KDA layers, giving the model periodic access to the complete token history. Because KDA’s state changes in sequence, it also carries information about token order, allowing K3 to extend its context to 1 million tokens without modifying positional encodings such as RoPE.

(recurrent mechanisms are making an insane comeback)

Across network depth, the problem is information dilution. A normal Transformer repeatedly adds each layer’s output into one running hidden state. As the network becomes deeper, useful early representations can become buried under everything added later. Attention Residuals gives later layers access to summaries of earlier stages, letting them selectively recover the representations they need instead of depending entirely on the latest accumulated state. K3 groups its 93 layers into blocks to make this retrieval practical.

(FYI, this is another one of Kimi’s structural innovations. One of the reasons that I’m such a Moonshot glazer is that while a lot of attention is put on how good their models are, people don’t appreciate just how much good research they do to push the field in innovative directions)

Across experts, the problem is communication. Mixture-of-Experts models increase capacity by routing each token through a small selection of specialist weights. However, moving large token representations between experts spread across different GPUs can become more expensive than the expert computation itself. Stable LatentMoE compresses K3’s representation from 7,168 to 3,584 dimensions before routing it to 16 of 896 experts. Quantile Balancing, normalization, and bounded activations prevent this unusually large expert pool from collapsing during training.

Across long tasks, the problem is execution. A model can answer difficult questions while still failing at work that requires hundreds of actions, tool calls, corrections, and changing application states. Moonshot post-trained K3 inside reasoning, coding, and agent environments where it had to act, inspect the result, recover from errors, and continue until an external system verified the outcome. This produces unusually persistent behavior, although it can also make K3 expensive, excessively proactive, and sensitive to whether its earlier reasoning history is preserved.

K3 points toward a future where models do not process every token, layer, or parameter uniformly. They will increasingly decide what to remember, what to retrieve, and which capabilities to activate. If that direction holds, the next research frontier shifts toward better selectors, reliable judges, and persistent state — while hardware, memory systems, and post-training data reorganize around those decisions.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

How Kimi K3 Uses KDA to Get Compressed Memory Without Losing Full Attention?

I will now attempt to summarize the ~roughly 30K words we wrote self attention into two important bullet points —

• Standard full attention lets every new token compare itself against every previous token in a sequence. That gives the model exact recall (within the window, not accounting for the model not overlooking/blurring the token, also subject to randomness behaving), but this causes the KV cache to expand linearly with sequence length. This means that memory footprint and bandwidth demands will break standard serving infrastructure when trying to hande the 1-million-token context that your AI girlfriend needs to remember how you like your little love letters to be written (something we broke down in depth when discussing how much a token actually costs) —

• Linear attention avoids that growing cache by routing inputs through a fixed-size recurrent state. Each token updates and reads from this compressed memory, keeping memory costs flat regardless of context length. However, this undrmines the precise recall of full self attention, leading to a more lossy compression of your corpus (see more here — How Long Context Inference Is Rewriting the Future of Transformers).

Moonshot built Kimi Delta Attention (KDA) to address this memory degradation. At its crux lies a simple idea — prioritize writing a delta-rule update that corrects an existing memory association before writing new data (generally a wise principle to consider w/ agentic systems these days), to keep KDA from overwriting representations blindly. It also assigns independent retention rates to individual channels, allowing separate parts of the state to preserve or discard features independently rather than forcing all channels through a single global forget gate. The mechanics of this change are quite interesting, so worth studying if you’re on the model layer and want to look at the different ways attention is evolving. For our purposes, this high level explaination will have to suffice.

Because compressed memory can still drop fine-grained details, K3 interleaves compressed state tracking with exact retrieval. Across its 93-layer backbone, the model runs a 3:1 hybrid layout: three KDA layers for cheap local token mixing, followed by one Gated Multi-Head Latent Attention layer that restores unrestricted global attention across the raw token history. This setup is topped off by a final global-attention layer to squeek out that extra performance.

This is essentially the same setup, scaled up, from their Kimi Linear research, where they introduced this architecture with a 48B param model and it outperformed the standard architectures (emphasis mine). “We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6× decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths.”

(although one might argue that a vanilla architecture is a bad baseline nowadays given how much is changing, but this was still directionally useful).

One interesting outcome of these changes is that your model now handles position across long contexts w/o needing standard positional encodings like RoPE. From an AI perspective, this is a change that deserves a deep-dive, which is what we will do next.

Why Does K3 Remove RoPE from Global Attention?

Standard self-attention has no concept of sequence order. The core score between two tokens comes down to a dot product divided by the square root of the head dimension:

Because dot products are permutation-invariant, swapping the order of the inputs produces the exact same arithmetic. Models use a causal mask to stop a token from looking forward, but it can’t tell whether a past key was generated 1 token ago or 100,000 tokens ago.

Rotary Position Embedding (RoPE) injects distance by rotating queries and keys in the 2D plane before computing their dot product. For a query at position m and a key at position n, the score becomes:

The absolute position terms m and n cancel out, leaving only the relative distance (n — m). The rotation angle theta_i for dimension pair i is set by a geometric base frequency:

Fast frequencies (low i) spin rapidly over short distances, letting the model distinguish adjacent tokens. Slow frequencies (high i) spin slowly, tracking long-range separation across the context window.

However, this approach struggles to scale context beyond your training limits. If you train at 8,000 tokens, the slowest frequency dimensions complete only a fraction of a single rotation. When you push the sequence to 1,000,000 tokens, those slow dimensions rotate into unseen coordinate regions, while the fast dimensions complete hundreds of full periodic cycles (remember that these are based on trigonometric functions, which lap around each other every 360 degrees/2 pi radians) . At lengths far beyond training, the model encounters relative distances and combinations of phase relationships that it was never trained on causing its positional coordinate grid is producing out-of-distribution values (not merely “sequence length” as is often misunderstood).

Standard fixes try to rescale theta. Position interpolation scales the sequence index down so that m / scale fits within the original 8K window. YaRN scales the base constant directly (e.g., jumping from 10,000 to 500,000 or higher) to stretch the wavelength of slow dimensions while interpolating fast ones. This works short-term but every extension requires manually selecting a new scaling factor and retuning hyperparameters for a fixed maximum length. AI promised to automate manual work, and now you end doing a lot of manual labor to make AI work. Isn’t this world simply wonderful?

To avoid the rising labor costs in China, Moonshot decided to skip the manual labor by forgoing by removing RoPE from its 24 global MLA layers. Instead of having global queries and keys carry position rotations, K3 tracks sequence order by feeding inputs into MLA pass through Kimi Delta Attention which tracks sequence evolution through an explicit recurrent state matrix S_t:

Because S_t depends strictly on S_(t-1), the state at step 100 cannot exist without executing the preceding 99 updates in chronological order. An earlier token’s signal decays over time through repeated multiplications of alpha, and is updated whenever newer tokens write to overlapping memory coordinates.

This creates a pretty interesting geometric diference:

• RoPE represents distance as a rigid geometric coordinate (n — m).
• KDA represents history as a dynamical system, encoding whatever information survives the intermediate state transitions.

By alternating three KDA layers with one NoPE MLA layer across its 93-layer backbone, K3 delegates sequence ordering to KDA’s state transitions, allowing the global MLA layers to perform pure content-based retrieval across already-ordered representations.

This simplifies the context extension a lot, and let Moonshot scale pre-training from 8K to 64K tokens, and then to 1M tokens in cooldown, w/o modifying frequency bases or interpolating position grids. Code base/Architectural simplicity is an often an overlooked aspect of performance, even though I think it’s one of the most important.

“Within this research setting, we found that differences in architectural complexity could account for 50% drops in productivity, three-fold increases in defect density, and order-of-magnitude increases in staff turnover.”

— How bad is Architectural Complexity. I would imagine this is worse now that AI Coding Assistants tend to be overtly verbose and overcomplicate code bases with extra tests, making maintaince harder.

The combination of KDA and NoPE allow Moonshot to not lose important signal across many tokens. However, this is not only one part of the battle when it comes to information dilution in transformers. Next, we will talk about how Moonshot built Kimi K3 to keep information across the layers (ensuring important information wasn’t diluted as the transformer reasons).

“Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer’s contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights…Scaling law experiments confirm that the improvement is consistent across model sizes, and ablations validate the benefit of content-dependent depth-wise selection. We further integrate AttnRes into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-train on 1.4T tokens, where AttnRes mitigates PreNorm dilution, yielding more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks.”

— [2603.15031] Attention Residuals

How Do Attention Residuals Let Kimi K3 Retrieve Features Across Model Depth?

Transformers solved horizontal context scaling by letting tokens query tokens across a sequence. Their solution for handling vertical depth has been a bit more…. primitive.

In a standard PreNorm architecture, each layer adds its computed output directly into the residual stream:

x_(l+1) = x_l + Layer_l(x_l)

By the time you reach layer 90, the hidden state is an unweighted sum of every operation that came before it. While this keeps gradients stable during backpropagation, it does unfortunately mean that your early, fine-grained representations get drowned out inside a massive accumulated vector.

(it’s perhaps a bit poetic that our AI, much like ourselves, overlooks its history and suffers from a strong recency bias/myopia)

Attention Residuals (AttnRes) fix this by running attention vertically through the depth of the network —

Instead of inheriting an unweighted sum of the entire stack, a layer computes attention scores over previous layer outputs to pull the specific representations it needs. This lets your system get very customized with the level of feature extraction it needs, enabling a better quality of reasoning.

More proactive ones here might be thinking about how running full depth-wise attention across 93 individual layers would have your fast GPU memory moving like Al-Teta’s Haramball (if it doesn’t cause memory crashes). You would be correct. K3 avoids this by grouping its 93 layers into eight 12-layer blocks, with a partial final block at the top. The model attends across these block-level checkpoints rather than individual layers, capturing the cross-depth retrieval gains without running out of memory bandwidth.

In Moonshot’s 48-billion-parameter Kimi Linear run across 1.4 trillion tokens, Block AttnRes matched the validation loss of a standard residual model trained with 1.25x more compute. It also kept hidden-state vector norms bounded across deep layers and distributed gradient updates far more evenly across the stack.

The two sets of solutions ensure that the “input” to AI reasoning is of a high quality. But no amount of “preprocessing” (again I use these analogously instead of literally) can save you if your information is being processed by a crayon snorting idiot. If we think of the last 2 stages as cleaning the input and priming the problem space, then the next natural layer of attack becomes how do we level up the raw intelligence + knowledge stored in the network weights.

The answer, as has been increasingly for everyone in the space, has been adopting Mixture of Experts. If you’re reasing this newsletter, you’re likely familiar with MoE so I won’t waste time describing for the sake of driving up word count. But that also begs the question — given how table stakes MoE is, why bother talking about it?

Typically more experts allow you to add more specialization. So the folk responsible for K3 decided to expertmaxx to the moon with a massive 896 experts.

If that number means nothing to you, let me put this into context. Normal Frontier models are b/w the 128–256 expert range. For additional context, Kimi K3’s 896 is roughly 3.5× the viral DeepSeek-V3 and 7× Qwen3.

So…why doesn’t everyone do this? There are several engineering constraints in mind. Kimi overcoming these is a massive deal. Let’s cover how Kimi did that since I predict that that others will copy this approach.

How Kimi K3 Uses StableLatent MoE to Make 896 Experts Usable

Let’s do some math to understand why Kimi’s approach is such a big deal:

• K3 runs 896 routed experts per MoE layer and picks 16 for every token. Activating only 1.8% of the pool looks cheap on paper, but life is unfortunately rarely that simple.
• This still means that the model runs 104 billion active parameters per token (over three times K2’s 32.6 billion). Above and beyond the usual suspects you’d be paying for, all 2.8 trillion parameters must stay resident across your GPU cluster because the very next token might route to an entirely different set of devices.
• This means that you have to bend over backwards keeping high-count expert parallelism from choking on device communication.
• This is why you don’t see labs full send on an ungodly number of experts.

So what does Kimi do differently? Standard expert parallelism sends the full hidden representation to every selected expert across the network. If you select more experts, your all-to-all communication volume scales linearly with the model’s hidden dimension.

Moonshot decided that if the full representation was a problem, then they simply shouldn’t send the full representation:

• Two shared experts process the full 7,168-dimensional representation to handle universal transformations.
• The model projects the token representation down to a 3,584-dimensional latent space before routing.
• The 16 selected specialist experts run their computation entirely on this 50% compressed latent vector. (this is an interesting application of latent spaces that I think will eventually lead to the shared universal latent space for everyone to reason over that I think is inevitable, although I might perhaps be the man with a hammer in this case)

Splitting shared work from specialist work halves the tensor size moved across the network during routing, letting K3 consult 16 distinct experts without saturating inter-GPU interconnects. However, this still has to deal with a massive problem: Routing across 896 experts introduces severe training instability. When you push sparsity that far, a handful of popular experts attract most tokens while the rest starve, causing routing collapse and exploding activation values across deep projection layers.

Moonshot stabilized the setup with three specific mechanisms:

• Quantile Balancing: Updates router bias values directly from the score distribution needed to hit each expert’s target load, preventing load imbalances without using blunt auxiliary loss penalties.
• Pre-reconstruction Normalization: Normalizes the routed latent outputs before projecting them back to the full 7,168-dimensional space, preventing activation spikes.
• SiTU-GLU Activations: Replaces standard SwiGLU with a bounded activation function to clamp internal representation drift across deep layer stacks.

This super specialization of Kimi K3 is what unlocks it’s high performance on the intelligence while keeping the costs low (the 16 experts per run is still okay) (this part is my speculation).

Working across these 3 layers of solutions, you get a model that is able to think intelligently. However, our world only validates intelligence if it help your McKinsey Consultant or investment banker make their slide decks faster. So how does Moonshot ,actually get Kimi K3 to actually work?

How Was Kimi K3 Trained to Handle Long-Horizon Agent Tasks in Production?

Once pre-training gives K3 its raw model capacity, Moonshot uses reinforcement learning to shape it’s behavior towards multi-tool execution (take notes b/c this is the correct way to use RL/post-training, not trying to inject more domain knowledge).

Moonshot trained K3 across three domains: general reasoning, general agents, and coding agents. Instead of tuning on simple conversation logs, the agent environments ran inside live terminals, codebases, web browsers, and simulated tools like Slack and Notion. The training rewarded an execution loop: act, check compiler or tool errors, patch the code, and keep going until an external verifier clears the output.

An interesting side effect of this execution heavy RL is that b/c the model is trained to survive long runs, its default is to keep pushing boundaries(so when user intent is ambiguous, K3 often makes assumptions and executes instead of stopping to ask for clarification). It also likely contributes to the brittleness of Kimi when thinking history isn’t passed properly to it (both these limitations are mentioned in their release notes here: Kimi K3 Tech Blog: Open Frontier Intelligence)

All of this is reflected in the benchmarks:

• On AA-Briefcase, which tests multi-step execution converting raw files into complex deliverables, K3 scored near Claude Fable 5 on analytical quality.
• That performance carries a steep test-time tax: K3 averaged 83 turns, 120,000 output tokens, 56.4 minutes of wall-clock time, and $10.57 per task.

So where does this leave us? Why should you care about Kimi K3? Why did we spend all these words studying the architecture?

What Does K3 Tell Us About the Next Phase of Frontier Architectures?

Imo, Kimi K3 is a strong signal that the next generation of breakthroughs will come from unlocking better information flow within models. Building meta-cognition into architectures, so models can selectively filter, prioritize or queue look up will unlock the next generation of intelligence for models.

In other words, future architectures won’t process every token/layer uniformly. We are moving toward systems that dynamically decide what to keep in recurrent memory, what to pull across network depth, and which specialist weights to activate on the fly. Assuming that to be true, our new research frontier shifts from building intelligent systems to building reliable judges and state.

As models change focus from intelligence to useful work completed, our AI stack will haveto reorg it’s priorities. Here are some trends I expect:

• Hardware and interconnects: Peak FLOPS matter less than device communication. Clusters need higher bisection bandwidth and low-latency interconnects rather than just denser compute dies.
• Memory management: Serving runtimes can’t rely on monolithic KV caches alone. Systems need hybrid memory managers that track dynamic recurrent state vectors alongside paged KV blocks without fragmenting GPU RAM.
• Post-training data: Static text scrapes are losing value. The high-margin training asset is verified execution data — multi-turn terminal logs, sandbox trajectories, and external test harnesses where models learn to recover from actual tool failures.

Plan accordingly.

Thank you for being here, and I hope you have a wonderful day,

Dev ❤

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction.

Help me buy chocolate milk

PS- We follow a “pay what you can” model, which allows you to support within your means. Check out this post for more details and to find a plan that works for you.

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

This article was originally published here. Sign up for free to never miss out and join 200K weekly readers getting the most important insights in AI delivered to their inbox.

A lot of people have been caught off guard by Google’s sudden string of high-profile exits, with Google’s stock reacting very negatively to this news —

On paper, Google should be the runaway winner of the agentic AI race. They have the complete stack, reputation, talent, and the cash to fight costly wars of attrition against Anthropic and OpenAI (their 2 biggest competitors). They have a strong research culture that spawns some of the most promising ideas in all of tech.

So…why are the Lionhearts at Google in a fetal position, while all the people that don’t match up as well are wailing on them (if you get this reference, we should be buddies)? And did this sudden collapse really come out of nowhere?

If you’re a regular reader at the Chocolate Milk Cult, the answer to the 2nd question is a clear no: we’ve been talking both about the deeper structural issues at DeepMind/Google and about how you should expect some very high profile exits because of that. For instance, take our livestream back in February, when Gemini 3.1 Pro dropped, and everyone was staring at the benchmark numbers — 77.1% on ARC-AGI-2, #1 on Artificial Analysis. I was less excited about the whole thing —

Six months later, here are some news items that are worth paying attention to:

• The flagship model is stalled. Gemini 3.5 Pro missed its target window, with internal reports pointing to persistent failures in complex coding workflows.
• Redundant teams are building overlapping tools. DeepMind, Google Cloud, and Android spent the last year building competing AI coding assistants. Gemini CLI and Code Assist are now being forcibly merged into Antigravity just to stop the internal cannibalization.
• Internal friction is leaking. DeepMind and Google Cloud’s have had a mean tug of war, with executive battles even playing out in the tech press.
• Key talent is walking. Senior researchers who built the core Transformer and Gemini foundations are leaving for lean, single-product AI startups. What about Google seems to alienate their high agency builders (this pattern is much deeper than just the celebrities).
• Leadership had to step in. Google just restructured DeepMind’s operating leadership with a very high-profile organizational reshuffle.

All of these stories point to the same underlying cause: Google as an institution is being torn apart from the inside by multiple competing philosophies on what it should be. This means that while individual teams and engineers might do great work, it’s hard for Google to move towards a meaningful long-term goal due to a lack of a deeper underlying identity.

This article will be one of the most honest and in-depth looks at Google and where things keep going wrong for a company that should have everything going for it. While I do have access to a lot of insider comments from high-level Google execs in my network (it’s how I was able to tell y’all that Demis’s faction at DeepMind wasn’t as “clean” as some engineers liked to imagine and that they were trying to do some hardcore lobbying to position Demis as the next Google CEO), I’ll keep most of the article focused on easily verifiable public facts and reports to avoid getting into an unproductive “he said, she said”.

Executive Highlights (tl;dr of the article)

• DeepMind, Cloud, Android, and Workspace have spent years building toward different goals, often duplicating tools and fighting over product ownership and TPU access.
• Google repeatedly fails to compound its own advantages. It buried multimodal embeddings, kept TPUs unnecessarily difficult to adopt, fragmented its coding agents, and fired the engineer behind a viral Workspace CLI while another team rebuilt the same thing.
• The same incentives distort Gemini. Benchmarks offer research teams autonomy and easy internal recognition; reliable agents require messy cooperation across models, tools, permissions, infrastructure, and products. So Google keeps producing intelligence that looks better on charts than it feels in actual workflows.
• Those incentives also drive out high-agency builders. Google creates extraordinary researchers, frustrates them when they try to ship, then watches them build competitors — or pays billions to bring them back.

The recent consolidation around Antigravity and Koray Kavukcuoglu suggests Google has decided to make Google Cloud it’s primary Champion. That could work. The real test will be whether this centralization of power enables deeper alignment, or if it will turn Google’s brightest into automatons.

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

Google Keeps Building Overlapping Products Instead of Compounding Value

In business, moats are anything that deters new competitors from entering your business stream. For example, if I wanted to start a soft-drink competitor to Coke, I would have to set up my own distribution partnerships and find my own set of poor people to steal water from. Moats provide incumbents with a lot of stability and time to react to any competition; consequently, investors and product managers love moats.

As an organization, Google has a very unique skill of ruining its own work in building these moats (outside of its standard search and ads space). The big G loves to turn what should be strong iterative loops of development, testing, and improvements into Jackson Pollock-esque clusterfucks.

Our first case study on this matter will be the coding agent mess.

Consider the following — Both OpenAI and the big G launched their own CLI products to try and capture some of Claude Code’s smash-hit success. Both Codex and Gemini CLI were vastly inferior products to CC. But they ended up with very divergent paths.

Today, Codex has bridged the gap against Claude Code and has had viral growth of its own. Additionally, the same core Codex engine follows a developer everywhere: inside the CLI, embedded in the IDE, in the web interface, and across cloud environments. Every fix, every state-tracking enhancement, and every context-window optimization compounds into the exact same platform layer.

Google ran the opposite play. Take a look at this passage from here —

“Google co-founder Sergey Brin and others were advocating for Google to move faster to seize opportunities in AI coding, but their efforts were slowed by competing factions within the company, two former employees said. Cloud computing unit Google Cloud, research lab Google DeepMind and the team behind the Android operating system are all building AI coding tools for developers, with involvement from some consumer product teams, too, people familiar with the work said.”

To stop the internal friction, Google is now consolidating those parallel workstreams into Antigravity so that state-tracking improvements, tool-use optimizations, and context management developed by one team can finally flow across all interfaces.

Which… yes. Obviously you would want that. It’s why we teach first-year engineers that software is a team sport. But it somehow took Google months to get there. (To be clear, at their scale, it’s understandable that you had multiple teams start the agent buildout simultaneously; what’s not acceptable is that they took this long to converge).

This reveals a fundamental difference between Google and leaner AI labs. OpenAI and Anthropic make bad product decisions, but when an architectural pattern works, they collapse organizational resources behind it. Google lets competing implementations survive, duplicate, and fragment.

Sometimes that fragmentation produces redundant product SKUs. Other times it produces structurally bizarrely self-destructive edge cases.

Justin Poehnelt had been an engineer at Google for nearly seven years when he built an open-source Google Workspace CLI designed for humans and AI agents. The tool gave agents a structured, low-latency protocol to interact directly with Gmail, Google Drive, and Google Calendar via standard API calls. It hit #1 on Hacker News and picked up thousands of active developer users overnight.

Google fired him.

Internal reports indicated that while individual engineering teams wanted to adopt his tool, other divisions raised concerns about external branding and unapproved release channels. Two days prior to his termination, Google publicly announced an official Workspace CLI was coming anyway.

Compare this response to OpenAI’s when they heard about JP getting fired —

A simple appreciation of high agency and an invitation to talk further (from the creator of OpenClaw, someone who was himself scouted by OpenAI).

So, a builder inside the company identifies a massive friction point in agent execution, builds a working primitive, and validates user demand in production. Instead of absorbing the tool, standardizing its security boundaries, and scaling it, Google’s corporate structure treated the initiative as a governance breach — firing the engineer while a separate team worked to replicate the exact same feature set from scratch.

That, my friends, is a sickness. A sickness of apathy and decay where the company fails to recognize and work on the exceptional work being done internally.

Cue story number 3: multimodal embeddings.

Google had a multimodal embedding model in Vertex AI that could embed text, images, and video into the same semantic space. Video embeddings were generally available by early 2024. You could actually take video, generate embeddings from it, and do things like semantic video retrieval.

That was not a typo. Google did functional video embeddings early 2024. These embeddings weren’t great (they were a nightmare to use since they were gated behind Vertex and couldn’t do audio), but this should have been a big deal regardless. The cutting-edge models at this time were Gemini 1.5, GPT -4 Turbo (do you even remember this one), and Claude 3. Some additional grounding — GPT-4o wasn’t even released then.

So why did almost no one talk about this? Simply put, it was hard to get to. The feature sat inside Vertex AI without clear integration into Google’s primary developer frameworks, developer marketing, or higher-level SDKs. Their dev-rel guys wrote some technical documentation, published sample notebooks, and then left it buried inside Cloud’s platform menu as a feature developers had to stumble across by accident. And actually using it was even worse, since you had to jump through 5 hoops and gluck gluck your nearest Google Cloud support rep b/c of how hard setting everything up properly was.

The result? Most people didn’t even know this existed, Google lost a LOT of money as other embedding models took center stage for semantic retrieval systems, and they had to pay a 100 M to license tech from the overpriced and overrated Contextual AI just to try to have a presence in this space(which has so far also failed).

It really didn’t have to be this way. With some marketing, easier accessibility, and an overall better rollout + integration into their ecosystem, they could have made a lot of waves in both the direct embeddings space and the indirect follow-on products from it (OCR, bulk document/video ingestion, or bundling it up with their knowledge graphs services and some Gen AI for selling a serious file QA contender (which is a market being eaten by Codex and Claude Cowork rn)).

This isn’t easy, and it could have still failed. But at least they would have failed attempting to make progress towards capturing a valuable slice of the world. Not failed because they were thrashing around with no overarching goal.

In all cases, the pattern is eerie: Google tends to overlook promising work being done in their own org. Sometimes it’s because it’s done by a little team that doesn’t know how to get attention; sometimes it’s because the work done with whatever is aligned with Google’s agenda (which is itself often dictated by competitors), and thus it suffers from attentional blindness. Sometimes it’s just because Google genuinely does a lot of good work, so people don’t know where to focus, and they tend to default to whatever is the default it doesn’t help that their team is a weird mix b/w PhDs who don’t study the industry and think about the ecosystems and MBAs who don’t think about much at all).

In all cases, it leads to very promising ideas being overlooked for eons before someone eventually gets that market and Google realizes that they have to play catch-up there. To really drill this pattern home, here is the most egregious example of this: TPUs.

Across the industry, engineering teams evaluate AI infrastructure almost exclusively in terms of NVIDIA hardware, often treating Google’s Custom Silicon (TPU v5p and Trillium) as an internal Google detail rather than a viable cloud hardware platform. That market perception stems from how Google organized hardware access internally. For years, Google Cloud did not fully control the supply of one of its most important products. The group selling TPUs sat inside Google’s core engineering organization, while most chips were reserved for internal use. In 2022, Thomas Kurian successfully lobbied to move that group into Cloud. Until then, Cloud reportedly needed approval from another part of Google before it could offer TPU capacity to customers.

This extremely delayed reorg had some pretty disastrous consequences for Google’s market position. TPUs were originally engineered to run on Google’s proprietary frameworks (JAX and XLA) while the broader machine learning community standardized on PyTorch. Because TPUs were largely internal, Google delayed deep, native integration between PyTorch and the TPU compiler stack.

Why does any of this matter? Because due to this delay, developers chose to pay NVIDIA’s hardware premium rather than rewrite their model code for JAX during the biggest chip demand boom ever.

Take a second to think about how stupid this is. Google built TPUs more than a decade ago b/c they realized that all current hardware was inefficient for Neural Networks. They are one of the people who helped make Gen AI mainstream. And at no point did any tubelight think of making their TPUs easily accessible to customers. Their teams were too busy patting themselves on the back by writing about their efficiency gains in the appendices of their papers. That is a generational, Lukaku first-touch level of fumble, and it reflects in the business growth of Nvidia and Google Cloud (keep in mind the image below is Nvidia Data Center vs ALL of Google Cloud)—

Caption: Google began offering Cloud TPUs externally in early 2018. Yet NVIDIA’s Data Center business produced $75.2 billion in its latest quarter, compared with $24.8 billion for the entirety of Google Cloud — not merely TPUs, but GCP, Workspace, and other enterprise products. Google TPU history, NVIDIA results, Alphabet results.

By late 2025, Google was forced to allocate heavy engineering resources to TorchTPU specifically to fix the software friction that prevented customers from adopting their hardware. This will do wonders for their hardware adoption

Even with structural adjustments, those internal friction points persist. Google Cloud executives continue to negotiate directly with DeepMind leadership over the allocation of constrained TPU clusters. The hardware supply sits inside the same company, yet different business units negotiate internal transfer prices and compute access as if they were third-party vendors. There is also a very large bloc of researchers who push for TPUs to stay in-house so that they can use them for experiments. This creates a lot of unnecessary conflict.

This conflict is the driver behind Google’s recent rupture. And in case you haven’t caught it, it contains Google’s two shining aspirations —

The Messy Relationship Between Google Cloud and DeepMind

The TPU fight points to a larger problem: DeepMind and Google Cloud aren’t trying to win the same game. DeepMind wants Gemini to be the undisputed best model on the planet. Google Cloud wants Google Cloud to win the enterprise market.

Those two goals have a strong overlap, but they aren’t strictly the same thing. Cloud doesn’t need Gemini to top every leaderboard, and Kurian doesn’t even strictly need his enterprise customers to use Gemini at all. If a Fortune 500 bank wants Anthropic’s Claude, Cloud will happily sell them Claude through Vertex AI, host the workload on Google hardware, plug it into BigQuery, Cloud Spanner, and Google IAM, and collect the margin on the entire surrounding platform.

That last sentence might sound like Heresy, but I’m citing Kurian directly here. Cloud aggressively positions itself as a neutral multi-model platform and has hired senior leadership directly from Anthropic to drive that push.

As you might imagine, this strategy doesn’t go over well with some of the more purist elements at DeepMind. But while this would be problematic enough as is, there is another deeper rupture that was causing an internal arms race. It isn’t as bombastic, so it’s been largely overlooked by the media/discussions, but imo it’s far more interesting since it hints at the future of model development and research. To see how, let’s go back to Gemini<>Cloud dynamics.

Theoretically, while Gemini isn’t needed to make Cloud a lot of money, it can still be exceptionally useful. Running an in-house model like Gemini on proprietary TPUs eliminates the third-party margin tax, lets Cloud undercut competitors on inference pricing, and captures economics that OpenAI or Anthropic can’t touch. In this world, the relationship b/w Google Cloud and DeepMind should still be tight. So why isn’t it?

It’s because for this vision to work, Google doesn’t need Gemini to be the best. Or even the 3rd best. It just needs it to be good enough at a really good price. Essentially, it needs to turn Gemini’s training focus from intelligence to intelligence per dollar.

If you found a way to have Gemini give up 5% on an abstract reasoning benchmark but in return it connects cleanly to BigQuery, handles enterprise permissions without hallucinating access tokens, and executes deterministic API calls across 20 turns, Kurian will build you a temple in the GCP HQ.

DeepMind, however, will drive up the prices of voodoo dolls out of their passion for you. Their focus and internal culture optimize for frontier capabilities, either by topping benchmarks or in doing cool and next-gen research. By itself, this isn’t a bad thing, but it creates an extreme problem when this culture (as it often is with research-heavy orgs) becomes hostile to the plumbing/boring engineering work needed to make the model boringly reliable within an org. At that point, there are only two ways forward:

1. You go all in on research, à la peak Bell Labs. Google has the free cash flow, core business moat, and the talent density (also diversity; once you look beyond the standard publications in the LLM research space, they actually have some insane projects floating).
2. You turn the research into a way to improve your product/feed your business (Amazon does this very well). Here Cloud would function as DeepMind’s massive real-world testing ground. When Gemini mangles a SQL query inside BigQuery or fails an enterprise IAM boundary, that failure telemetry should flow straight back to London to guide the next post-training checkpoint.

Both these directions can work. But both require commitment, and a clearly defined order of priorities. There can only be one king at a time, and this wasn’t a decision that Google was willing to make so far. So, the two divisions have spent years in bureaucratic turf wars over developer tooling, product boundaries, and TPU allocations. The Information documented how AI Studio was bounced between Cloud and DeepMind. Reuters reported that Cloud executives openly welcomed Koray Kavukcuoglu taking operational control of DeepMind because he is commercially minded and actually focuses on product execution.

From that lens, the leadership shakeup is a clear signal that Google has seen DeepMind’s struggle to really establish a dominant lead, factored in Cloud’s 82% quarterly + 390% backlog growth, and decided that Google Cloud is the right horse to back.

Hopefully, this consolidation addresses one of Google’s biggest problems: the company has spent much of the AI boom optimizing for the kind of intelligence that photographs well but disappoints in person. Why is that putting aside Gemini 2.5 Pro, Gemini has failed to capture the attention of the larger community in the same way that Claude and OpenAI have?

Why Does Google Keep Optimizing for the Wrong Kind of Intelligence?

To be clear, Gemini isn’t a bad model. It routinely matches or beats OpenAI and Anthropic on math, abstract reasoning, and multimodal benchmarks. Every few months Google drops a checkpoint alongside a small rainforest of charts showing that the line went up again. Think back to 3.1 Pro’s release, and how it was meant to be a slam-dunk against the rest of the industry. The numbers were compelling, influencers went wild, and the entire ecosystem stood on attention. Then people used it for a sec, and pretty much everyone went back to Claude Code or Codex to get actual work done.

Why would this happen? And why have Google’s releases failed to inspire the same level of awe as the others, even when the benchmarks tell us that we should love the models?

If you’re familiar with the industry, you’d know that the answer is benchmark-maxxing (we even broke it down extensively here). The more interesting question is why does Google seem to be particularly badly hit by benchmark-maxxing.

In an org like Google, benchmarks are the ultimate political currency. A research team can train a model, run an eval script, post the score on X, all w/o needing to navigate internal politics and negotiate access with the mail/drive teams. In other words, optimizing around a benchmark provides two holy grail outcomes in corporate environments —

1. Autonomy: the team can work by themselves with no blockers.
2. Legibility: middle management can point to the benchmark wins and thus continue to justify their existence. If they have a particularly good result, then they even get visibility into the ecosystem.

Almost by definition, this is anti-thetical to what agentic systems need. Models that thrive in agentic loop have to be able to coordinate different tool schemas/outputs, have to be extremely reliable so that you they don’t produce wildly different outputs due to slight variations in outputs, and has to preserve complex workflows/state across 40 steps. In case of Google, this work cuts across Cloud, Workspace, security, infrastructure, and product teams. Since one teams failures can reflect badly on everyone (and credit is also shared, with the model team the highest profile receipient), loss aversion keeps teams from coordinating properly.

(You don’t have to take my word on this, their own customers have complained about this experince)

Sources: Google Drive Help (2024); Gemini Help (2026); Google Workspace user report (2025); BigQuery user report (2024); Google’s BigQuery warning (2026); Google Cloud command failures (2024); “Gemini Code Assist is a mess(2025)”.

Antigravity and the operational promotion of Koray Kavukcuoglu is likely a push to bring DeepMind to finally yield to commercial pressure and optimize for completed work, not great benchmark scores.

This will be useful, but alongside this, Google needs fix their deeper human incentive alignment issue. Inside a bureaucracy, the safest career move is to stay in your lane, collect cross-functional approvals, and let every VP touch the steering wheel. High-agency builders do the opposite: they see a broken primitive, bypass the committee, and ship.

Think back to Justin Poehnelt (Google engineer that built the viral Workspace CLI and then was fired for it). A less high profile version of his profile plays out a lot more than you’d think. Let’s see some case studies:

• All eight authors of the original Transformer paper left to start or lead competitors like Character.AI, Cohere, Sakana AI, Essential AI, Inceptive, and Adept.
• Noam Shazeer left in 2021 when Google refused to ship the chatbot he built, founded Character.AI, and came back in 2024 via a $2.7 billion licensing and talent deal to co-lead Gemini. Less than two years later, he left for OpenAI.
• AlphaFold co-creator and Nobel laureate John Jumper left DeepMind for Anthropic, followed by senior personnel departures across the core Gemini teams.

Google routinely attracts the people who push the frontiers, frustrates the builders who want to productionize them, and then buys their startups back at massive valuations when the frustrated builders eventually leave. This has two problems:

1. Obviously, losing the high agency people is frustrating. They leave with ideas, prototypes, and a lot of institutional knowledge. These people will either go to more aggressive competitors (very bad) or start their own startup. In the latter case, their brands allow them to raise at high valuations, so Google will have to buy them back at a very high number.
2. The people left are the types who are more comfortable with bureaucracy (either the political types or the complacent types). As their numbers grow (and they find themselves in more senior positions), they will deepen the babu-giri culture at the Big G.

It’s no coincidence that Google tends to be the biggest poaching ground for AI Labs (Google is the big tech company most likely to lose it’s seniors to AI Labs).

(This chart doesn’t include DeepMind, which would increase Google’s number even more.)

In other words, Google rewards the same qualities in both models and employees: locally measurable output, procedural compliance, and organizational legibility. This is where Google’s next battle will be fought.

Conclusion: What’s Next For Google.

When Napoleon met the Third Coalition at Austerlitz, he faced a larger army with experienced soldiers and far more artillery. He did not win because every French officer stopped thinking and waited for instructions. His corps could move independently, improvise, and fight on their own — but they were all moving towards the same objective. The coalition’s commanders were coordinating different armies, different priorities, and different chains of command. This meant that the Napoleonic Corps could all take the best actions for themselves individually, all while being aligned on the larger vision (while the Coalition was always paralyzed by multiple chains of command).

That is the real test for Google. Bringing DeepMind, Cloud, and the product teams together only matters if it creates one direction without requiring everyone to think the same way. Too little alignment, and Google returns to the turf wars that got it here; too much control, and its best builders either stop taking risks or leave. If it gives Cloud, DeepMind, and Google’s builders a shared direction while preserving their freedom to act, the company may finally make its extraordinary advantages compound.

In other words, the question is whether Google becomes one army — or merely one larger committee. Personally, I’m optimistic based on my internal conversations with the company.

Thank you for being here, and I hope you have a wonderful day,

Dev ❤

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction.

Help me buy chocolate milk

PS- We follow a “pay what you can” model, which allows you to support within your means. Check out this post for more details and to find a plan that works for you.

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

This article was originally published here. Sign up for free to never miss out and join 200K weekly readers getting the most important insights in AI delivered to their inbox.

Thousands of engineering teams are burning margin, forcing heavy frontier models to handle basic production tasks because smaller, cheaper open-weight alternatives can’t deliver reliable outputs. And on the surface, they’re completely right.

A model like Qwen3–4B — one of the best small open-weight architectures on the market — gets basic arithmetic wrong almost all the time, landing a correct answer just 32% of the time. With odds like that, you might as well hit the roulette and gamble like a proper degenerate.

But if you dig into the failure data, an interesting stat emerges — it actually computes the correct answer inside its internal states an impressive 80% of the time. Somehow, the model gets to the correct answer and then veers off track. This tells us that the issue isn’t in the base intelligence (the model has the knowledge somewhere), but in navigation (discovering the knowledge reliably and then stopping). In fact, when we tracked whether models finished their responses naturally versus hitting the token limit, every single naturally completed answer was correct (across 500 generations). Wrong answers are destinations that our model failed to reach, not destinations that our model lacked in its map (full EOS analysis in the appendix).

This means that models get trapped in an optimization loop — formatting, restructuring, and elaborating indefinitely — until they brick the run before ever stating the final result. What we need is a system that reliably find and unlocks the knowledge already hidden in the language model.

Last time I wrote about Latent Space Reasoning, the system we broke down required a pretty sophisticated setup requiring trained judges, specialized aggregation, and a universal embedding projected space. Since then, our team at Irys has been exploring simpler ways to unlock the hidden knowledge in models that bring the power of latent space reasoning to less technical teams. And our results have been very compelling —

• Injecting just two random vectors into the model’s embedding space before it starts generating forces it to break the lock-in, spiking that 32% success rate straight to 51.6%. Re-read that number — we almost doubled accuracy by adding 2 random tokens.
• Bump that to ten random vectors with a basic plurality vote, and the success rate hits 72%. (Use plurality, not majority voting — — majority actually performs worse than baseline when individual seed accuracy is below 50%. Details in the appendix.)
• At ten vectors, every single task in the benchmark gets solved by at least one seed — 100% oracle coverage extracted purely from random noise.
• On a Redis debugging task, the baseline model spit out 14 incoherent words. With two random tokens, every single seed produced a complete 650-word diagnostic plan.
• On 12 legal reasoning tasks, the top seed beat the baseline on 11 of them.
• Smaller models with perturbation beat or matched larger models of the same family more than half the time, proving that this technique unlocks a new axis of capabilities w/o needing increased hardware or training investment.

These results allow us to get better performance from smaller models than from bigger models AND for lower cost (small model + multiple runs is much cheaper than big models).

^^Experiment details on GitHub. Once again, our system unlocked performance that parameter scaling did not.

Given how impactful our findings are for democratizing intelligence for everyone, we’ve open-sourced what we could on our latent space reasoning repo — github.com/dl1683/Latent-Space-Reasoning. While the effects of our research are statistically significant (McNemar p < 0.001 across two separate models), we recognize how small our sample size is. This is where we’d love to collaborate with the rest of our open community to scale up the experiments and look into different outcomes.

Executive Highlights (tl;dr of the article)

The finding: A Qwen3–4B model computes correct arithmetic answers 80% of the time but only states them 32% of the time. The model isn’t failing to reason — it’s failing to finish. It gets trapped formatting answers it already computed.

The fix: Injecting 2 random vectors into the model’s embedding space before generation breaks the lock-in. No training, no fine-tuning, one line of code. Accuracy jumps from 32% to 51.6% on a single seed, 72% with plurality voting across 10 seeds, and 100% oracle coverage — every task solved by at least one seed.

It generalizes. On a Redis debugging task, baseline output was 14 incoherent words. With 2 random tokens, every seed produced a 650-word diagnostic plan. On 12 legal reasoning tasks, the best seed beat baseline on 11. On incident response, evolved vectors surfaced honeypot deployment, MITRE ATT&CK analysis, and HSM credential rotation from a 4B model that baseline could only get “rotate credentials, check logs” out of.

Direction doesn’t matter. Random noise and carefully optimized projections produce identical results (Mann-Whitney p = 1.000). The perturbation provides energy, not information. Leading candidate explanation: stochastic resonance. Three independent groups (Kim et al., Shi et al., Pfau et al.) reached the same conclusion through different methods.

Small models + perturbation beat bigger models. Scaling from 4B to 14B bought 4 percentage points on our arithmetic benchmark. Changing how we interrogated the 4B model bought 40. The 4B runs on a $429 GPU at 200 tokens/sec. The 14B needs a $1,999 GPU at half the speed. At cloud rates, 10 perturbation seeds cost $0.009/query versus $0.45 for GPT-5.6 Sol thinking tokens — 56x cheaper.

Practical guardrails. Use 8-bit quantization, not 4-bit (4-bit nearly eliminates the effect). Use plurality voting, not majority (majority performs worse than baseline). If your model already scores 75%+ on a task, perturbation can hurt mean accuracy — it helps stuck models, not cruising ones.

What doesn’t work. Our original verbosity claim was a code bug. The scorer is barely trained (broken on 9/12 legal tasks). DeepSeek showed a negative mean effect at high baseline. The data breach task exposed hallucination when the model lacked the knowledge entirely. Sample sizes are modest: 25 arithmetic, 5 planning, 12 legal tasks.

What’s next. Temperature comparison (the test that determines if this is a new technique or a mechanistic insight), gated attention probe, position-shift ablation, scorer improvement. Full pipeline preregistered and open-source.

The thesis: The industry’s default answer to “my model isn’t good enough” is a bigger model. Our results say that for a meaningful class of tasks, the capability was already in the small model — the bottleneck was the inference path. Lifting the ceiling gets the headlines. Raising the floor is where the new value gets created.

Everything is open-source: github.com/dl1683/Latent-Space-Reasoning

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Want access to a repository containing all of our research? 300+ files containing our notes of various experiments, discussions with cutting-edge teams, and insights into where the industry is headed next. Get a Founding Member Subscription to AI Made Simple. Want to talk to me for details/get my insights into the tech ecosystem? Reach out to me through any of my socials over here or reply to this email.

Every month, the Chocolate Milk Cult reaches over a million Builders, Investors, Policy Makers, Leaders, and more. If you’d like to meet other members of our community, please fill out this contact form here (I will never sell your data nor will I make intros w/o your explicit permission)- https://forms.gle/Pi1pGLuS1FmzXoLr6

How Standard Formatting Might Be Making Small Models Fail

The standard narrative (small open-weight model hallucinations/mistake are a result of their lower intelligence) is true quite often, but as we’ve derstand why, let’s quickly remind ourselves of what models are doing when they do inference.

Language models generate text autoregressively — one token at a time. Because each token becomes part of the hard context window for every subsequent decision, the first token constrains the second, and the first two dictate the third. In our research across multiple models and sizes across multiple domains, we found that the first 20 generated tokens tend to disproportionately impact the AI context generation (even in very long context generations; research shared in our comprehensive proprietary AI research map available to all founding members of the Chocolate Milk Cult over here). This leads to the “autoregressive lock-in” phenomenon we have discussed extensively in the past.

Image Source

In practice, this will manifest in several ways:

1. Models will spend tokens on the formatting and run out of their token budget, even when they know the answer.
2. Models might follow a reasoning path, realize it’s no good (either themselves or by your injection), but the false path will STILL influence the outputs.
3. The models you pay for by token are occasionally setting their token budgets on fire to format answers they’ve already computed. We can call this structural failure formal presentation mode. Left to its own devices, the model burns through its footprint producing immaculate headers, enumerated steps, and clean LaTeX expressions. In other words, it commits to a visual template and mindlessly fills it in — performing the aesthetic of math rather than actually doing the math (anyone getting shades of pulling up a terminal in front non coders to seem “hackery”).

Knowing that, we decided to compute the geometry of the model's search space (nerd stuff, not the most relevant here; check next section, GitHub experiments or gimme a shout if you want deets) and then used that geometry to disrupt that default trajectory by prepending a couple of random, noisy tokens to the embedding space. This pushed the models into exploratory computation mode — a messy, stream-of-consciousness scratchpad.

Look at the text execution on the arithmetic problem (45 + 23) * 17 - 89:

Without prefix (Wrong — bricks on the formatting template):

Let me solve this step by step.
Step 1: Calculate 45 + 23 = 68
Step 2: Multiply by 17: 68 x 17 = ...

With prefix (Correct — hits the exact payload in half the footprint):

ok so first 45+23 thats 68, then times 17... 68*10=680, 68*7=476, so 680+476=1156, minus 89, 1156-89=1067

The random prefix doesn't inject any new knowledge into the models (it can’t). Instead, it seems to bypass a lot of the hacking behaviors injected from the alignment process and forces the system to compute directly.

Ever since Wei et al. dropped their foundational Chain-of-Thought paper, the consensus has been that intermediate steps boost accuracy. But the ecosystem completely missed the real bottleneck: chain of thought worked best when the ideas occur in overlapping knowledge regions—

This proved that intelligence was a “search through the latent space” issue (in hindsight, this paper had much bigger implications than any of us realized). It also gives us an interesting framing for the ubiquity of the formal presentation mode: the alignment hacking present in modern post-training pushes form over substance. The random token perturbation allows us to attack this gap by undercutting the formatted alignment spaces and leads to a system that wastes 800 tokens on beautifully formatted, incomplete garbage (base LLM), losing to a model that spends 400 tokens on a messy, complete solution (our perturbed variant).

Take one second to appreciate how insane this is. The most common engineering band-aids either operate way too late after the trajectory is already committed (like temperature sampling or best-of-N), require heavy gradient updates (fine-tuning), or try to extract diversity during token selection rather than the computation step (self-consistency). We avoid all of that by directly attacking the root of this issue, and are able to get significantly better results out of this with a solution that’s practically free.

In simple words, change where the model starts computing and how it searches a predefined space, and you can dramatically update its results. So how exactly does that work? We will look at that next.

What If You Shift Where the Model Starts Computing?

In the vein of data augmentation techniques like RandAugment, we take two vectors of random numbers, scale them to match the model’s native embedding magnitude (an RMS of ~0.022 for Qwen3–4B), and prepend them as positions 0 and 1 before your prompt tokens. The intuition here was similar to what made RA so effective — the randomness should inject a level of diversity into your search space, ensuring better performance.

Our thrifty twosome now sits at the start of every model call, shifting the computational trajectory through every subsequent layer from the very first real token onward. Apparently, this randomness is all you need to start unlocking model performance. We tested this approach against a simple, uncalibrated version of the latent space reasoning pipeline we had shared earlier:

• Optimized projections vs. Random noise: Mann-Whitney p = 1.000. The distributions are identical.
• Euclidean vs. Hyperbolic mutations: p = 1.000. Same result.

In other words, this setup provided the same level of geometric exploration as a more complicated exploration framework. This leads to our next interesting question — why would random tokens make LLMs better?

Why Pure Noise Unlocks AI Model Reasoning

Truth be told, no fucking idea. That’s one of the reasons I’m open-sourcing this, so that one of you can figure this out while I take all the credit.

Until that happens, I’ve had some researchers from Irys and the larger Chocolate Milk Cult looking into this. Our leading candidate explanation is stochastic resonance — the physical phenomenon where adding noise to a non-linear system amplifies weak signals that would otherwise fall below an activation threshold. This is an idea we explored over here when looking into why Brains are so much more efficient than LLMs by using Sub-Landauer patterns (patterns that encode some useful information, but are not given “full energy” so they only activate when a bunch of Sub-Landauer patterns trigger together to collectively overcome the energy threshold) —

Mapping this to NNs, standard transformers are like a non-linear threshold system: the correct reasoning path exists in the model’s parameters but sits just below the greedy selection boundary. Random embedding noise provides the activation energy needed to push the computation over that threshold. Because the noise provides energy rather than information, optimizing its direction adds zero value (which is why directional perturbations without tweaking magnitudes didn’t present much utility).

We are currently running experiments to isolate stochastic resonance from competing accounts (attention redistribution, positional offsets from shifted RoPE IDs, and attention-sink disruption). If you want in, just pull up and start running some experiments yourself.

In the meantime, we have some reason to be confident in our assessment of the situation since independent teams are reaching the exact same conclusion across different setups:

• Kim et al. (“Random Soft Prompts”): Showed random, unoptimized embedding vectors match the accuracy of heavily trained soft prompts on math tasks.
• Shi et al. (“Meaningless Tokens”): Found inserting sequence strings of meaningless discrete tokens consistently boosts reasoning via activation redistribution.
• Pfau et al. (“Let Me Speak Freely”): Demonstrated that simple repeated filler tokens (“…”) allow hidden computation in the residual stream.

All 3 seem to be pointing at the same thesis — the perturbation matters much more than specific tuning. At least until we have a more rigorous map of the latent space itself.

Why Perturbation Changes the Unit Economics of LLM Reasoning

Why go through this effort? For the vibes? Streetcred? To find some way to spend my Codex tokens (unfortunately, that’s not true for me since I don’t get paid to tokenmax)?

Nah, it’s coz I’m a gareeb, and I need to find ways to make AI cheaper. A lot cheaper so that Irys company lunches can go from two packs of Oreo shared b/w all of us, to at least 4 Chipotle bowls for the whole team. So watch your dearest Dev Dev fight his way to put more meat in his mouth.

Turns out that even accounting for the 10 extra inference passes of the smaller model (+ the oracle), latent-space reasoning extracts far more capability per dollar without ever forcing you onto a bigger model. This changes your business in two meaningful ways —

1. You can route more work to smaller models safely and get the same threshold of performance. This opens up a whole new class of latency-sensitive work that couldn’t be done before, while also driving down the costs of the existing agentic loops where you might be worried about costs.
2. The magnitude of the performance shifts means that you can deploy the same level of intelligence in a much cheaper class of hardware.

Both will massively speed up the Jevon’s Paradoxification of AI, ultimately massively upping the value generated by AI.

Following are some rough markers of short-medium ROI I put together across various scenarios (image is a summary, we’ll break them down)—

Raw Performance

On our arithmetic benchmark:

• Qwen3–4B scored 32% under ordinary greedy inference.
• Prepending a two-token perturbation raised that to 51.6%.
• Ten perturbed trajectories with plurality voting hit 72%.
• Qwen3–14B — 3.5 times the parameters — scored 36% (the small delta b/w 4B base and 14B isn’t surprising here since the arithmetic isn’t going to be novel for even the 4B model).

In other words, scaling the model bought four percentage points. Changing how we interrogated the small model bought forty. The 14B model isn’t even competing here. This becomes much more impactful when you consider the macroeconomics and the memory supercycle (companies like Micron and Nvidia are committing highway robbery b/c of how expensive High Bandwidth Memory has become in GPUs) and how LSR (even single pass with perturbation) gets us higher gains than scaling, w/o increasing the hardware requirements (the implications of this will be made clear in the next sections).

The Hardware Class Gap

Model size dictates the minimum hardware required to run the system. From Qwen’s official AWQ benchmarks:

• Qwen3–4B: 2.9 GB VRAM, 199.7 tokens/sec
• Qwen3–14B: 10.0 GB VRAM, 96.5 tokens/sec
• Qwen3–32B: 19.1 GB VRAM, 47.7 tokens/sec

The 4B model sits in a completely different deployment class, meaning that we get cheaper cards, more room for context, and double the throughput of 14B. How much cheaper? Take a look at the difference:

• RTX 5060 Ti (16 GB): $429 at launch
• RTX 5090 (32 GB): $1,999 at launch — before the rest of the rig

On Runpod’s current serverless rates, a 16 GB worker runs $0.58/hour versus $1.58/hour for the 32 GB tier. That is a 2.7x hourly premium for a larger model that generates at half the speed.

Factoring in current cloud rates, power draw, and the May 2026 US residential electricity baseline (18.44 cents per kWh):

• $1,000 local machine (16 GB): pays for itself after ~1,830 GPU-hours
• $3,250 local machine (32 GB): pays for itself after ~2,205 GPU-hours

At 240 GPU-hours per month, you are looking at your investments paying back more than a month earlier —

Worth stressing here is that the lower upfront hardware costs enable market creation since the costs lower the barrier for participation.

Unit Economics vs. Frontier Models

Let’s now look at the calculations from a cloud costing perspective. Using standard market baselines (Qwen3–4B at 4-bit, 2,048 token sequence, continuous prefix noise at positions 0 and 1):

• 10 Perturbation Seeds ($0.008 per query): 10 parallel generation passes = 20,480 tokens at $0.40/M. The 2 prefix vectors add 0.097% overhead — pure rounding error.
• 10 Seeds + Scorer MLP ($0.009 per query): A 300K-parameter MLP evaluates candidate trajectories in one pass on short sequence embeddings, adding ~$0.001.
• GPT-5.6 Sol Thinking Tokens ($0.45 per query): OpenAI’s flagship bills thinking tokens at the output rate ($30.00/M). Mid-range queries burn ~15,000 thinking tokens before answering ($30.00 × 0.015).

Computing the difference, we get Sol being 56x more expensive than our system.

Take a second to consider what that means at scale. At 10,000 queries per day, that is $90/day versus $4,500/day. Over a month, you are comparing $2,700 to $135,000. At 50,000 daily queries, the perturbation approach stays under $15,000/month while Sol crosses $675,000.

Now, this is not meant to say that our system will replace Sol/big models everywhere. Ultimately, no matter how much we might wish, latent space reasoning can’t make existing models more knowledgeable. So in many tasks, a heavier model will simply outperform a lighter model, no question. Similarly, if your model already scores well on a task (75%+), perturbation can actually hurt mean accuracy since it disrupts trajectories that were already efficient. The oracle still improves performance, but the average gets pulled down by bad perturbations.

However, this also leaves a large gulf of techniques where the simpler models have the base knowledge, but they get stuck in unhelpful attention sinks they can’t avoid. And your constraints limit you from hitting bigger models. In that case, we believe that Latent Space Reasoning with Perturbations can help your model punch above it’s weightclasses.

Conclusion: The Overlooked AI Revolution

The market is currently pricing in only one economic revolution: pushing the capability ceiling. Frontier labs are burning billions to maximize peak capacity. As economic pressures mount, this will lead to higher prices for consumers as the labs have to justify their massive capex buildouts. We’re already seeing this manifest with the higher token prices in recent models (both directly in PPM and indirectly with models consuming more tokens for their tasks).

Take a second to consider that there is a second revolution: raising the floor. The smartphone didn’t win by being a faster laptop. It won by delivering a fraction of a laptop’s compute at a hundredth of the cost, completely changing the deployment envelope. That cost collapse birthed entirely new economies — ride-hailing, mobile logistics, the app ecosystem — because adequate compute became cheap enough to integrate a billion new participants.

Delivering adequate intelligence at a radically lower cost changes who gets to participate in the system. It makes problems economically tractable that simply never pencil out at frontier pricing. You cannot integrate the broader economy when operators are bleeding out on $135,000/month inference bills.

APPENDIX: Full Experimental Detail

A1. Arithmetic: Dose-Response, Controls, and Voting

Setup. Prefix-length sweep from 0 to 8 tokens. Qwen3–4B at 4-bit quantization. 25 multi-step arithmetic tasks (nested expressions, 3–6 sequential operations).

Dose-response. Baseline (0 tokens): 32.0%. 1 token: 42.7% (+10.7pp, n=3 seeds). 2 tokens: 51.6% (+19.6pp, n=10) — — the peak. 3 tokens: 44.0% (+12.0pp, n=10). 8 tokens: 44.4% (+12.4pp, n=10). McNemar p = 0.000015 at 2 tokens. Performance peaks at 2 and drops at 3 and 8 because of token budget competition: correct runs average 718 tokens, wrong runs hit the 1,024 ceiling and get truncated. At 8 prefix tokens, the model branches into multiple partial strategies and exhausts its budget before concluding any of them.

Controls. Zero embeddings (adding positions only): 36.0%, +4pp. Mean embedding (identical values at both positions): 36.0%, +4pp. Random noise (diverse values): 51.6%, +19.6pp. No chain-of-thought: +0pp. The effect requires both diverse prefix values and step-by-step reasoning. Remove either and it vanishes.

Voting at n=10 seeds. Plurality (most common answer): 72% on 4B, 56% on 8B. Oracle (best of 10): 100% on 4B, 80% on 8B. Majority (>50% agreement): 40% on 4B, 12% on 8B — worse than the 16% 8B baseline. Majority fails because when individual accuracy is below 50%, the threshold demands agreement that doesn’t exist among mostly-wrong seeds. Correct answers cluster (different trajectories converge on the same right answer). Wrong answers scatter (each truncation produces a different spurious value).

Caveat. The n=3 scout runs overestimated effects: Qwen3–4B dropped from 60% (n=3) to 51.6% (n=10). 25 tasks establish significance but not precise effect sizes.

A2. EOS Completion

Setup. 250 perturbation responses on Qwen3–4B, 250 on Qwen3–8B. Tracked whether each response finished with a natural end-of-sequence token or hit the 1,024 max token limit.

4B results. Finished + correct: 94 (37.6%). Finished + wrong: 0 (0.0%). Truncated + correct: 35 (14.0%). Truncated + wrong: 121 (48.4%).

8B results. Finished + correct: 32 (12.8%). Finished + wrong: 0 (0.0%). Truncated + correct: 40 (16.0%). Truncated + wrong: 178 (71.2%).

P(correct | finished naturally) = 1.000 on both models. Zero exceptions across 500 responses. Same pattern holds for baseline greedy decoding on this benchmark. EOS responses average 718 tokens. Truncated responses hit 1,024. Perturbation increases the EOS rate: 37.6% vs 24% at baseline. Tested on 25 arithmetic tasks only.

A3. Planning Tasks

Setup. 5 tasks: fraud detection system design, incident response, healthcare data platform, Redis cache debugging, Oracle-to-PostgreSQL migration. 3-way comparison: greedy baseline, random perturbation (5 seeds), evolved latent vectors (5 seeds). Qwen3–4B at 4-bit, 2,048 max tokens, greedy decoding. LLM judge scored on coherence, correctness, completeness, specificity, actionability.

Attention sink rescue. On Redis debugging, baseline collapsed to 14 incoherent words — — stopped generating immediately in a degenerate attention pattern. Every perturbation seed produced 650–710 word complete diagnostic plans. Likely mechanism: attention sinks (Xiao et al.) cause earliest token positions to accumulate disproportionate attention; under greedy decoding, degenerate early-position patterns propagate through every subsequent token. Random noise at those positions disrupts the pattern before it forms.

Evolutionary search. A 300K-parameter MLP scorer evaluates candidate latent vectors and guides an evolutionary population. On incident response, baseline produced “rotate credentials, check logs.” Evolved vectors produced honeypot deployment, MITRE ATT&CK lateral movement tracking, tiered HSM credential rotation, and immutable container rebuilds from verified base images.

LLM judge tally. Perturbation won 3/5, evolution won 2/5, baseline won 0/5. Sample size: 5 tasks.

A4. Legal Reasoning

Setup. 12 tasks across 5 categories: FTC unfairness, GDPR classification, disparate impact, SaaS contract review, acquisition due diligence, data breach triage, IP risk portfolio, negotiation leverage, regulatory response, contractor misclassification, corporate veil piercing, whistleblower retaliation. Same 3-way comparison. Qwen3–4B at 4-bit, 2,048 max tokens. Blind evaluation — condition labels stripped, order randomized, scored on Legal Accuracy, Analytical Depth, Practical Utility, Structural Quality, Completeness (1–10 each).

Oracle results. Best of 5 seeds beat baseline on 11/12 tasks (92%). Average lift: +1.6 points on a 10-point scale. Largest lifts: negotiation leverage 2.0 → 5.4 (+3.4), contractor misclassification 2.2 → 5.6 (+3.4), IP risk 3.6 → 6.4 (+2.8), whistleblower 3.2 → 5.6 (+2.4), FTC unfairness 5.2 → 7.2 (+2.0).

Qualitative examples. Contractor misclassification: baseline gave surface analysis. Best perturbation seed applied state-specific tests (California ABC test, New York economic reality test, Texas common law) with per-jurisdiction liability estimates. Corporate veil piercing: one seed identified “SubCorp appears to be a real operating business, not just a sham shell” — a Delaware Chancery Court doctrine element other outputs missed.

Boundary condition. GDPR controller/processor classification (the 1 loss): exceeded the model’s knowledge. Perturbation seeds consumed full 2,048-token budgets in thinking loops without substantive analysis.

Mean vs oracle gap. Most random seeds produce lateral moves or slight regressions. The gap between 92% oracle and mixed mean results is the scorer problem. Sample size: 12 tasks.

A5. Cross-Model and Quantization

Cross-model results on the 25-task arithmetic benchmark:

Qwen3–4B at 4-bit: 32.0% → 51.6% (+19.6pp), 100% oracle. 80% answer-anywhere at baseline, 32% stated. Perturbation barely changes answer-anywhere (80% → 82%) but jumps stated accuracy. Convergence aid — helps finish, not compute.

Qwen3–8B at 8-bit: 16.0% → 28.8% (+12.8pp, p = 0.000177), 80% oracle. 32% answer-anywhere at baseline, raised to 50%. Exploration aid — finds answers the model couldn’t compute before.

DeepSeek-R1-Distill-Qwen-1.5B at 4-bit: 76.0% → 74.4% (-1.6pp), 100% oracle. Near ceiling already. Perturbation disrupts efficient trajectories more often than it finds better ones.

phi-2 unquantized: 12.0% → 18.7% (+6.7pp), 28% oracle. Low ceiling limits gains.

Quantization. Qwen3–8B at 4-bit: +1.3pp (null result). Same model at 8-bit: +12.8pp (p = 0.000177). Working hypothesis: 4-bit rounds to 16 distinct weight values, too few for perturbation to create meaningfully different trajectories. 8-bit (256 values) preserves enough diversity. Based on one model comparison — may not generalize.

A6. Limitations, Failures, and Next Experiments

Verbosity bug. First analysis claimed perturbation increases verbosity. Artifact of a field-ordering bug (commit 6c69284). Perturbation actually produces 3% fewer words on Qwen3–4B.

Data breach hallucination. All 11 outputs (baseline, perturbation, evolution) fabricated legal content — invented statutes, fictional agencies, incorrect deadlines. Task exceeded the model’s knowledge.

Broken scorer. 300K-parameter MLP was non-deterministic on 9/12 legal tasks. Fix applied (deterministic projection with fixed seed), needs clean re-run. The 48pp gap between 100% oracle and 51.6% mean accuracy on arithmetic is the scorer’s failure to select the correct seed.

Sample sizes. 25 arithmetic, 5 planning, 12 legal. Statistically significant (McNemar p < 0.001 on two models) but effect sizes will shift with scale.

Next experiments (preregistered Phase A pipeline):

Temperature comparison. Head-to-head: prefix perturbation vs temperature sampling vs prompt rephrasing, same tasks/models/compute budgets. If temperature at n=10 matches perturbation’s oracle and mean, the contribution is mechanistic insight rather than a new technique. If embedding-level perturbation accesses trajectory classes token-level sampling can’t, the technique claim holds.

Gated attention probe. Qwen3.5–4B uses hybrid gated DeltaNet + gated attention layers that eliminate attention sinks via post-softmax sigmoid gating. If perturbation disappears on gated architectures, the effect depends on attention sinks specifically. If it survives, the mechanism operates below the attention layer.

Position-shift ablation. Soft prefixes shift RoPE position IDs for downstream tokens. Decomposes the total effect into: position offset, embedding diversity, and their interaction.

Scorer improvement. Domain-specific training, more sophisticated architectures, learned aggregation. The unresolved problem is selecting the correct candidate from among seeds.

Thank you for being here, and I hope you have a wonderful day,

Dev ❤

It takes time to create work that’s clear, independent, and genuinely useful. If you’ve found value in this newsletter, consider becoming a paid subscriber. It helps me dive deeper into research, reach more people, stay free from ads/hidden agendas, and supports my crippling chocolate milk addiction.

Help me buy chocolate milk

PS- We follow a “pay what you can” model, which allows you to support within your means. Check out this post for more details and to find a plan that works for you.

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

Check out my other articles on Medium. : https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819

For decades, progress in the tech industry has been framed as a compute problem.

Software systems must constantly track process states. A network connection is listening, handshaking, established, or closing. A checkout flow collects shipping, takes payment, or finishes. A regex matcher tracks whether it is before the first character, inside a repeating section, or finished.

All share a core skeleton: a finite set of situations and a finite set of events that transition between them. A finite state machine (FSM) is that skeleton. An FSM replaces scattered boolean flags with a single legal state and one explicit transition rule per event.

Instead of spreading behavior across nested if-else chains, boolean flags, and callbacks, an FSM centralizes every state and legal transition. You can audit it as a table: from state X, on event Y, go to state Z. If a transition is absent, the move is illegal. This prevents bugs where a system enters an impossible combination of flags.

FSMs can be quite weak since they are memoryless (they only remember the current state), but when the task fits inside fixed memory requirements, they can be hard to beat.

Let’s study how Finite State Machines work in more detail.

This article is part of our open source effort to share the fundamentals of computer science to everyone. We have learning paths, searchable topics, and are tryign to always add more topics/improve the writeups on existing pages. Join our effort by contributing to the Github of this project over here.

Do you want to support our mission of providing high quality educational material and work opportunities at the cutting edge of tech to everyone, irrespective of their socio-economic background? Please consider subscribing to AI

Help me buy chocolate milk

We run on a “pay what you can” model — so if you believe in our mission of open soucring , there’s likely a plan that fits (over here).

PS — Supporting this work doesn’t have to come out of your pocket. If you read this as part of your professional development, you can use this email template to request reimbursement for your subscription.

Why This Exists

Software systems constantly track where they are. A socket is listening, handshaking, or closed. A checkout flow collects shipping, takes payment, or finishes all share the exact same skeleton: a finite set of states, and explicit events that move you between them. A finite state machine (FSM) is a great way to represent that skeleton since an FSM replaces scattered boolean flags with a single legal state and one explicit transition rule per event. This gives you tremendous visibility into your system since you no longer need to debug nested if-else chains. The structure of the transition table formed allows you to check if a move is legal or not. While they do give up computing power, their efficiency makes them perfect for memoryless systems.

The Obvious Approach

The default instinct for state tracking is imperative control flow. You declare a few flags like isOpen or isAuthenticated, and then you scatter if-checks across your handlers. For two flags, this works perfectly fine. Scale that to five flags, and you suddenly have 32 possible combinations. Most of those states are completely illegal, and nobody on your team can actually enumerate which ones are reachable.

For pattern matching specifically, the obvious alternative is backtracking — trying a path and rewinding if it fails. That feels flexible right up until a carefully crafted input forces exponential rewinds and crashes your regex engine. FSMs reject this entirely. Instead of guessing which combination of booleans describes the system, you use a single variable drawn from a known, finite set.

Six boolean flags create 64 combinations you have to verify, and adding just one more doubles your entire state space. Bugs love to hide in these untested combinations, eventually surfacing as a connection that is somehow simultaneously open and closed.

Backtracking hits a much harder, physical wall. If you run a naive backtracking engine on the pattern (a|a)*b against a string of 25 as without a trailing b, the engine explores over 33 million paths before it finally gives up. It can’t fail early because it doesn’t actually understand what the pattern requires.

Both of these approaches fail because they defer the hard work of understanding the structure until runtime. FSMs front-load that work by compiling the structure into a graph once, turning execution into a predictable walk.

The Core Insight of Finite State Machines

A deterministic finite automaton (DFA) guarantees exactly one next state for every combination of your current state and the incoming symbol. The transition table effectively becomes the entire program. Running it just means doing one table lookup per character: you find the row for your state, the column for your character, and read the next state. There’s no stack and no recursion.

A DFA diagram makes the one-state-at-a-time rule visible through labeled transitions. Source: Wikimedia Commons, commons.wikimedia.org/wiki/File:DFAexample.svg.

The actual power here is compression. Look at the pattern ab*c. State S1 doesn’t care how many bs it just saw. It only remembers that it caught the leading a and is allowed to consume more bs before the final c. The machine discards the exact count because it’s completely irrelevant to the match.

The trap state R takes this compression even further. Once a prefix goes bad, no future suffix can possibly save it. Instead of tracking every single bad prefix, the finite state machine just drops them all into one permanent rejecting state.

How Finite State Machines Work

A DFA requires exactly five things: an alphabet of valid symbols, a finite set of states, a starting state, a set of accepting states, and a transition function that maps every state-symbol pair to a destination.

In our ab*c example over the alphabet {a, b, c}, S0 is the start. Reading an a moves you to S1. Once you’re at S1, reading a b simply loops back to S1. Reading a c pushes you to S2, which is our only accepting state. Any character that violates this exact sequence drops you into the trap state R, where you stay forever.

Feed the machine abbc. You move from S0 to S1 on a, loop at S1 twice for the bs, and jump to S2 on c. The input runs out exactly at an accepting state, so it’s a match. Feed it abba, and it reaches S1, reads the final a, finds no valid transition, and drops to R to reject.

Why Finite State Machines Work

Finite state machines work when your system’s future behavior depends strictly on a finite summary of the past. For ab*c, you only care about four situations: waiting for a (S0), inside the b region (S1), finished (S2), or dead (R). Making finer distinctions won’t change the outcome.

This concept is called the Myhill-Nerode equivalence. Two histories are mathematically equivalent if they yield the exact same accept or reject outcome for every single possible continuation. If the number of equivalence classes is finite, you can build a DFA. If it’s infinite, you can’t. It’s the exact same reason a lexer doesn’t need to memorize every digit of a number literal to know if the next character extends it. It only needs to know it’s currently in an inNumber state.

Cost and Complexity of Finite State Machines

Running a DFA on an input of length n costs O(n) time. Every single character triggers exactly one transition lookup. If you store that table as a 2D array, lookups are O(1). You get zero backtracking and entirely eliminate branching based on input content.

The memory footprint scales as O(S * A), where S is your state count and A is the alphabet size. A byte-oriented scanner looking at 256 possible inputs across 50 states needs a 12,800-entry table. At a byte per entry, that’s roughly 12.5 KB, which sits very comfortably inside an L1 cache.

The real tax here is the construction phase. Compiling a nondeterministic finite automaton (NFA) into a DFA via subset construction can theoretically blow up to 2^n states. While that worst-case scenario is rare, you still have to measure the compiled finite state machine’s size rather than the raw regex length to make sure it fits your hardware constraints.

Real-World Uses of Finite State Machines

Production regex engines like RE2 and Rust’s regex crate compile patterns down to automata specifically to guarantee linear-time matching. When your text editor colors your syntax, it’s running a lexer DFA. The entire TCP connection lifecycle—moving from LISTEN to SYN_SENT to ESTABLISHED—is just an explicit state machine defined in RFC 793. UI libraries like XState use this model for checkout flows to ensure that illegal screens are mathematically unrepresentable. Game AI uses them to toggle between patrol and chase routines. Zoom out to the hardware level, and every digital circuit from a traffic light to a USB controller is fundamentally a finite state machine.

Where Finite State Machines Fail

Finite state machines fall apart when they need to count indefinitely. Matching balanced parentheses like ((())) requires you to track an unbounded nesting depth. Because a DFA with k states can only ever count up to k, unbounded nesting demands a pushdown automaton—an FSM paired with a stack. This is exactly why you can’t parse HTML with regular expressions.

Finite state machines also lose their value when you try to hide them. If your codebase relies on scattered booleans implicitly managing state transitions behind the scenes, you throw away the visibility benefits entirely.

Finally, using a single trap state can be too blunt. In network protocols, lumping recoverable errors, timeouts, and permanent failures into one dead state completely discards the context you need to safely resynchronize.

Finite State Machines: A Worked Example

Let’s build a finite state machine that accepts binary strings with an even number of 0s over the alphabet {0, 1}. You only need two states: E for an even count and O for an odd count. E serves as both the start state and the only accepting state.

From E, reading a 1 keeps you at E, while reading a 0 moves you to O. From O, a 1 keeps you at O, and a 0 moves you back to E.

That’s the entire machine: two states and four transitions. Test it against 1001. You start at E, read 1 and stay at E, read 0 and move to O, read another 0 to move back to E, and finally read 1 to stay at E. You end on an accepting state, which correctly flags the two 0s as a match.

Sources and study next

The canonical reference is Sipser’s Introduction to the Theory of Computation, chapters 1 and 2, which proves the equivalence of DFAs, NFAs, and regular expressions. Hopcroft, Motwani, and Ullman’s Introduction to Automata Theory covers DFA minimization and the Myhill-Nerode theorem. For practical regex engines, Cox’s “Regular Expression Matching Can Be Simple And Fast” (2007, swtch.com/~rsc/regexp/regexp1.html) explains why DFA-based engines avoid catastrophic backtracking.

On this site, study Trie (Prefix Tree) for character-labeled state graphs, Thompson NFA Regex Engine for nondeterministic execution, Subset Construction for compiling NFA state sets into DFA states, and CSV Parser State Machine for a small applied parser. For the broader computation hierarchy, compare with Pushdown Automata (FSM plus a stack) and Turing Machine (FSM plus an infinite tape).

Learning map

Before this topic, review Graph BFS, Trie (Prefix Tree).

This unlocks: Geofence Point-in-Polygon Index Case Study, KMP Prefix Function, Aho-Corasick Automaton, Finite-State Transducer Static Map, Distribution Feeder Outage Restoration Switching Case Study, zkVM Execution Trace AIR Case Study, E-Graphs & Equality Saturation, Pratt Parser Expression AST, TLA+ State-Space Model Checking Primer, Alloy Relational Model Finder Case Study, SMT Solver Theory Combination Case Study, Bytecode Stack Virtual Machine, Interpreter Dispatch Table & Threaded Code, Thompson NFA Regex Engine Case Study, Subset Construction: NFA to DFA, UTF-8 Decoder DFA Case Study, CSV Parser State Machine Case Study, JSON Parser Stack Case Study, Cellular Automata, Arithmetic & ANS Coding, DEFLATE Case Study, Video Codec Reference Frame DAG Case Study, AV1 Tile OBU Superblock Case Study, Log Template Drain Parser, UI State Machine Workflow, Circuit Breakers & Deadlines, HPACK Dynamic Table HTTP/2 Case Study, SGLang Compressed FSM Runtime Case Study, Kubernetes Node Pressure Eviction Signal Case Study, Kubernetes Job Indexed Completion Ledger Case Study, Kubernetes PV/PVC Storage Binding Case Study, Code World Models Case Study, Financial Contract Lifecycle Event Model, TLS 1.3 Handshake, Markov Chains & Steady States, Constrained Decoding, JSON Schema Constrained Decoding Token Mask, Adaptive Computation Time Halting, Nav2 Costmap Inflation Layer Case Study, Nav2 Behavior Tree Recovery Case Study.

Translate the mechanism into a reusable rule you can apply in later designs.

Use this page as a standalone study point, then follow its Next Up topics to stay on track.

Micro checks

• Can you explain one correct-by-construction rule for this operation in one sentence?
• Can you state what changes in O(1), O(log n), or O(n) terms?
• Can you point to one scenario where this operation is the wrong tool?
• Can you write a one-line example where this topic is the controlling bottleneck?

Try this now

Project prompt: Try the same sequence with one assumption removed and explain which proof step breaks.

Use the unlocks line as your completion check: if Finite State Machines can explain the transition to Geofence Point-in-Polygon Index Case Study, KMP Prefix Function, Aho-Corasick Automaton, you are ready to continue.

Thank you for being here, and I hope you have a wonderful day.

Dev ❤

I put a lot of work into writing this newsletter. To do so, I rely on you for support. If a few more people choose to become paid subscribers, the Chocolate Milk Cult can continue to provide high-quality and accessible education and opportunities to anyone who needs it. If you think this mission is worth contributing to, please consider a premium subscription. You can do so for less than the cost of a Netflix Subscription (pay what you want here).

Subscribe

If you liked this article and wish to share it, please refer to the following guidelines.

That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow. The best way to share testimonials is to share articles and tag me in your post so I can see/share it.

Share

I regularly share mini-updates on what I read on the Microblogging sites X(https://twitter.com/Machine01776819), Threads(https://www.threads.net/@iseethings404), and TikTok(https://www.tiktok.com/@devansh_ai_made_simple)- so follow me there if you’re interested in keeping up with my learnings.

Reach out to me

Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.

Small Snippets about Tech, AI and Machine Learning over here

AI Newsletter- https://artificialintelligencemadesimple.substack.com/

My grandma’s favorite Tech Newsletter- https://codinginterviewsmadesimple.substack.com/

My (imaginary) sister’s favorite MLOps Podcast-

https://machine-learning-made-simple.medium.com/

My YouTube: https://www.youtube.com/@ChocolateMilkCultLeader/

Reach out to me on LinkedIn. Let’s connect: https://www.linkedin.com/in/devansh-devansh-516004168/

My Instagram: https://www.instagram.com/iseethings404/

My Twitter: https://twitter.com/Machine01776819