---
title: "When Models Learn"
description: "Today's AI doesn't learn after it's trained. Test-time training changes that, \u0026 the tradeoff it creates, one model per user instead of one model for everyone, may be the more efficient architecture despite what it costs a GPU provider to serve."
categories: ["AI","Startups"]
keywords: ["test-time training","TTT layers","test-time compute","inference economics","scaling laws","AI infrastructure","reasoning models","GPS recalculation","per-user model serving cost"]
ai_summary: "Explains test-time training through the analogy of a GPS learning a persistent shortcut around daily traffic rather than a one-time reroute: the model takes a gradient step on the prompt it's answering, so its weights change as it works. Traces three implications, flat memory instead of a linearly growing KV-cache, the provider cost of serving a separate model per user, \u0026 faster inference, then states the tension as a tradeoff between serving long context and serving many people, \u0026 grounds it in concrete use cases, a coding agent that earns back its per-user cost over a long session versus a one-off query a shared frozen model handles just as well."
date: 2026-08-17
lastmod: 2026-08-17
canonical_url: https://www.tomtunguz.com/test-time-training-impact/
author: "Tomasz Tunguz"
---


Every model you've ever used froze the day its training ended. The answers are the same even if you have used it every day.

What if a model kept learning as you use it?

A GPS learns a persistent shortcut around daily traffic on northbound Highway 101, not just a one-time reroute. Test-time training does that to a model as it works.[^1] As you use the AI, the model changes its weights, changes how it thinks about its memories, to answer you better.

{{< email_image src="nhifmhc3i6bhnolzjaqm" alt="Two-part diagram. Top: standard inference, three colored users send queries into one shared gray frozen model with a lock icon, which answers all three from the same unchanged weights, one batched copy. Bottom: test-time training, three colored lanes each starting from the same gray base checkpoint, where a query arrives, hits the shared base model, then a highlighted gradient-step burst where the weights visibly shift, producing a differently colored updated model per user, then an answer drawn from that user's own updated weights." width="540" height="490" >}}

The changes are more profound than finding an off-ramp to an access road past a highway junction chokepoint.

Memory requirements plummet. A standard transformer keeps a KV-cache, a running record of every earlier token, so its memory grows linearly with context, every additional token adds to the running record. Test-time training folds that history into a fixed-size set of weights instead of a growing cache, so memory stays flat no matter how long the conversation runs.

The model provider now has to serve a separate model to each person. Once a model updates on your prompt, it is no longer the model that answered your neighbor's, so a single checkpoint serving millions of users becomes millions of slightly different models, each shaped by the person using it. That divergence is the provider's problem to solve: a GPU provider needs a copy in flight per user instead of one shared copy for everyone, which means more compute, more chips, to serve the same number of people.

It's much faster. Stanford research on small models indicates it can be up to 2.7 times faster, because a test-time trained model's inference latency stays constant no matter how long the context runs, the way a standard transformer's does not.[^12] In-Place TTT also ships drop-in, lifting a 4b model to competitive 128k-context performance with no retraining.[^3]

Here is the tension. Standard AI is limited by memory, test-time AI is limited by compute & chips, so a provider picks based on whether it's serving long context or serving many people.

That cost is only worth paying where personalization earns its keep. A coding agent that learns your codebase's conventions, the resilient persistent bugs, ultimately should provide some form of lock-in via memory, so the per-user cost pays for itself. A one-off customer support question doesn't need any of that. A shared, frozen, potentially fine-tuned model answers it just as well & costs the provider far less to serve.

Test-time training will be a key part of the discourse throughout the end of 2026 & beyond. It has the potential to change the current economics of AI.

[^1]: [Sun et al., Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://arxiv.org/abs/2407.04620)

[^3]: [In-Place Test-Time Training](https://arxiv.org/html/2604.06169)

[^12]: [End-to-End Test-Time Training for Long Context](https://arxiv.org/abs/2512.23675)