Stack

Scaling worked.

Every new architecture paper arrives with a small funeral for the transformer. The funeral keeps getting postponed.

In 2020, researchers at OpenAI measured power-law relationships between language-model loss and model size, data and training compute. DeepMind's Chinchilla work later showed that the allocation mattered: many large models had too many parameters for the amount of data used to train them. A 70-billion-parameter model trained on far more data beat much larger models at the same compute budget.

Engineers extended scaling to more dials.

The next wave is already spreading compute and learning across the full life of a system: training, reasoning at the moment of use, memory, tools, synthetic environments and feedback from results. The transformer may remain in the machine for a long time. It will carry less of the machine by itself.

Compute moves to the moment of use

A conventional language model spends most of its compute before the customer asks a question. At inference, its capabilities are largely fixed and a standard decoder follows one left-to-right path, calling the model repeatedly for the next token without branching or revising its earlier choices.

Reasoning systems change that budget. They generate candidate paths, search, check intermediate steps, call tools and spend more time on harder requests.

In a 2024 paper, Charlie Snell and colleagues found that an adaptive test-time compute strategy could outperform a model fourteen times larger on problems where the smaller model already had some chance of success, under a matched compute budget. The gain depended heavily on problem difficulty. Extra thinking was valuable when the task sat in the right zone. Easy tasks and hopeless tasks did not deserve the same budget.

DeepSeek-R1 offered another data point. Its R1-Zero model developed reasoning behaviour through large-scale reinforcement learning without supervised fine-tuning first. It also produced poor readability and mixed languages, so the full R1 used cold-start data and several training stages. DeepSeek then distilled reasoning behaviour into six smaller models.

For builders, inference cost becomes a product decision. A tax question, a code migration and a request to change a button colour should not receive the same number of reasoning steps. Products will need budgets for latency, search, verification and model calls, much as cloud systems already budget storage and bandwidth.

Data becomes designed, generated and checked

The open internet is a remarkable dataset. It is also finite, legally contested and increasingly filled with machine-written material.

A research forecast by Villalobos and colleagues estimated that, if earlier trends continued, training datasets could reach the size of the available stock of public human text sometime between 2026 and 2032. Treat the date range as a forecast. The pressure behind it is already useful: another indiscriminate scrape cannot remain the only data strategy.

Synthetic data can work very well when the system has a judge.

AlphaGeometry combined a neural language model with a symbolic deduction engine. Its creators generated 100 million synthetic examples without human demonstrations, and the deduction engine could verify the proofs. The system solved 25 of 30 olympiad geometry problems in the paper's test set. DeepSeek-R1 used rule-based rewards for domains such as mathematics and code, where answers can be checked.

The judge matters. A Nature paper on model collapse found that indiscriminate recursive training on generated material caused models to lose information from the tails of the original distribution. The authors argued that real human interaction data becomes more valuable as generated content fills the web.

Verification makes synthetic data useful. Recursive synthetic data can bleach out the rare and awkward parts of reality. Those rare parts often contain the customer problem worth solving.

Memory becomes part of the architecture

Attention lets a transformer look back at its context with precision, but the cost grows quickly with sequence length. Several research programmes are trying to give models a cheaper way to carry state.

Mamba uses selective state-space layers and scales linearly with sequence length. Its authors reported five times higher inference throughput than comparable transformers in their experiments, although the headline language results were at modest model sizes by frontier standards. Mamba-2 later connected state-space models and attention through a shared mathematical framework.

Google researchers took a different route with Titans, published at NeurIPS 2025. Titans combines attention with a neural long-term memory module that learns what to store during use. The paper reports a needle-in-a-haystack retrieval experiment beyond two million tokens, which is evidence of long-range retrieval rather than general understanding across a document of that size.

Hybrid models already exist. Jamba interleaves transformer and Mamba layers and adds a mixture of experts, aiming to keep precise attention where it helps and cheaper recurrence where it can carry the sequence.

None of these results names a settled successor. They point towards systems with several kinds of memory: attention for exact local access, retrieval for records, recurrent state for continuity and explicit databases for facts that must survive an audit.

Models learn from video, action and consequence

Text describes the world after somebody has compressed it into words. A system that must move a robot, operate software or run a process needs to predict what happens after an action.

Meta's V-JEPA 2 was trained on more than one million hours of internet video. The team then post-trained an action-conditioned world model on fewer than 62 hours of unlabelled robot video and used it to plan pick-and-place tasks on robot arms in two labs without task-specific training in those environments.

That is still research, and a warehouse is less forgiving than a benchmark. The direction matters. Video, sensor streams, user actions and environment feedback provide training signals that text alone cannot.

In a business product, the equivalent data may be less glamorous: the invoice was paid, the code passed its tests, the parcel arrived, the customer reopened the ticket, the analyst overruled the recommendation. These outcomes connect a generated answer to what happened next.

The same argument applies at system level.

AlphaGeometry is useful for another reason. Its language model proposes promising geometric constructions. A symbolic engine checks them. Each component does the work it is good at.

The ReAct research pattern similarly interleaves language-model reasoning with actions against external systems. Current products add search, code execution, calculators, databases, policy engines and human approval. Buyers experience that whole arrangement: model, memory, tools, rules and people. The model checkpoint is one component.

This is good news for builders. Frontier training remains concentrated among companies that can afford very large compute clusters. System design is open to far more people.

Where I think this goes

Everything above comes from public papers and technical reports, several of them preprints. The next points are my bets.

The transformer will remain an important component, much like the relational database survived document stores and search indexes. New systems will combine attention with recurrence, learned memory, specialised experts and external tools. A single clean replacement looks less likely than a messy, effective mixture.

Models will do more learning during use, but companies will put fences around it. Persistent memory, retrieval and small adapters are easier to inspect and delete than a system that rewrites itself after every conversation. Enterprise buyers will demand audit trails, retention controls and a way to forget.

Outcome data will become a stronger advantage than prompt collections. The useful record includes the request, the model's action, the result, the human correction and the eventual business outcome. Companies that collect this with permission can improve evaluation, routing and post-training around their real work.

Verification will split from generation. One model drafts, another checks, a deterministic tool calculates, and a person approves the cases with real downside. The best combination will change by task.

Smaller models will carry more production traffic. Distillation, routing and specialised training let teams reserve expensive reasoning for the few requests that need it. The biggest available model will often be a poor default for cost, latency and privacy.

World models will first matter in constrained environments where actions and outcomes are observable: robotics, games, industrial processes, laboratories and software. General common sense learned from video is a longer bet.

What builders should do now

Build the model layer so it can be replaced. Models will change faster than customer workflows.

Own your evaluations. Measure the errors that cost the customer time, money or trust, rather than relying on a general benchmark.

Capture outcome data with clear permission and retention rules. A pile of conversations is less useful than a smaller set connected to verified results.

Give generation a checker. Use tests, calculators, schemas, policy rules or human review wherever a fluent mistake can cause damage.

Make inference budgets visible. Decide which requests deserve more search and reasoning, and what the user gets when the budget runs out.

Treat memory as customer data. Let people inspect it, correct it, export it and delete it.

The sensible move in 2026 is to build a product whose value survives a model swap. When the next architecture arrives, you should be able to plug it in on a Tuesday rather than rebuild the company.