From Scaling Laws to Agentic AI

This is roughly the path I take my Columbia students through when they ask where large language models are headed. The honest answer is that what we are building changed in about two years, and a lot of the intuition from the scaling era no longer applies.

For a few years there was a simple answer to almost every question in this field. How do you make the model smarter? Make it bigger. How do you plan next year's system? Assume more compute. That answer was not lazy. It was backed by one of the most reliable empirical findings machine learning has ever produced. It also stopped being the whole story, and the students who understand why are the ones who will build what comes next.

The scaling-law era

In 2020, a team led by Jared Kaplan published a result that shaped the next several years [1]. They found that a language model's loss, which is basically how wrong it is, falls as a smooth power law as you increase model size, dataset size, and training compute. Plot it on log-log axes and it is a straight line, holding across many orders of magnitude.

Fig. 1: Across orders of magnitude, model error dropped as a straight line on log-log axes. That predictability is what turned scaling into a strategy. Source: Kaplan et al., 2020 (illustrative shape).

That straight line turned research into planning. If you knew the line, you could predict how much better a model would get before you trained it, justify the spending, and place the bet. The jump from GPT-2 to GPT-3 and onward was, in large part, that prediction being acted on again and again. Bigger reliably meant better, and for a while nothing else mattered as much.

But look at the axes. They are logarithmic. Each additional step down in loss costs roughly ten times the compute of the one before it. The relationship held, and it also got more expensive at every step.

The correction nobody advertised

In 2022, a DeepMind team led by Jordan Hoffmann pointed out that everyone had been reading the scaling laws wrong [2]. Given a fixed compute budget, the field had been pouring it almost entirely into parameters and skimping on training data. When they balanced the two, the result was striking. Their model Chinchilla, at 70 billion parameters trained on far more data, beat Gopher, a 280 billion parameter model, on a wide range of tasks. A model four times smaller won, because it was trained on the right amount of data.

Fig. 2: First the field spent compute on bigger models, then on more data, then on letting a fixed model think longer at answer time. Sources: Kaplan et al. 2020; Hoffmann et al. 2022; OpenAI 2024.

The lesson from Chinchilla is not the specific ratio. It is that even the most trusted scaling result had a hidden assumption in it, and the moment someone questioned the assumption, the optimal strategy flipped. Scaling laws tell you how to spend compute. They do not tell you that spending more compute the same way is always the right move.

The wall you cannot buy your way past

By 2024 the limits were visible. Training the next model was not only ten times more expensive, it was running low on the one input the scaling laws assumed was effectively unlimited, which is high-quality data. You can always build a bigger cluster. You cannot as easily produce another internet of high-quality text to train it on. The relationship still held, but the returns were getting more expensive and the data was running short.

So the interesting question stopped being how much bigger can we make the model, and became something more useful. Is training the only place we are allowed to spend compute?

Spending compute when you ask, not just when you train

The answer, and the key shift of the last two years, was no. In late 2024 OpenAI showed with its o1 models that you can let a fixed model think longer at the moment you ask it a question, and accuracy rises with that inference-time compute [3]. Instead of one quick pass to an answer, the model generates a long internal chain of reasoning, checks itself, and works the problem before it replies.

This is a different axis, and it reorganizes everything downstream. Training compute is a cost you pay once, up front, spread across every future user. Inference compute is a cost you pay every single time someone asks a hard question. A model that thinks longer is more capable and also more expensive per answer, which brings the problem back to questions of systems and cost. I wrote about where that leads for the economics in The Real AI Cost Problem.

From a model to an agent

Once a model can spend effort at answer time, the next step is obvious in hindsight. Let it take actions, not only produce text. Let it call a tool, read the result, and decide what to do next.

That loop is the definition of an agent, and its clearest early form was a method called ReAct, which interleaves reasoning steps with actions so the model can plan, use a tool, observe what came back, and adjust [4]. It sounds simple. It changes the unit of intelligence. A chatbot is a single forward pass, one question in and one answer out. An agent is a loop with tools and memory that keeps going until the task is actually finished.

Fig. 3: A model answers in one pass. An agent reasons, acts on the world, observes the result, and repeats, drawing on tools and memory. Loop pattern after Yao et al., 2022.

This is a real advance, and it also has a cost worth being explicit about. A single agent task is not one model call. It is often dozens, each one reasoning, each one possibly calling a tool, each one paying that inference-time cost. The intelligence got better. The number of times we run the model per useful outcome went up by an order of magnitude.

Why I teach this as a systems problem

Here is where my hardware and systems background makes me see this differently than a pure machine learning course would.

When intelligence was a single large model, the work was training. When it is a loop of many calls across tools and memory, the work becomes orchestration: which model handles which step, how requests get routed, where the answers get cached, how much each step is allowed to cost. That is not a modeling problem. It is a systems problem, and it connects directly to the hardware and the full stack I wrote about in Why Every AI Engineer Needs to Understand Hardware. The control point is moving from the model to the layer that decides which model runs and when, which I covered in The Routing Layer Decided What Shipped.

My take, from the systems side

I do not think we hit a ceiling on scaling so much as we discovered the field had been treating one axis as the only axis. Training bigger still helps. Training on more and better data still helps. But the frontier moved to what a model does at answer time, and to the systems that wrap many model calls into something useful. That is good news for people who think in systems and bad news for anyone who assumed the last decade's approach of just buying more parameters would keep working forever. The next hard problems are about coordination and cost, not just capability.

What actually survives

When students ask me what to learn so they are not obsolete in two years, I do not tell them to memorize the current scaling coefficients or the latest agent framework. Those get replaced. I tell them to learn how the shifts happened. Understand that a trusted result can hide an assumption. Understand that compute can be spent at training time or answer time, and that the two have completely different economics. Understand that intelligence is becoming a system of models and tools and memory, not a single artifact you download.

The scaling era rewarded whoever could afford the biggest model. The era we are entering rewards whoever can design the smartest system around models that are, increasingly, a commodity. That is a different skill, and it is a more interesting one.

---

References

[1] Jared Kaplan, Sam McCandlish, et al., "Scaling Laws for Neural Language Models," arXiv:2001.08361, 2020.

[2] Jordan Hoffmann, Sebastian Borgeaud, et al., "Training Compute-Optimal Large Language Models," arXiv:2203.15556, 2022. The 70B-parameter Chinchilla, trained on 1.4 trillion tokens, outperformed the 280B-parameter Gopher.

[3] OpenAI, "Learning to reason with LLMs," September 2024. Performance improves with both train-time and test-time compute.

[4] Shunyu Yao, Jeffrey Zhao, et al., "ReAct: Synergizing Reasoning and Acting in Language Models," arXiv:2210.03629, 2022.