From FLOPS to Intelligence per Watt

There is no working, adopted Intelligence-per-Watt metric today. I want to say that plainly, up front, because I think it’s one of the most consequential open problems in the field — and because most conversations about AI efficiency proceed as if the metric already exists and we’re just arguing about the number.

This is Part 3 of a series on the real fault lines in AI system design. This one is about Progress — and about a bet I think the industry made without quite admitting it was a bet.

We bet the industry on one scaling law

The premise was simple: more parameters, more compute, more training data would reliably keep yielding more capability. It’s not a crazy premise — it worked, repeatedly, for years. But I think it’s worth putting some numbers next to it. 77% of real-world AI queries are, by independent measurement, practical and everyday enough that they never needed frontier-scale capability in the first place. Token volume is growing roughly 1,300× year over year. And even the vendors selling the most frontier compute now openly price a spectrum of tiers — free, mid, premium — rather than a single undifferentiated product.

The key question I’d like to pose here is this: is the biggest or the most complex system really the approach we, as a society, should follow to build an intelligent infrastructure? For me, intelligence is the value that we, as humans, actually get when one of these agentic systems performs a task for us:

• Intelligence is different from peak FLOPS, whether per second or per unit of energy. FLOPS are like horsepower in a car: they measure how much power you have, not how efficiently it serves your needs. A 100-horsepower car can deliver a better experience for city driving than a 300-horsepower one — quieter, smoother, and maybe safer.
• Intelligence is not what the systems we use measure. We are charged by the token. But are 100 tokens in one system more or less efficient than 10 tokens in another? Today we don’t have a good way to measure it.
• Sustainable intelligence is not the target for the top players in the industry. In many cases we are trying to solve problems by brute force. Are we forcing ourselves to evaluate the goodness of AI systems with the wrong metrics?

In this entry I’ll set out my first thoughts on the problem I see in how we evaluate AI today. The main challenge, as I see it, is that the way we evaluate it runs opposite to what we need: something that forces the design of scalable, sustainable systems, fit to provide the right level of intelligence to their users. Peak performance != sustainable intelligence.

Same answer, different bill

Today, most agentic systems are a black box with a single entry point. You write a prompt; some time later you get a response. Everything in between — how many agents were invoked, which models they ran on, how many times they passed work back and forth before agreeing — is invisible to you. I want to make that concrete, because I think the consequences are easy to underestimate.

Take an ordinary task — call it Task A: reconcile a quarter’s worth of supplier invoices against purchase orders and delivery notes, and flag every discrepancy with the reason it was flagged. It’s well specified, it matters to the business, and it’s exactly the kind of work enterprises are handing to agents right now.

You have access to three agentic systems that can do it:

• System 1 runs an elaborate flow: a planner that splits the work, three specialist agents for extraction, matching and policy checks, and a critic that reviews their output and sends it back when it isn’t satisfied — all on mid-sized models.
• System 2 runs exactly the same flow, but every agent sits on a larger, more capable model.
• System 3 runs a deliberately simple flow — one agent that extracts and matches, a second that checks — on small models.

You give Task A to all three. All three get it right by your criteria: every discrepancy found, nothing invented. From where you sit, the three of them did the job. Each delivered an intelligence score of 100.

Now look at what each one consumed to get there: 100, 75 and 25. Pick whichever unit matters to you — energy, money, carbon. The ratio is the point.

The ordering isn’t the one most people would guess. System 2 runs bigger models than System 1, yet consumes less. The reason is in the flow: System 1’s mid-sized agents miss things, so its critic keeps sending their work back, and the loop runs four rounds before it converges. System 2’s stronger agents get it right the first or second time. Same flow, same number of agents, a very different number of interactions. And System 3 beats both by a factor of three to four, simply because Task A never needed that much machinery in the first place.

From the user’s side, none of this is visible. Frameworks that charge by the token give you one partial signal — cost — and it’s a useful one for understanding the financial implication. But things go well beyond cost. A token is not a unit of work: a token from a small model and a token from a frontier model are very different amounts of energy, and the number of agent-to-agent exchanges behind an answer, or its carbon footprint, doesn’t appear anywhere.

As an individual you can, with some effort, make a choice: try all three, notice the difference, keep the cheapest one that works. That doesn’t scale. When agentic systems call other agentic systems, no human is in the loop to make that comparison, and no mechanism I know of today makes it automatically and factors it into the decision. For a company, the question is sharper still: how do you evaluate the level of intelligence you’re actually buying, and tie it to the consumption KPIs you’re accountable for — energy, cost, emissions?

That, to me, is the real gap. We have no general means to evaluate, automatically and without a human judging every result, what level of intelligence a system actually delivered for a task. Without one, the only lever left is the one we keep pulling: bigger and bigger systems and models, trying to solve the problem with brute force.

Closed models, closed agents: you can’t even see the flow

Everything above gets worse when the models and the agents are closed.

With a closed service, you send your prompt to the company you’re using. You can choose between models — but from a menu of names that tell you about price tier and recency, not about the level of “intelligence” each will actually provide on your task. You can’t choose what agentic flow runs behind the scenes. You can’t control the budget it’s given to perform its operation: how many steps, how many calls, how much reasoning before it answers. Some services now go further and route each request to a model of their own choosing.

What comes back is an answer and an invoice. You can understand the impact in terms of the cost you pay. But two things stay out of reach:

• The alternatives. You don’t know what other options, if any, would have achieved the same goal — another model, another flow, a smaller budget — or which of them would have been the more sustainable way to get there.
• The footprint. You don’t really understand the impact of your choices on what matters beyond money, such as carbon. Two requests with the same price can sit on very different energy and emissions profiles, and nothing at the boundary tells you which one you got.

I don’t think this is anyone’s bad faith. It’s the natural consequence of selling intelligence as an opaque service, priced by the token. But it means the people best placed to make the sustainable choice — the ones who know what the task actually needs — are the ones with the least information to make it.

What’s already being built toward this

The field isn’t standing still. There are already many good efforts to keep a model’s accuracy while cutting what it costs to run, at every level of the stack:

• Number formats and quantization. FP8 is now used even for training at frontier scale; block-scaled microscaling formats such as MXFP4 have been standardised through the Open Compute Project; and post-training methods such as GPTQ and AWQ bring weights down to four bits with little loss in accuracy.
• Pruning and sparsity. SparseGPT showed that very large models can be pruned to half their weights in one shot, and structured 2
  sparsity has hardware support in current accelerators.
• Mixture-of-experts. Activating only a fraction of parameters per token turns “bigger model” into “cheaper token” rather than only “slower token” — provided the hardware does expert-parallel routing well.
• Distillation, speculative decoding, cascades and routers. Smaller students trained from larger teachers; small draft models that let a large model verify rather than generate; and systems such as FrugalGPT and RouteLLM that send each query to the cheapest model likely to answer it well.

These techniques are good. They’re remarkable, in fact, because together they provide a plethora of different ways to optimise models and AI systems, each landing at a different balance of accuracy and resource consumption. But they don’t solve the problem of understanding the intelligence a system delivers. If anything, they make it harder. Every technique multiplies the number of configurations a single task could run on, and choosing which of them gives you “more intelligence” in a more sustainable way is now a search over thousands of options with no compass.

I said this plainly myself over a year ago: the shift is from monolithic models to distributed, agentic systems. It was never about scaling bigger. It’s about scaling smarter.

And honestly — more is still needed. Sparsity, precision and open orchestration make today’s architecture more efficient, but none of the three, alone or combined, is guaranteed to be enough for the next real leap in capability. That remains an open research question, not a solved one.

The gap: what a real metric would actually require

The closest attempt so far — a 2025–2026 Stanford/Together AI research programme — measures generic benchmark accuracy per watt across a broad set of local models and accelerators, and it represents real, credible progress. It is still not the metric I think the industry needs, because a benchmark score is not the intelligence a specific task requires.

The obvious near-term approach is to learn the metric from people. Collect feedback on results — accepted, rejected, corrected, rated — and over time learn, for particular classes of queries, the intelligence users actually perceive they received. Combine that with what each request consumed, and you have a first, empirical intelligence-per-watt estimate. Preference-trained routers such as RouteLLM already work along these lines to decide which model a query deserves.

It’s a reasonable place to start, and I think someone should build it. But it has drawbacks that stop it from being the answer:

• Human perception is always perception. It’s difficult to make quantitative.
• It varies across humans. The same answer is excellent to one user and inadequate to the next.
• It isn’t based on a scientific formulation. It tells you what people liked, not what the system delivered.

What we actually need is a formalisation of this new metric — a definition of intelligence delivered per unit of resource. As I see it, it has to be:

1. Mathematical — defined, not voted on.
2. Repeatable and homogeneous across agentic flows and LLMs — so that System 1, 2 and 3 above can be placed on the same scale, whoever built them and whatever models they run.
3. Verifiable — something a third party can check, not a figure a vendor reports about itself.
4. Open and adopted — a common definition the industry actually uses, the way it uses a benchmark suite or a standard interface today, rather than one more proprietary score.

It would also have to be counted end to end — across the whole path an action takes through a system and a datacentre, not only the inference step on one chip — and it would have to respect that healthcare, finance and logistics define a “good” intelligent action in ways that have very little in common.

I’ll be honest: we don’t know how to formalise it yet. We’re working on it, and I don’t believe anyone has it today. But it is extremely needed to design new systems.

The metric you optimise for is the system you get. We need to design the next generation of agentic systems — hardware and software both — so that they’re forced to optimise toward intelligence delivered per watt. Otherwise we’ll keep designing extremely complex systems and models that tend not to be efficient, because the optimisation function used to build them still isn’t intelligence. It’s the continuation of the old trend: peak performance, or at best peak performance per watt. That’s definitely not the way to go.

I think this is also the fault line where the industry’s public narrative and its private capital allocation are furthest apart. Publicly, efficiency, specialisation and intelligence-per-watt dominate the conference circuit and the research literature. Privately, training budgets continue to flow overwhelmingly toward larger frontier runs. I don’t think this is hypocrisy so much as an unresolved bet: the industry hasn’t yet decided whether the next major capability gain comes from one more order of magnitude of scale, or from architectural specialisation — and until a working Intelligence-per-Watt metric exists to settle the question with evidence rather than intuition, I expect both strategies to keep being pursued simultaneously, at real cost to both.