The end of tokenmaxxing, almost

At Amazon, according to the Financial Times, some people kept an agent busy with work nobody had asked for, just to burn tokens, and a group of employees had built a leaderboard that put whoever burned the most at the top. Amazon took it down in May, and this week it published the internal memo that tells its engineers what it expects of them now that they work with AI. The memo closes out the year of tokenmaxxing, when plenty of tech companies measured AI by how much they spent on it. Counting tokens was the easy part, and the part that matters, knowing whether AI makes people better at the job, still barely gets measured.

The number that counts itself

Any company that buys AI tools for its engineers wants to know whether they’re worth it. A real answer is slow to get: you have to look at what ships, how good it is and what it costs, and then wait months to see it. Token usage, on the other hand, is on the vendor’s dashboard from day one, keeps climbing and sorts neatly from top to bottom. No wonder it became the answer.

There’s been no shortage of examples this year. In March, Nvidia CEO Jensen Huang said he’d be “deeply alarmed” if a $500,000 engineer didn’t consume at least $250,000 worth of tokens. In April, word got out that a Meta employee had built Claudeonomics, a leaderboard of the heaviest token users with titles like “Token Legend,” and two days later it was gone. Amazon’s was KiroRank, named after Kiro, its AI coding tool, and the company retired it at the end of May with a plea to staff: “Please don’t use AI just for the sake of using AI.”

None of this is new. The economist Charles Goodhart described it in 1975, and the anthropologist Marilyn Strathern boiled it down to the line now known as Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure.” Once usage is ranked and rewarded, it stops telling you anything about the work behind it.

The bill put the brakes on

Since the spring, companies have been hitting the brakes, and not because they found a better measure. In April, Uber’s CTO admitted that the AI budget he’d planned for 2026 was already blown. In August, Microsoft executive vice president Jay Parikh told staff that “tokenmaxxing is not what we are optimizing for,” that everyone needed to watch how they consume tokens, and that each division would get a budget. As of 1 October, Accenture covers $29 a month per person for Claude, and anything above that has to be charged to a project or a team. Companies have gone from rewarding usage to budgeting it, and they’re still looking at the same number.

Amazon, halfway

The company’s most senior engineers sent the memo in September to its developers and their managers, along with the new role guideline, and Amazon published it on its website on 27 October. It’s blunt about token usage, tool adoption and counts of code changes: “We are all aware of how deeply flawed these metrics are, primarily because they are disconnected from what matters.” It even warns that, used “as a target instead of a measure,” they can drive up costs, since “tokenmaxxing is at odds with frugality.”

One paragraph later, it says those same metrics “correlate well with the outputs that matter,” that using them “with care and judgement” is appropriate when engineers think about their careers and their organization’s performance, and it recommends looking at the teams in the 10th percentile of AI adoption to understand why they aren’t embracing it. The two don’t fit together. If a number counts toward your career, it’s a target, which is exactly what the memo warns against, and Goodhart does the rest.

The hard part

What should be measured is whether teams deliver better, and that isn’t easy. First, people’s own sense of it can’t be trusted. In 2025, METR, an independent lab that evaluates AI models, had 16 experienced developers work on real tasks from open-source projects they’d contributed to for years, with each task randomly assigned to be done with AI or without it. Before starting, they expected to save 24% of their time. They took 19% longer, and afterward they still believed they’d been faster. With early-2026 tools, METR thinks developers are probably faster now, though it calls its data “only very weak evidence” of how much.

Second, what goes up isn’t always what matters. In July, Microsoft researchers estimated that engineers who started using Claude Code or Copilot CLI merged about 24% more pull requests. Faros AI, which sells engineering analytics, followed teams as they went from little AI to a lot, and found their pull requests spent five times as long in review and carried nearly 29% more bugs. Count only the pull requests and everything seems to be improving. The cost shows up later, in review and in production.

That’s why the old metrics make more sense, measured team by team: how long a change takes to reach production, and what share of changes fail once they get there. They’re two of the DORA metrics, from the Google research group that has spent more than a decade studying how teams deliver software (not to be confused with DORA). In June, DORA wrote that for token usage, company-wide aggregates are “meaningless” and individual scores are “harmful,” and suggested looking at usage by team, by application or by use case instead.

The leaders’ test

By the memo’s standard, a leader, whether a manager or a technical lead, knows AI well enough when they’ve used it on their own production work, can describe a task where it helped and one where it didn’t, and can explain that judgment to the team. It’s a test that can’t be delegated or put on a dashboard, and that’s what makes it worth something. Engineers get a line that matters just as much: “If your work does not benefit from current AI tooling, that is a finding, not a failing.”

Passing that test means paying attention to how AI gets used, which token counts don’t show. In the experiment Anthropic published in January, 52 engineers, mostly junior, had to learn a Python library they’d never used. The group with AI averaged 50% on the quiz afterward, compared with 67% for the group that coded by hand, but scores varied a lot with how people used it. Those who had it explain the code or asked about the concepts averaged 65% or more, and those who handed it the work averaged under 40%. The groups were small, two to seven people each, and the authors don’t claim the pattern caused the score.

Fewer people are starting out, too. According to a Stanford study, by July 2025 employment of software developers aged 22 to 25 in the US was nearly 20% below its late-2022 peak. The authors say they don’t show that AI is the cause. Even so, if the ones who do get in learn to delegate before they learn to understand, it’s hard to see where the people who review all those pull requests a few years from now will come from.

Where to start

Amazon isn’t offering a better metric, and it says it will find one “over time.” Nobody has one yet. In the meantime, the leaders’ test is a good place to start. Write down one task this week where AI helped and one where it didn’t, with how long each took and why. If you lead a team, run it on your own work before you ask anyone else to. And if your company has a token leaderboard with names on it, suggest taking the names off, treating each team’s usage as one more cost and judging delivery with those two DORA metrics.

Picaporte Labs is one person, in Spain. If your company keeps count of tokens too, write to me and tell me what it measures and what has changed.