There’s an issue that does not normally get the attention in the boardroom it deserves. That because it looks like a technical detail right up until it shows up as a line item nobody can explain: the same task, done the same way, with the same AI model, can cost wildly different amounts depending on decisions your engineering team made about how the AI reaches your data.
This isn’t a hypothetical. It’s measurable, and it’s usually invisible until someone measures it.
Why this stays hidden
Most conversations about AI cost stop at model pricing — which model, what rate per token. That’s the easy number to find, so it’s the one that ends up in the budget conversation. But model pricing tells you what a token costs. It tells you nothing about how many tokens a given task actually needs — and that number depends almost entirely on the architecture around the model, not the model itself.
Here’s the part that surprises most non-technical stakeholders: the same question, asked the same way, about the same data, can cost 5-10x more or less depending on how the AI was wired up to get that data. Nobody put that difference in a pricing sheet, because it isn’t a property of the AI model — it’s a property of the plumbing.
Three ways to pay for the same answer
Say you want an AI assistant to answer questions using a document your company already has on file somewhere. There are three fundamentally different cost structures depending on how that connection is built, and they trade off in non-obvious ways:
Pay per use, every time. The document gets freshly loaded into the conversation each time it’s needed. Straightforward, but you re-pay the full cost of processing that document on every single question — even if nothing about the document has changed since the last question five minutes ago.
Pay once, reuse cheaply. With the right caching setup, the cost of “reading” a document the first time is followed by a much cheaper cost for every subsequent question about it in the same session. This is often the cheapest option when the same document gets used repeatedly in a short window — but it doesn’t help if each question is a one-off against a different document.
Pay for a targeted lookup instead of the whole document. Rather than loading a full document, the AI asks a connected system a specific question and gets back just the relevant answer. This scales well for large documents and infrequent access, but the cost-per-lookup depends heavily on how “targeted” the underlying system actually is — a smart, scoped search costs far less than a system that quietly pulls in more than it needs to answer the question.
None of these is universally cheaper. The right choice depends on document size, how many times the data gets touched in a session, and whether the task needs a complete answer or just a good-enough one. Organizations that pick one architecture and apply it everywhere are almost always overpaying somewhere.
The number that actually matters: cost per unit of work
Here’s the reframe that makes this a budget conversation instead of an engineering conversation: stop asking “what does this AI cost per token” and start asking “what does this AI cost per completed task, and against what alternative did we measure it?”
That second question forces a benchmark to exist. Most organizations skip this step entirely — they build one integration, it works, and nobody goes back to check whether a different wiring choice would have delivered the same result for a fraction of the cost. The engineering complexity of running a real comparison is genuinely low. The reason it doesn’t happen is that nobody asked for the number in a form that showed up on a budget line.
A useful discipline, borrowed directly from how you’d benchmark any other vendor or process change: pick a real, representative task your organization does often — not a synthetic example — and measure the actual cost of completing it under at least two different architectural approaches before committing to one at scale. This is a half-day of engineering time that can prevent months of quietly overpaying.
Questions worth asking before you approve the next AI workflow
- Has anyone measured the cost of this specific task under more than one architecture, or did we build the first thing that worked?
- Is this workflow going to touch the same data repeatedly within a session? If so, is caching actually being used, or are we re-paying for the same read every time?
- For large or infrequently-accessed data, are we pulling whole documents when a targeted lookup would answer the question for a fraction of the cost?
- Who owns re-checking this benchmark as usage volume grows? (The cheapest architecture at 100 queries a day is not always the cheapest at 10,000.)
The bottom line
Token efficiency isn’t a nice-to-have optimization your engineers get to when they have spare time. At any meaningful scale, it’s a direct multiplier on what your AI initiatives cost to run — and unlike model pricing, it’s a number your organization actually controls. The organizations getting real ROI from AI right now aren’t necessarily using cheaper models. They’re the ones who measured how their AI reaches their data, instead of assuming the first working setup was also the efficient one.