The Measurement Gap

We are pushing AI-mediated interactions into the world at enormous scale: customer support, education, coaching, onboarding, healthcare triage. Hundreds of millions of conversations a day are now conducted between humans and systems that were not there a year ago. This is happening very quickly, and it raises a question that I think deserves more attention than it is getting: how do we know whether these interactions are actually good for the people on the receiving end?

The short answer is that, for the most part, we do not.

We measure what the system did. We count tickets closed, sessions completed, messages sent, and response latency. These numbers are easy to collect and they look convincing on dashboards, but each one quietly encodes a theory about what matters. The theory behind “tickets closed” is that the goal of a support interaction is to close the ticket. The theory behind “sessions completed” is that the goal of a learning experience is to finish it. These are reasonable-sounding proxies, and they are also, in many cases, wrong.

The actual goal is that the human on the other end felt helped, felt understood, and left the interaction more capable or more confident than when they arrived. That goal does not appear on any dashboard I have seen, and I think its absence is a significant blind spot in how we deploy AI today.

The metrics we choose encode our theory of what matters. If that theory is shallow, the AI will optimise for shallowness.

There is a pattern that has been running through companies over the past year or so, and although the specific numbers vary, the shape of it is consistent enough to be worth describing. A team puts AI on the front line of customer support. Within a few months a substantial fraction of tickets are being resolved without a human ever touching them. Response times collapse, support costs fall, and leadership is, understandably, pleased with the results. Some quarters later the renewal numbers come in slightly softer than expected, and nobody can quite point to why.

What is happening, I think, is that a great many of the messages that look like questions are not really questions. They are frustration dressed up as a question, or confusion dressed up as a question, or quiet erosion of confidence in the product dressed up as a question. A reasonably attentive human representative picks up on that — the second follow-up in a week, the slightly terse phrasing, the third feature in a row the customer cannot quite find. They shift register. They ask what is actually going on. Sometimes they escalate the account to someone who can do something about it.

The AI, on the whole, does not. It answers the literal question and moves on. The customer’s problem, narrowly defined, has been solved. The customer, more broadly considered, has not. They feel about as unheard as they did before they sent the message, and possibly more so now that the reply has arrived in twelve seconds and reads like it was assembled somewhere else.

The obvious response is to route the hard cases to a human, and most teams do exactly that. But the consequence is that the human team is now handling nothing but hard cases. The routine ticket that used to break up the day, the quick resolution that let someone feel competent before lunch, the easy interaction that reminded them they were good at this work — those are largely gone. What is left is permanent hard mode. The way I have heard the shift described, in one form or another by several people in different companies, is something like this:

“I used to help people. Now I clean up messes.”

The people on those teams who have the most options elsewhere tend to be the first to use them. Morale and turnover do not show up on the same dashboard as resolution rate, and usually not in the same quarter, which means by the time the loss is visible it has already happened.

I recognise this pattern because I have seen a version of it in my own field. I spent eight years building adaptive learning systems, and one of the most important things I learned in that time was that completion and retention are almost unrelated; a student can finish every module in a course and remember very little of it a month later. The metric that clients cared about (completion rate) and the outcome that actually mattered (durable learning) were measuring different things, and optimising for one did not reliably improve the other.

The same disconnect appears wherever AI mediates a human experience. Engagement metrics for a learning platform do not tell you whether anyone actually learned anything. Completion rates for an onboarding flow do not tell you whether the new hire feels oriented or overwhelmed. Session duration for a coaching conversation does not tell you whether the person left with clarity or confusion. In each case, the metric captures the system’s activity while the human’s experience goes unrecorded.

What concerns me is that AI is extremely good at optimising for whatever metric you give it, which means the gap between what the dashboard shows and what the human actually felt can widen quickly, quietly, and at enormous scale.

We are deploying AI into human-facing interactions at a pace that far outstrips our ability to understand what those interactions actually feel like from the human side.

I do not think this is an intractable problem. It is, however, one that the industry has not yet taken seriously enough. We spend a great deal of energy on capability benchmarks, accuracy, hallucination rates, and cost per token, and comparatively little on the question of whether the people on the receiving end of all this capability are actually better off for the experience.

That question is worth working on. It will require new approaches to sensing what is really happening in AI-mediated interactions, and new ways of thinking about what “success” means when the human experience is the thing that matters most. I believe the tools and methods to do this are within reach, and I intend to write more about what that might look like in practice.