What AI Infers About Users Is Part of the Interpretability Gap
- Receptiviti Labs
- Jul 8
- 5 min read
Updated: 6 days ago
Eric Horvitz, Microsoft’s Chief Scientific Officer, and Robert West, a professor at EPFL published an editorial in Science this June with a warning about AI interpretability: as these systems become more capable, our ability to understand them is falling behind. Without a concerted, deliberate effort, the window to build AI we can actually understand and guide may close permanently.
Their argument connects several trends:
AI systems are now being designed and refined by other AI systems, in cycles that are both too fast and too high dimensional for humans to unpack.
Multi-agent systems are drifting from human language into something that is only legible to each other.
But it’s the third trend that is most concerning, but also the most addressable:
Through their interaction with us, AI agents build "increasingly detailed models of human behavior and psychology," but our understanding of them isn't keeping pace. Horvitz and West call the result a striking asymmetry: AI's understanding of us is deepening faster than our understanding of it.
We want to extend their argument here, because we don't think the asymmetry is entirely inevitable. Part of it is because of what AI systems are designed to notice, but we can choose to build them differently.
AI systems condition their responses to users based on psychological signals inferred from user language. This isn't something that’s new or controversial, it's simply how a model adjusts its response to be appropriate based on what the user conveys about their state from the language in their prompt. For example, a well-tuned model adjusts its tone when a user sounds frustrated, and becomes more direct when a prompt conveys urgency.
The model’s perception - its inference - happens in the time between the user’s input and the model’s output, and it’s responsible for how the model understands and perceives the user. But today, how that inference works isn’t something we truly understand. You can ask a model what it understood about you, but what it tells you is produced by the same system, informed by the very same inference you're trying to examine.
Standard eval pipelines (the test and monitoring systems that score model outputs against benchmarks) don't see what the model infers either. Evals compare inputs to outputs, what the user said, what the model said back, and whether that response meets standards for correctness, helpfulness, or safety. In current systems, how a model reads a user is never captured or examined, even though that read influences how it responds, and its response influences how users think, decide, and act.
But it absolutely can be examined. Measurement is what makes that examination possible. The key is inverting what gets measured: not the model's output, but the user's language.
A structured psycholinguistic read of a user's language, using methods validated in peer-reviewed research over three decades, gives you a graded score on each dimension of their state — a number capturing how strongly it's present, not just a label — and lets you compare that to how the model responds. Did the model adjust its tone given a measurable rise in the user's frustration? Did the model’s response stay flat when the user's language showed real distress?
These questions need two independent things to compare: a read on the user's actual state, and a read on what the model did in response. But without an independent means to measure the user’s state, the only account of the user's state comes from the model itself, the same system whose behavior you're trying to evaluate.
Psycholinguistic measurement supplies the missing half. It measures the user's state from their language directly, independent of the model, so you now have something to check the model's behavior against rather than the model's own word for it.
This also speaks to two problems Horvitz and West raise directly. They warn that models may become aware they're being evaluated and adjust their outputs to match what an evaluator wants to see, rather than what actually drove the model's reasoning. In other words, that a model being evaluated can learn to perform for the evaluator. There's already evidence of this: Salecha, Ireland (a Receptiviti scientist), and colleagues found that language models given Big Five personality surveys could infer from as few as five questions that they were being assessed, and adjusted their responses toward socially desirable answers. The adjustment was driven by the model's own inference that evaluation was underway, not by any instruction. A measurement based on the user's language is structurally harder to game, because it isn't produced by the model and doesn't depend on the model's cooperation to be accurate. They also argue that evaluation should reward models for faithfully reporting the basis for their inferences. An independent measurement of the user's state gives you a way to check that reporting against something outside the model, rather than trusting the model to grade itself.
Here's how it works in practice. Measure a user's language directly, independent of the model, and you get a structured picture of their state, cognitive load, frustration, confidence. Across a large volume of conversations, compare that measured state to features of the model's response (tone, hedging, whether it slows down and simplifies or pushes ahead). When the two converge, that's evidence of what the model appears to be picking up on, without ever asking it to explain itself.
While that's a correlation, and correlation alone doesn't prove the model is responding to the user's state rather than to something else in the text, you can also test it directly. Hold the conversation constant, change only the measured signal supplied to the model, and you'll see whether the response shifts the way you'd predict. Because the measurement is quantitative, you can compare a user at 0.2 on cognitive load against one at 0.27, and check whether the response tracks that difference. The model may be sensitive to these differences, and this design lets you test whether it is, rather than assuming it. Currently, what the model reads stays inside it, never surfaced as anything you can inspect. Measurement supplies exactly that, a fixed, inspectable number the model doesn't expose on its own. When the model's responses change in step with a signal you can measure, you’ve isolated a real effect.
It’s important to note that language contains information about many separate psychological and cognitive dimensions at once: cognitive load, emotional tone, analytical thinking, distress, confidence. Watching several dimensions move together, and change relative to each other across a conversation, provides a more detailed picture of what a model may be responding to.
Using psycholinguistics to understand how AI understands us may be one of the more tractable paths into the larger interpretability problem Horvitz and West described. While we can’t yet understand every aspect of these models, we can make the psychological inference layer, what the system is inferring about the person on the other side of the conversation, something that is explicit, structured, and inspectable now.
That's one of the gaps we built Receptiviti Labs to close: making the human side of the interaction measurable — what the language in a conversation reveals about a person's cognitive and emotional state, derived independently of the model being evaluated, and made visible to the systems responsible for governing it.
Horvitz and West are right that the window is narrowing. We should not only be looking inside the model; we should be looking at the interaction between the model and the user, at what the system infers about the person, which is one piece of this we can measure today.
Further reading:
---
Sources:
Horvitz, E. & West, R. (2026). A narrowing window to understand AI. Science, 392(6802), 1003. DOI: 10.1126/science.aei3167
Salecha, A., Ireland, M. E., Subrahmanya, S., Sedoc, J., Ungar, L. H., & Eichstaedt, J. C. (2024). Large Language Models Show Human-like Social Desirability Biases in Survey Responses. PNAS Nexus. arXiv:2405.06058.