AI Has a Model of You. But What If It’s Wrong?

Imagine you’ve trusted the same AI assistant for months. It knows your history, remembers your preferences, and it seems to understand you. Then one day it gives advice that seriously harms you. You could conclude that the AI simply made a bad recommendation, but it’s also possible that the recommendation was a symptom of a more foundational problem – one where the AI fundamentally misunderstood the person it was trying to help.

An adaptive AI system build a model of the person it believes it is interacting with, and every recommendation, decision, and response it generates is based, in part, on that model of the person. As AI systems become more involved in mental health, healthcare, education, insurance underwriting, and financial services, the consequences of inaccuracies in that model of the person become increasingly significant.

The model of the person

The quality of the model of the person also influences the quality of the decisions the AI makes and the outcomes it helps create. Depending on the application,, those decisions can affect everything from medical treatment, to learning, to access to services, to people’s financial security, and their quality of life.

Today, the model of the person usually exists only inside the AI model itself. In most AI systems, there is no independent, scientifically grounded representation of the person that can be inspected, compared, verified, calibrated, or systematically improved. You can ask an AI what it believes about you, but you’re still asking the same system whose understanding you’re trying to verify, and its answer is just another interpretation of that same internal state.

The missing architectural question

AI researchers have devoted enormous efforts to improving reasoning, memory, planning, retrieval, tool use, and representations of the world outside the system. As AI systems are now becoming increasingly responsible for decisions about people, the model of the person is becoming one of these systems’ most important components. But with so much at stake, the model of the person shouldn’t continue to be confined to an opaque internal state.

As models improve, their inferences about people will undoubtedly improve as well. But better inference doesn’t solve the problem, especially when other scientific disciplines rely on independent references, for example, to calibrate instruments, validate observations, and improve increasingly sophisticated models. An AI’s model of the person should really be no different.

Psychology has already been working on this problem

Well before large language models existed, psychologists were challenged to figure out how to move beyond their intuition and produce observations about people that are consistent, comparable, reproducible, and grounded in empirical evidence.

Over several decades, psycholinguistic research developed scientifically validated methods that would help, by deriving stable psychological variables from natural language. Rather than asking what people said, researchers asked what language could reveal about cognition, emotion, personality, social relationships, and psychological change. These methods don’t directly observe psychological state, but they produce standardized, quantitative variables of psychological characteristics and processes derived from language, developed and validated through decades of research. For AI systems that increasingly interact through language, this approach provides an independent scientific foundation for representing important aspects of the people they interact with.

Grounding the model of the person

To be clear, psycholinguistic variables do not replace AI reasoning, they ground it. While an LLM might infer that a user is confused, anxious, cognitively overloaded, or becoming more analytical, independent psycholinguistic analysis produces standardized psychological variables to quantify Cognitive Load, Analytical Thinking, Anxiety, Anger, or Self-focus, and does so independently of the model.

Because the variables are quantitative and standardized, they can be tracked over time, compared across interactions, validated against independent outcomes, and used consistently across different AI systems and model versions. But like any scientific measurement approach, these methods have strengths, limitations, and appropriate contexts for use.

Inference creates the model of the person. Independent psychological measurement provides an external scientific reference that can ground and inform it. Together, they produce a model of the person that is both adaptive and scientifically grounded.

A new foundation for human-centered AI

As AI systems are increasingly trusted to make recommendations and support decisions that impact people, the quality of the model of the person will increasingly determine the quality of those decisions. Systems that make decisions about people should not rely only on opaque internal models of the people they serve. They should also be grounded in an independent representation of the person that can be inspected, compared, challenged, and improved over time.

An independent representation doesn’t just make AI systems more accurate. It makes their understanding of people transparent enough to be inspected, challenged, improved, and ultimately trusted. To be clear, this doesn’t mean replacing inference; it means grounding inference in an external scientific reference.

The model of the person is becoming one of the most important components of modern AI systems. It should no longer remain an opaque internal state. Grounding the model of the person in an independent scientific representation may become one of the defining architectural shifts in the next generation of AI.