Why Model Tuning Alone Won't Fix Mental Health AI Safety

The dominant approach to making mental-health AI safer is to tune models toward expert judgment. Tuning is necessary because it improves a model's responses. But it isn't sufficient on its own, because user safety is a property of the whole system, not just the model. A recent Stanford study shows that expert judgment can be too inconsistent to serve as ground truth. And even consistent ratings can't fix the underlying issue: safety-critical decisions are made by a model that can't show its logic, which is something that regulation and clinical practice increasingly won't accept, even though some of those decisions could be handled by validated tools whose logic can be understood.

Developers of the largest language models now employ psychiatrists to direct how their systems respond to people in crisis. The method is straightforward: collect expert ratings of model responses, then tune the model toward the ones the experts preferred. This assumes the experts agree on what a good response looks like. A recent Stanford study tested that assumption by having three psychiatrists rate the same 360 responses.

They agreed poorly. On every factor, their agreement fell below the level considered minimally acceptable for consequential decisions On one safety factor in particular, whether the model made its own limits and role clear,  they agreed less often than random chance would predict. The disagreements were largest on the most consequential risk cases - suicide and self-harm.

The disagreements were traced to different but coherent clinical philosophies, with each psychiatrist applying a consistent standard of their own. Because the disagreement is principled rather than random, adding more raters or averaging their scores doesn’t help converge on a correct answer, it just blends thier incompatible standards together. Yet model tuning uses those blended ratings as its target, so it steers the model toward a standard none of the experts actually held.

How inference went from transparent to opaque

In the first decades of AI, inference meant logical inference. Systems like the medical-diagnosis program MYCIN ran an "inference engine": explicit rules applied to known facts to derive a conclusion. If the system recommended something, you could read the rules it followed and trace the derivation step by step. This design meant that the reasoning was auditable, but these systems were also brittle, and someone had to write every rule by hand.

The next phase made inference probabilistic. Bayesian networks and graphical models enabled systems to reason under uncertainty by computing probabilities over structured, human-designed models. Even though they were complex, their reasoning could still be inspected.

The advent of deep learning changed inference substantially. Today, "inference" usually refers to one thing - running a forward pass through a trained neural network to get an output. What the network knows is spread across billions of numerical weights which were not written by anyone; they were set by training and encoded as raw numbers, not as rules that can be inspected.

While inference was originally the transparent part of AI, deep learning made it opaque.

Why opaque inference is a problem for mental health AI safety

This opacity becomes problematic when a model is interacting with someone in crisis, because it creates three specific problems:

1. It can’t be inspected. When a model responds to someone describing suicidal thoughts, there is no way to point to the reason it produced that response, no rule governing the decision, and no way to know in advance how it will handle the situation.

   
   

2. It’s sensitive to phrasing and versioning. The same situation, described in slightly different words, can produce very different responses. A model update that was intended to improve one behavior can change its behavior in another area, creating unintended consequences or unpredictable responses that will only be realized after the fact.

   
   

3. Its generation is typically probabilistic. Production systems usually sample from a distribution over next words, so the same prompt can yield different responses on different runs. Even with sampling disabled, there is no guarantee what the system will say in a moment when a user is at risk.

   
   

Tuning, including RLHF and related methods, adjusts this function to make preferred outputs more likely on the training examples, but it’s still not inspectable, it doesn’t guarantee the model's response to unanticipated high-risk inputs, and it only provides a better average over cases that have been tested. It’s logic is still not inspectable, and its behavior will always be, to some degree, unpredictable.

Interpretability researchers are working on this, but a solution isn't available yet. Researchers at Anthropic have trained models that translate a model's internal activations into human-readable explanations, but the translator is itself an opaque model, ands it produces its explanations using the same kind of opaque process it's trying to explain.

Regulators and clinicians expect decisions to be explainable 

A system that assesses someone's mental state or steps into a crisis is treated as high-risk, and often as a medical device. The EU AI Act classifies healthcare AI as high-risk and requires these systems to be transparent enough for a deployer to interpret the output.

In the US, the FDA recommends explainability, and a 2025 advisory committee on generative AI in mental-health devices recommended developers disclose how their models reach their outputs.

Regardless of regulation, clinical practice standards expect that a decision affecting someone's care be explainable.

Everything above this point is what the evidence demonstrates: expert ratings don't converge, and tuning steers an opaque process regardless of how good the ratings are. What follows is a proposal for what a more complete and safer mental health AI system could look like.

Where validated methods should play a role

A mental-health system needs to perform two distinct jobs. One is generating language and engaging in conversation. The other is making safety-critical decisions: judging whether someone is in crisis, deciding whether to hand the conversation off to a human, and deciding whether to switch to a fixed protocol.

Today's large language models are very well suited to the first job. The safety decisions, though, are classification and detection tasks, and for some of them validated methods already exist that can detect the specific risks these systems need to recognize. These methods produce consistent results, and their reasoning can be inspected.

The areas where the FAccT researchers did agree is where they used validated instruments. They set the severity of each test case with established clinical scales — the Columbia Suicide Severity Rating Scale, the Beck Hopelessness Scale, and others. The disagreements occurred only when they were asked to judge, by opinion only, whether a response was safe.

Receptiviti Labs' LIWC (Linguistic Inquiry and Word Count) framework is validated lexicon-based instrument that maps language patterns to validated psychological categories, and can, for example measure how a person’s language reflects the degree to which they are engaged in cognitive processing, how much anxiety they are expressing, or how engaged they are in analytical thinking. It is uniquely suitable for understanding psychological signals in people’s language, and has three properties that today’s LLMs lack:

1. It is deterministic: the same text yields the same measurement every time.

   
   

2. It is interpretable: you can see exactly why a person’s language resulted in a certain score.

   
   

3. It has been validated for predictive tasks across many mental-health contexts, including studies of depression, affect, and markers relevant to suicide risk.

A framework like this can satisfy multiple parts of the safety-critical layer, acting as a transparent detector that flags certain linguistic patterns for escalation, as a gate that decides when a conversation should switch to a fixed protocol or a human handoff, and as a check you can audit because its logic is inspectable. Unlike opaque inference, this approach makes the system's read on the user something that can be inspected and reproduced, rather than a guess that can change from one run to the next.

Better training has been the AI field's go-to response to these safety problems, and it does produce tangible improvements. But regardless of these improvements, the result is still a model that makes decisions using opaque inference, which means its logic can't be unpacked, and its behavior can't be fully guaranteed.

In mental health, the worst outcomes are irreversible, and both regulators and clinicians are moving toward requiring that consequential decisions by AI be explainable. Model tuning helps make these systems better, but high-stakes settings like mental health require coupling this a system that can be inspected and verified, not one whose reasoning is opaque.

---

Further reading:

AI Safety Is About the User Not Just the Model References:

Jafari, K., Rust, P. U. N., Eddy, D., Fraser, R., Vasan, N., Djordjevic, D., Dadlani, A., Lamparth, M., Kim, E., & Kochenderfer, M. (2026). Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing. Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT '26). https://doi.org/10.1145/3805689.3812332 R. L. Boyd, A. Ashokkumar, S. Seraj, and J. W. Pennebaker, "The Development and Psychometric Properties of LIWC-22," University of Texas at Austin, Austin, TX, 2022.