Measuring Psychological Change From Language, Without Asking How People Feel
Updated: Sep 17
Psychological changes can be measured from what people say without asking them how they are feeling.
Several years ago, Receptiviti demonstrated this approach at scale by tracking psychological changes in the language of thousands of healthcare workers during a once-in-a-lifetime event.
During the COVID-19 pandemic, Reddit comments from approximately 6,500 self-identified doctors and nurses were analyzed using Receptiviti's psycholinguistic measures. Subjects weren’t recruited to complete surveys or asked about their mental wellbeing. Instead, their language from their Reddit posts was analyzed to measure changes in their cognitive load, analytical thinking, anger, fear, anxiety, stress, and empathy. Their comment language was analyzed daily, making it possible to track psychological changes as they developed.
The same underlying approach has significant applications for human-AI interaction safety. If we want to understand how interacting with AI affects people, surveys and self-reports provide one source of information. The language people produce during those interactions offers another source of data that enables psychological change to be measured in the moment, and over time.
Measuring Psychological Change From Language Over Time
Among doctors, cognitive load rose after the pandemic began, peaked in 2021, and remained elevated through 2023. Nurses showed a smaller increase that subsequently declined. These changes were measured without asking them to report whether they were becoming more cognitively overloaded, less analytical, or more anxious.
The complete Healthcare Workers Mental Wellbeing Index tracks their cognitive load, analytical thinking, anger, fear, anxiety, stress, and empathy from before the onset of the pandemic through October 2023.
A New Measurement Channel for AI Evaluation
This same method can be applied to AI conversations. As part of an evaluation harness, these measures complement traditional model and task metrics, helping teams evaluate both how the system performs and how the person interacting with it is being impacted. Because the measurement method is independent of the AI model and doesn't rely on model inference, it offers a consistent measurement framework across models and versions.
AI evaluation is increasingly considering the effects systems have on the people using them. This method provides another source of evidence for measuring the how they're impacting users.
The index was the basis of a 2022 paper, "Tracking Mental Health Risks and Coping Strategies in Healthcare Workers' Online Conversations Across the COVID-19 Pandemic”.
