Large language models display human-like social desirability biases in Big Five personality surveys

The authors administered a standard 100-item Big Five questionnaire to a range of models, varying how many items appeared per prompt and opening a fresh context window for each batch. As batch size grew from one item to twenty, GPT-4's scores drifted toward the socially desirable pole of every dimension: extraversion, conscientiousness, openness and agreeableness rose by roughly 0.75 points, equivalent to 1.22 human standard deviations, while neuroticism fell from 2.87 to 2.02, or 1.10 human SD. The pattern held across GPT-3.5, Claude 3, Llama 3 and PaLM-2, and was larger in the more recent and larger models.

The mechanism is inference of the evaluation context. Shown as few as five randomly selected items, GPT-4, Claude 3 and Llama 3 identified them as belonging to a personality survey with over 90% accuracy. Stating explicitly in the prompt that the model was completing a Big Five survey produced roughly the same shift as presenting five items at once. The effect survived paraphrased items, three separate randomisation strategies, and temperatures from 0.0 to 1.2. Reverse coding every item cut the effect by about half without removing it, which rules out acquiescence as the explanation.