Psychometric Evaluation of AI Behavior Using Situational Judgment Tests

Researchers have introduced a new framework for evaluating the behavioral consistency of Large Language Models (LLMs) using psychometric methods. The study addresses the uncertainty surrounding persona conditioning, investigating whether it creates stable behavioral structures or merely superficial variations. By employing Situational Judgment Tests (SJTs), Multidimensional Item Response Theory (MIRT), and structured synthetic personas, the team treats model responses as observations of latent behavioral variables. Their analysis of large-scale datasets reveals that persona-conditioned behaviors remain stable across multiple runs. Furthermore, latent trait scores derived from this method successfully predict performance on external benchmarks such as TruthfulQA and EmoBench. The findings are validated through human annotation, benchmark evaluations, and internal consistency analyses. The authors clarify that these traits represent stable behavioral tendencies rather than human personality. This scenario-based psychometric evaluation is presented as a more reliable alternative to classical self-report approaches for assessing LLM behavior. To facilitate further research in this domain, the team has released the associated datasets, offering a robust tool for understanding and steering AI behavior in diverse contexts.