Validating Elicited Beliefs from LLMs: A Decision-Theoretic Framework

Researchers have proposed a new decision-theoretic framework to validate the beliefs reported by Large Language Models (LLMs) in high-stakes environments. As LLMs are increasingly used for critical decisions, such as clinical diagnoses, it is vital to determine if their stated probability judgments align with their actual actions. This study introduces methods to test the mutual consistency between an agent's elicited probabilities and its decisions, characterizing whether actions could stem from a 'near-rational' decision-maker holding those beliefs. Notably, the framework allows for empirical testing without assuming specific utility functions. When applied to stylized clinical diagnosis tasks, the analysis revealed that while LLMs' reported beliefs are imperfect summaries of the information underlying their decisions, the discrepancies are minimal for the most advanced models. This work provides a crucial tool for assessing the reliability and coherence of AI agents in sensitive applications, ensuring that their verbalized confidence matches their operational behavior.