New Framework Uses Subjective Logic to Assess AI Dataset Trustworthiness and Bias

Researchers have introduced a novel formal framework for assessing the trustworthiness of AI training datasets, specifically targeting global properties like bias that emerge at the dataset level. Published on arXiv, this study addresses a critical gap in prior work, which primarily focused on individual data points rather than holistic dataset evaluation. Built on Subjective Logic, the proposed approach enables uncertainty-aware evaluations, effectively quantifying uncertainty in scenarios where evidence is incomplete, distributed, or conflicting. The authors instantiated this framework to evaluate bias and conducted experimental assessments using a traffic sign recognition dataset. Results indicate that the method successfully captures class imbalance while remaining interpretable and robust in both centralized and federated learning contexts. This development offers a significant advancement for ensuring fairness and reliability in AI systems by providing a structured mechanism to evaluate dataset quality amidst complex and uncertain data environments.