The Silent Vote: Improving Zero-Shot LLM Reliability by Aggregating Semantic Neighborhoods
Researchers have introduced a novel method called Semantic Softmax to address reliability issues in Large Language Models (LLMs) used for zero-shot classification. The study identifies a phenomenon termed Renormalization Bias, where standard constrained decoding discards probability mass assigned to semantic synonyms, leading to artificial overconfidence and poor calibration. This loss of information is referred to as the Silent Vote. To mitigate this, the proposed Semantic Softmax layer aggregates scores from the semantic neighborhood of target labels during inference, effectively recovering lost linguistic nuances. The approach was evaluated on Qwen-3 and Phi-4-mini models using the GoEmotions and Civil Comments datasets. Results demonstrated consistent improvements across key metrics, including substantial reductions in Expected Calibration Error (ECE) and Brier Score, alongside enhanced discriminative performance in AUROC and Macro-F1. This research offers a more calibrated and accurate alternative for zero-shot classification tasks by accounting for subtle linguistic variations, thereby improving the overall trustworthiness and performance of LLMs in complex reasoning scenarios without requiring additional training data.
Wire timeline
The Silent Vote: Improving Zero-Shot LLM Reliability by Aggregating Semantic Neighborhoods
Researchers have introduced a novel method called Semantic Softmax to address reliability issues in Large Language Models (LLMs) used for zero-shot classification. The study identifies a phenomenon termed Renormalization Bias, where standard constrained decoding discards probability mass assigned to semantic synonyms, leading to artificial overconfidence and poor calibration. This loss of information is referred to as the Silent Vote. To mitigate this, the proposed Semantic Softmax layer aggregates scores from the semantic neighborhood of target labels during inference, effectively recovering lost linguistic nuances. The approach was evaluated on Qwen-3 and Phi-4-mini models using the GoEmotions and Civil Comments datasets. Results demonstrated consistent improvements across key metrics, including substantial reductions in Expected Calibration Error (ECE) and Brier Score, alongside enhanced discriminative performance in AUROC and Macro-F1. This research offers a more calibrated and accurate alternative for zero-shot classification tasks by accounting for subtle linguistic variations, thereby improving the overall trustworthiness and performance of LLMs in complex reasoning scenarios without requiring additional training data.
cs.AI updates on arXiv.org