Fine-Grained Contradiction Analysis in Scientific Peer Reviews via IMPACT and TIDE
Researchers have introduced a novel approach to address conflicting expert judgments in scientific peer reviews, a growing challenge for conference editors due to increasing submission volumes. Moving beyond traditional binary contradiction detection on isolated sentences, this work proposes a fine-grained analysis framework that operates on full reviews. The team presents RevCI, an expert-annotated benchmark featuring evidence-level contradiction annotations with graded intensity labels. They also introduce IMPACT, a structured multi-agent framework integrating aspect-conditioned evidence extraction, deliberative reasoning, and adjudication to model disagreement intensity. For efficient deployment, IMPACT is distilled into TIDE, a small language model capable of predicting contradiction evidence and intensity in a single forward pass. Experimental results demonstrate that IMPACT significantly outperforms existing single-agent and generic multi-agent baselines in evidence identification and intensity agreement. Meanwhile, TIDE achieves competitive performance at a substantially lower inference cost, offering a scalable solution for managing reviewer disagreements in academic publishing.
Wire timeline
Fine-Grained Contradiction Analysis in Scientific Peer Reviews via IMPACT and TIDE
Researchers have introduced a novel approach to address conflicting expert judgments in scientific peer reviews, a growing challenge for conference editors due to increasing submission volumes. Moving beyond traditional binary contradiction detection on isolated sentences, this work proposes a fine-grained analysis framework that operates on full reviews. The team presents RevCI, an expert-annotated benchmark featuring evidence-level contradiction annotations with graded intensity labels. They also introduce IMPACT, a structured multi-agent framework integrating aspect-conditioned evidence extraction, deliberative reasoning, and adjudication to model disagreement intensity. For efficient deployment, IMPACT is distilled into TIDE, a small language model capable of predicting contradiction evidence and intensity in a single forward pass. Experimental results demonstrate that IMPACT significantly outperforms existing single-agent and generic multi-agent baselines in evidence identification and intensity agreement. Meanwhile, TIDE achieves competitive performance at a substantially lower inference cost, offering a scalable solution for managing reviewer disagreements in academic publishing.
cs.AI updates on arXiv.org