Evaluating AI Tools in Academic Research: Useful for Exploration, Risky for Precision

A new study published on arXiv evaluates the integration of artificial intelligence tools into scientific research workflows, specifically focusing on question-answering and literature review applications. The research highlights a significant gap in existing benchmarking methods, which often overlook human-centered criteria like usability and interpretability. To address this, the authors propose a framework combining human-centered and computer-centered metrics. Findings indicate that while AI tools provide valuable overviews and enhance efficiency in early research stages, they are unreliable for precise information extraction. A critical issue identified is the low accuracy of Explainable AI features, where highlighted sources frequently do not match generated answers, thereby shifting the validation burden back to researchers. Furthermore, literature review tools demonstrated low reproducibility and transparency, making them unsuitable for systematic reviews. The study concludes that while AI aids exploratory tasks, rigorous human verification remains essential. It underscores the need for improved explainability and careful integration of these tools to ensure practical applicability and maintain research integrity.