Evaluating Large Language Models for Question Answering over Datasets

A new research paper submitted to arXiv investigates the effectiveness of large language models (LLMs) in performing data analytics tasks, specifically answering questions based on datasets. The study evaluates model performance in two distinct scenarios: directly answering questions using a dataset file as input, and generating SQL queries from relational database schemas to retrieve answers. The researchers also assess how different prompting strategies influence outcomes. The experiment compares state-of-the-art LLMs against smaller, more resource-efficient models across two datasets with varying question difficulties. Results indicate that while large LLMs demonstrate strong capabilities in data interpretation and query generation, smaller, cost-effective models exhibit significant limitations. This research provides critical insights into the practical application of AI in data analytics, highlighting the trade-offs between computational cost and analytical accuracy. It contributes to a deeper understanding of how organizations can leverage LLMs for automated data questioning while recognizing the current constraints of lighter-weight models.