Qwen-Based RAG System Achieves High Accuracy in Ukrainian Document Understanding Task
Researchers have developed a novel retrieval-augmented generation (RAG) pipeline for the Fifth UNLP shared task, focusing on multi-domain document understanding in Ukrainian. The system, titled 'Qwen Goes Brrr,' utilizes an off-the-shelf approach built around three core strategies: contextual chunking of PDFs, question-aware dense retrieval with reranking conditioned on answer options, and constrained answer generation. The architecture leverages Qwen3 models, specifically Qwen3-Embedding-8B for retrieval, a fine-tuned Qwen3-Reranker-8B for passage ranking, and Qwen3-32B for final answer selection. Experimental results demonstrate significant performance improvements, with reranking boosting Recall@1 from 0.6957 to 0.7935. Furthermore, using the top-two reranked passages increased answer accuracy from 0.9348 to 0.9674. The team achieved scores of 0.9452 on the public leaderboard and 0.9598 on the private leaderboard. The study concludes that preserving document structure and incorporating answer-space awareness into relevance estimation is more effective than complex downstream heuristics under strict competition constraints.
Wire timeline
Qwen-Based RAG System Achieves High Accuracy in Ukrainian Document Understanding Task
Researchers have developed a novel retrieval-augmented generation (RAG) pipeline for the Fifth UNLP shared task, focusing on multi-domain document understanding in Ukrainian. The system, titled 'Qwen Goes Brrr,' utilizes an off-the-shelf approach built around three core strategies: contextual chunking of PDFs, question-aware dense retrieval with reranking conditioned on answer options, and constrained answer generation. The architecture leverages Qwen3 models, specifically Qwen3-Embedding-8B for retrieval, a fine-tuned Qwen3-Reranker-8B for passage ranking, and Qwen3-32B for final answer selection. Experimental results demonstrate significant performance improvements, with reranking boosting Recall@1 from 0.6957 to 0.7935. Furthermore, using the top-two reranked passages increased answer accuracy from 0.9348 to 0.9674. The team achieved scores of 0.9452 on the public leaderboard and 0.9598 on the private leaderboard. The study concludes that preserving document structure and incorporating answer-space awareness into relevance estimation is more effective than complex downstream heuristics under strict competition constraints.
cs.AI updates on arXiv.org