RAQAG : A framework for automatically generating Q&A datasets with retrieval-augmented generation

doi:10.1016/j.knosys.2026.115842

RAQAG : A framework for automatically generating Q&A datasets with retrieval-augmented generation

2026-04-01

0

PRE

AI

DOI：10.1016/j.knosys.2026.115842

Original

Original request for help

Save

Abstract

En 中文

Objectives: Large Language Models (LLMs) necessitate high-quality Question-Answer (QA) datasets for fine-tuning and evaluation, but current resources are constrained by limited scale, diversity, and structured annotations. This scarcity is especially pronounced in business and management research (e.g., UTD24 journal domains like accounting, finance, marketing, strategy, and operations), where domain-specific QA datasets are rare beyond finance-focused ones. Methods: To bridge this gap, we present RAQAG, a Retrieval-Augmented Question Answer Generation framework that constructs QA datasets through a closed-loop process involving data synthesis, quality verification, and feedback optimization. RAQAG consists of three core modules: data preprocessing, QA generation, and QA evaluation. The QA generation module utilizes a dynamic retrieval-augmented generation approach, incorporating a three-tier retrieval strategy (semantic vector search, BM25 retrieval, and BGE Reranker) to support dynamic paragraph segmentation and enhance document grounding. The QA evaluation module employs RACAR, a multi-dimensional evaluation framework that rigorously defines five core dimensions: Relevance, Agnosticism, Completeness, Accuracy, and Reasonableness, along with dynamic indicators like evidence coverage. Novelty: RAQAG presents a novel framework that integrates LLMs with a closed-loop process for generating high-quality QA datasets. RAQAG enables the creation of large-scale, domain-specific QA corpora, e.g., UTD24 _ QA, which includes 170,226 instruction-style QA pairs derived from 17,542 articles. A standardized multiple-choice benchmark, UTD24 _ QA Benchmark, is developed for systematic evaluation of LLMs in business and management knowledge. Findings: Experiments demonstrate that RAQAG produces higher-quality QA datasets, achieving a 9.45% improvement in RACAR scores compared to the best existing baseline. The framework exhibits robust performance across varying input granularities and hyperparameter settings. The engineering-oriented QAG system, featuring metadata-driven orchestration and visualized pipelines, further validates the effectiveness of RAQAG. The source code is publicly available at https://github.com/Fakerfigure/QAG_System.git .

Keywords:

RAQAG

Question-Answer generation

Retrieval-Augmented Generation

Large Language Models

QA dataset construction

RAQAG : A framework for automatically generating Q&A datasets with retrieval-augmented generation

Abstract

Journal

Organization