Boosting Socio‑Economic Q&A: The ARAG Framework Merges Structured Data Analysis with RAG

Background

Large language models (LLMs) such as GPT, ERNIE‑Bot and Gemini have dramatically improved question‑answering, especially for data‑analysis and information‑retrieval tasks. Professional economic and urban‑planning queries (e.g., growth attribution, indicator interpretation, site‑selection) still require simultaneous reasoning over structured statistical tables and unstructured textual sources, which existing solutions handle separately.

Key Technical Challenges

Separate deployment : Current LLM‑driven analysis modules and Retrieval‑Augmented Generation (RAG) modules are isolated, preventing joint “structured + unstructured” reasoning.

Data‑format incompatibility : Analysis pipelines accept only tabular data, while RAG pipelines accept only free‑text, leaving multi‑format queries unsupported.

Hallucination : When user queries contain inaccurate assumptions, systems often generate factually incorrect answers because they lack robust fact‑checking.

ARAG Framework (Adaptive Retrieval‑Augmented Generation)

ARAG tightly couples an LLM‑driven data‑analysis module with a RAG module. The workflow consists of three stages: preprocessing, dual‑module collaboration, and result integration.

1. Preprocessing – Entity Extraction

The LLM parses the user query and extracts three entity types that guide downstream processing:

Geographic entity (e.g., “Beijing”)

Temporal entity (e.g., “2023Q1”)

Target entity (e.g., “hot‑pot restaurant”)

2. Data Analysis Module

This module quantifies the relationship between a primary indicator (specified by the user) and its influencing indicators (IF‑indicators) . The procedure includes:

Primary‑indicator matching

Step 1 – Vector search : Encode all column names in the statistical database with a dense embedding model and retrieve the top‑3 most similar names to the query term.

Step 2 – LLM selection : Prompt the LLM to choose the best match, resolving terminology gaps (e.g., “GDP” ↔ “Gross Domestic Product”).

Indicator change analysis

Compute the primary indicator’s trend over the user‑specified time window (e.g., YoY GDP growth).

Identify IF‑indicators from a pre‑constructed knowledge graph that links economic metrics to policy or market factors. The graph is built offline by extracting indicator relations from government reports and news.

Generate Python code that loads the relevant columns, aligns timestamps, and calculates the percentage change of each IF‑indicator.

Alignment check (hallucination mitigation)

Fact alignment : Verify that the computed changes are consistent with the query intent (e.g., if the user asks about “growth” but the data shows a decline, the system flags the discrepancy).

Relevance alignment : Ensure that each IF‑indicator’s variation is causally related to the primary indicator, discarding unrelated metrics.

# Example Python snippet generated by the LLM
import pandas as pd

def compute_if_changes(df, primary_col, if_cols, start, end):
    # Filter by time window
    mask = (df['date'] >= start) & (df['date'] <= end)
    sub = df.loc[mask]
    # Primary indicator change
    primary_change = (sub[primary_col].iloc[-1] - sub[primary_col].iloc[0]) / sub[primary_col].iloc[0]
    # IF‑indicator changes
    if_changes = {}
    for col in if_cols:
        change = (sub[col].iloc[-1] - sub[col].iloc[0]) / sub[col].iloc[0]
        if_changes[col] = change
    return primary_change, if_changes

3. RAG Module

The RAG component retrieves unstructured documents (news articles, policy reports) that provide contextual evidence for the analysis.

Time‑sensitive retrieval : Filter the document corpus by publication date to keep only items within the query’s temporal window, then perform vector similarity search between the query embedding and document‑segment embeddings.

Multi‑criteria filter : Retain a segment only if it satisfies all three conditions:

Indicator relevance – the text mentions the primary indicator.

Temporal alignment – the mentioned time period matches the query.

Geographic relevance – the location matches the extracted geographic entity.

Source‑referenced summary : Prompt the LLM to produce a concise summary that explicitly cites each source’s title and URL, guaranteeing traceability and limiting hallucination.

Performance Evaluation

Evaluation used the G‑Eval protocol (four dimensions scored 0–5) on 50 queries, each evaluated five times. Compared systems were ChatGPT‑4o Search and Perplexity.

Domain relevance, temporal relevance, information richness, analysis depth : ARAG achieved higher average scores on all four dimensions.

Ablation study : Removing the RAG component reduced analysis depth sharply; removing the data‑analysis component lowered domain relevance and information richness, confirming that both modules are essential.

Hallucination test : On 20 deliberately erroneous queries (e.g., “Why did Beijing’s GDP decline?” when it actually rose), ARAG detected all 20 errors, whereas ChatGPT‑4o Search detected 8 and Perplexity detected none.

Representative Application

ARAG enables spatiotemporal big‑data Q&A such as “Where in Beijing should I open a hot‑pot restaurant to maximize profit?” The system combines quantitative metrics (population density, traffic flow, rental prices) from structured databases with up‑to‑date policy and news information retrieved by the RAG module, producing a high‑confidence, low‑hallucination recommendation.