How AutoLink Turns Schema Linking into an Interactive Database Exploration

Introduction

Text‑to‑SQL (also called NL2SQL) aims to let users who do not know SQL query databases using natural language. Large industrial databases often contain hundreds of tables and thousands of columns, making it infeasible to feed the entire schema to a large language model (LLM) because of token explosion, high computation cost, and noisy irrelevant columns.

What is Schema Linking?

Before generating SQL, schema linking extracts the subset of tables and columns that are truly relevant to the user question.

Effective schema linking compresses the full schema into a concise, relevant sub‑graph, reducing noise and token usage.

Limitations of Existing Methods

Database‑level approaches (e.g., MCS‑SQL, SQL‑to‑Schema) input the whole schema at once. They suffer from:

Unscalable token length for large databases.

Need for multiple sampling or decoding rounds, which dramatically increase cost.

Rapid performance degradation as schema size grows.

Element‑level approaches (e.g., dual‑encoder, cross‑encoder, CHESS) evaluate each table/column independently. Their drawbacks include:

Linear computational complexity with schema size.

Requirement to keep many candidate columns, re‑introducing noise.

Inherent trade‑off between high recall and low noise.

Human Engineer Insight

When faced with an unfamiliar database, engineers do not read the whole schema at once. Instead, they iteratively hypothesize a small set of potentially relevant tables/columns, verify them with simple SQL or metadata queries, discover missing pieces, and repeat until the schema is sufficient to answer the question.

AutoLink Motivation

Schema linking should be an autonomous, progressive exploration rather than a one‑shot filter.

AutoLink adopts this human‑like workflow by starting from a small, incomplete schema and letting an LLM‑driven agent iteratively expand it.

Core Agent Actions

@explore_schema : Execute exploratory SQL (e.g., sample values, inspect metadata) to gather concrete evidence about columns.

@retrieve_schema : Perform semantic vector‑store retrieval to obtain “virtual” column names or abstract concepts missing from the current schema.

@verify_schema : Construct minimal executable SQL to test whether the candidate schema can answer the question; error messages become precise diagnostic signals.

@add_schema : Add newly validated tables/columns to the candidate set.

@stop : Terminate when the schema is deemed sufficient or a maximum number of rounds is reached.

Algorithmic Flow

The agent receives:

Instruction prompt defining the goal and available actions.

User question.

List of all table names in the database.

An initial schema retrieved by a single semantic query using the user question (e.g., top‑k = 50‑100 columns).

At each round the agent:

Reasons over the current history to decide whether the schema is sufficient.

If not, selects one or more actions (explore, retrieve, verify).

Executes the actions in the appropriate environment (database or vector store).

Updates the history with observations and adds newly discovered schema elements.

The process stops when @stop is issued or a preset round limit (e.g., 10) is reached. The final schema is the union of the initial schema and all @add_schema additions.

Experimental Evaluation

AutoLink was evaluated on two benchmarks:

Spider 2.0‑Lite : average >800 columns, max >3000 columns.

BIRD : average 80 columns.

Key results:

On Spider 2.0‑Lite, AutoLink improves strict recall (SRR) by 27.2 % over the next‑best SQL‑to‑Schema method while reducing maximum token consumption by 87.7 %.

Even when recalling a similar number of columns, AutoLink’s SRR is ~40 % higher than dual‑encoder / cross‑encoder baselines.

Methods that rely on full schema or multi‑round sampling (e.g., MCS‑SQL, RSL‑SQL) degrade sharply on large databases, whereas AutoLink maintains high SRR.

In SQL generation, AutoLink achieves competitive execution accuracy (EX): 34.92 % on Spider 2.0‑Lite with DeepSeek‑R1 (token usage 38 K, < 50 % of ReFoRCE) and 68.71 % on Bird‑Dev with Gemini‑1.5‑pro, surpassing CHESS and RSL‑SQL.

Ablation studies confirm that @retrieve_schema, @explore_schema, and @verify_schema each contribute positively, with retrieval being the most critical.

Scalability analysis shows that while all methods’ SRR drops as column count grows, AutoLink’s decline is far slower, retaining ~90 % SRR on databases with >3000 columns.

Key Insights

Higher SRR correlates strongly with higher SQL execution accuracy; incomplete schemas inevitably lead to erroneous SQL regardless of model hallucination mitigation. AutoLink’s iterative expansion yields a high‑recall, low‑noise schema that enables robust downstream SQL generation, especially in large‑scale industrial settings.

Conclusion

AutoLink reframes schema linking as an interactive, agent‑driven process that combines semantic retrieval with lightweight SQL probing. This design delivers superior recall‑cost trade‑offs, strong scalability, and improved downstream SQL performance, offering a practical solution for industrial Text‑to‑SQL applications.

-- Check sample values containing specific keywords (e.g., "INTOX")
SELECT DISTINCT descript
FROM incidents_2016
WHERE descript LIKE '%INTOX%'
LIMIT 5;

-- Filter columns by dual conditions (table attributes + column semantics) via metadata
SELECT column_name
FROM census.INFORMATION_SCHEMA.COLUMNS
WHERE table_name LIKE '%tract%'
  AND table_name LIKE '%2018%'
  AND LOWER(column_name) LIKE '%income%'
LIMIT 5;