Using ANTLR4 for SQL Parsing, Completion, and Validation in SparkSQL-based Data IDE

The article describes the design and implementation of a SQL parsing engine based on ANTLR4 for a data development IDE used in a large‑scale offline data platform. It explains the background, the need for an open‑source, controllable ecosystem, and the choice of ANTLR4 as the core parsing technology.

ANTLR4 Overview

ANTLR4 is a powerful parser generator that can read, process, and translate structured text. Its key features include expressive grammar definitions, automatic AST generation, error handling, extensibility, and a rich tool ecosystem.

Application Scenarios

Apache Spark uses ANTLR as part of its SQL parser.

Twitter employs ANTLR for query language analysis.

IBM integrates ANTLR for DSL parsing in enterprise solutions.

Getting Started with ANTLR4

// MyGLexer.g4
lexer grammar MyGLexer;
SEMICOLON: ';';
LEFT_PAREN: '(';
RIGHT_PAREN: ')';
COMMA: ',';
DOT: '.';
LEFT_BRACKET: '[';
RIGHT_BRACKET: ']';
LEFT_BRACES: '{';
RIGHT_RACES: '}';
EQ: '=';
FUNCTOM: 'FUNCTION';
LET: 'LET';
CONST: 'CONST';
VAR: 'VAR';
IF: 'IF';
ELSE: 'ELSE';
WHILE: 'WHILE';
FOR: 'FOR';
RETURN: 'RETURN';

// MyGParser.g4
parser grammar MyGParser;
options { tokenVocab=MyGLexer; }
program: statement* EOF;
statement: variableDeclaration | functionDeclaration | expressionStatement | blockStatement | ifStatement | whileStatement | forStatement | returnStatement | ...;

SQL Grammar Example

lexer grammar SqlLexer;
COMMA: ',';
LEFT_PAREN: '(';
RIGHT_PAREN: ')';
IDENTIFY: (LETTER | DIGIT | '_' | '.')+;
fragment DIGIT: [0-9];
fragment LETTER: [A-Z];
SEMICOLON: ';';

parser grammar SqlParser;
program: statment* EOF;
statment: queryStatment SEMICOLON?;
queryStatment: SELECT columnNames FROM (tableName | '(' queryStatment ')') whereExpression? relationsExpresssion? SEMICOLON?;
columnNames: columnName (COMMA columnName)*;
tableName: IDENTIFY AS? tableAlis;
... // other rules omitted for brevity

The article then details how the parser is integrated into the IDE to provide real‑time syntax analysis, auto‑completion, and validation. It outlines the workflow:

Obtain the parser instance.

Locate the token at the caret position.

Generate the AST.

Collect contextual information via listeners.

Use ANTLR‑C3 to retrieve candidate rules and keywords for completion.

public getSuggestionAtCaretPosition(sqlContent: string, caretPosition: CaretPosition, preferredRules: Set): Suggestions | null {
  const sqlParserIns = new SqlParse(sqlContent);
  const charStreams = CharStreams.fromString(sqlContent);
  const lexer = new SqlLexer(charStreams);
  const tokenStream = new CommonTokenStream(lexer);
  tokenStream.fill();
  const allTokens = tokenStream.getTokens();
  let caretTokenIndex = findCaretToken(caretPosition, allTokens);
  const parseTree = sqlParserIns.program();
  const tableEntity = getTableEntitys();
  const core = new CodeCompletionCore(sqlParserIns);
  core.preferredRules = preferredRules;
  const candidates = core.collectCandidates(caretTokenIndex, parseTree);
  // process candidates to produce rules, keywords, and tableEntity
  return { rules, keywords, tableEntity };
}

It also discusses listener implementation for context collection, showing how to capture query statements, table names, and aliases by extending the generated listener classes.

export class SqlEntityCollector implements SqlParserListener {
  enterQueryStatment(ctx) { this.pushQueryStmt(ctx); }
  exitQueryStatment(ctx) { this.popQueryStmt(); }
  exitTableName(ctx) { this.pushTableEntity(ctx); this.setCurrentTable(ctx); }
  exitTableAlias(ctx) { this.pushTableEntity(ctx); }
  // other helper methods omitted
}

Further sections cover syntax validation, error recovery strategies, performance considerations (caching tokens/parse trees, grammar simplification, prediction mode selection), and integration with the Monaco editor. The article concludes with a discussion on applying large language models (NL2SQL) for intelligent code assistance and future directions such as deeper ANTLR grammar refactoring and model integration.