Full-Stack Evolution of a Game Data Analysis Agent

Background and Challenges

Game analytics demand deep domain knowledge and sophisticated data‑processing skills, creating high entry barriers. The project aimed to combine personal game‑business experience with LLM capabilities to help colleagues extract insights, but faced several technical hurdles:

Understanding extensive game‑specific jargon and unique analyst requests.

Reconciling LLM output uncertainty with the deterministic nature of data queries.

Enabling the agent to perform autonomous game‑data analysis.

Implementing fine‑grained permission controls for enterprise‑level data security.

Architecture Evolution

Three major versions were built and iterated:

v1.0 (LangChain Chain) : Linear chain execution; unable to handle follow‑up questions or complex error recovery.

v2.0 (Dify Workflow) : Low‑code visual workflow; suffered from scalability limits, hard‑to‑debug logic, and restricted customization.

v3.0 (LangGraph) : Introduced graph‑based execution with state and custom nodes, allowing flexible token management, modular middleware, and robust debugging.

Each version’s goals, tool selections, and failure analyses are documented, showing why earlier approaches were abandoned.

Core Engineering Practices

Key concepts of LangGraph were defined:

Graph : Execution flow of agents.

State : Shared data structure that evolves with each step.

Node : Individual logic unit.

Edges : Define data flow between states.

Threads : Conversation sessions.

Runs : Single agent execution.

Detailed design includes:

1. Overall Graph Design

Three parallel pipelines handle simple replies, fixed‑query tasks, and exploratory analysis loops. The third pipeline uses a TODO planner combined with a ReAct loop to balance token usage and modularity.

2. Knowledge Integration

Static knowledge (e.g., ODPS SQL syntax) is hard‑coded in system prompts using structured prompts. Dynamic knowledge is stored in a RAG‑backed knowledge base managed by Dify, covering game aliases, base info, metric definitions, terminology, and historical query cases. Knowledge preprocessing follows a feature‑engineering‑style pipeline: data cleaning, reshaping (pivot), and formatting.

3. Prompt Construction

User prompts are XML‑structured into sections such as <UserQuestionBackground>, <GameInfo>, <MetricReference>, <TODOState>, <ExecutionHistory>, and <RecentReflection>. Prompts are regenerated each loop to conserve tokens, with message compression applied when the message queue exceeds thresholds.

4. ReAct Architecture

The agent loop follows a middleware pattern where each lifecycle hook (before_model, after_model, etc.) can modify state or messages. Middleware examples include BaseAnalysisMiddleware, tool‑specific middleware, and MessageCleanupMiddleware.

5. Tool Design

Important tools and their context handling: search_available_tables_tool: Searches tables with permission‑aware filters. table_info: Retrieves schema, partition, and preview rows; injects results as XML. table_lineage_relationships: Provides upstream lineage via OpenAPI. sql_tool: Executes SQL, registers DataFrames in a data layer, and stores summaries in state. think_tool: Forces a reflection step. notebook_tool: Persists intermediate conclusions. code_utils: Executes sandboxed Python for data analysis.

Tool selection follows research showing that limiting the number of tools improves LLM performance.

6. Token Optimization

Full‑stack token observability.

Syntax‑level token compression (e.g., simplifying markdown tables).

Context pruning after each loop.

Hybrid compression based on time decay and semantic relevance.

Prefix caching to leverage LLM response caching.

7. Security – Prompt‑Level Permission Control

Two permission models are supported:

Authenticated user via RBAC, automatically fetching allowed tables and games.

Custom ak/sk credentials for higher‑privilege queries.

SQL validation uses sqlglot to parse into an AST, then applies strict checks:

Table‑level whitelist against user permissions.

Mandatory game_id filter for row‑level security.

Detection of bypass patterns such as OR 1=1 or fuzzy LIKE conditions.

Automatic safe rewriting to inject missing filters.

White‑list of allowed AST node types to prevent obscure constructs.

Comprehensive unit tests cover edge cases.

8. Deployment and Observability

The final system runs entirely inside the corporate intranet:

Agents service behind a BFF that handles internal authentication.

Dify provides knowledge‑base and lightweight agent UI.

Custom communication plugin translates LangGraph event streams to Dify SSE.

Self‑built operation console offers state hot‑patching, memory visualization, and stateless safety.

Distributed scheduler (Node.js workers + Postgres) replaces missing Cron support.

Web UI visualizes the ReAct chain, supports CSV‑to‑ECharts rendering, and includes watermarks for data protection.

Network design isolates services within Docker, uses an internal Nginx gateway, and enforces strict token limits and timeouts for long‑running AI calls.

Conclusion and Future Work

The project demonstrates a reproducible, secure, and extensible approach to building LLM‑driven game‑data analysis agents. Planned improvements include rewriting with the latest frameworks, further lowering the usage barrier, expanding skill libraries, enhancing shared memory across projects, and refining human‑in‑the‑loop workflows.