Exgraph: A Graph Execution Engine for Task Orchestration

This article introduces exgraph, a graph execution engine designed by Baidu Search to address the challenges of complex task orchestration. The engine is built around three core components: graph description language, graph execution engine, and integration extensions.

The graph description language is a text-based, human-readable format that describes operators and their dependencies within a task. It supports serial execution, parallel execution, interruption, and selection paradigms to meet diverse scenario requirements. The language uses operators as the smallest execution units, serial groups for sequential execution, concurrent groups for parallel execution, and attributes for parameter passing.

The graph execution engine is the core of exgraph, responsible for efficiently executing tasks based on the graph syntax tree generated from the description language. It implements a Job interface with a DoImpl method that takes a context parameter for passing information between operators. The engine includes features like object containers for concurrent-safe data sharing, dependency injection and extraction using struct tags, and built-in interruption and skipping mechanisms.

Exgraph also provides optimization features such as object pooling for operators and objects, and execution optimization where operators run in the current goroutine unless a timeout is specified. The engine supports various scenarios including different logic paths for PC and mobile, and PHP strategy migration to Go with sampling verification.

The article demonstrates how exgraph can simplify complex business logic description, improve code readability, and provide a unified mechanism for handling diverse execution scenarios in search architecture.