From Workflow to HermesAgent: Language‑Driven Image Selection for Gaode Street Rankings

Background

Gaode’s street‑ranking product relies on a huge POI image library, and the visual presentation directly influences user trust. The legacy pipeline stitched together over 50 SQL files, eight execution steps, and prompt‑based LLM calls, leading to difficult maintenance, slow iteration, heavy manual intervention, and limited ability to address specific bad cases.

Overall Architecture

The new system adopts a mixed architecture: deterministic workflow nodes handle high‑throughput, parallelizable tasks (e.g., CLIP recall, aesthetic filtering, duplicate detection), while an LLM‑powered ReAct Agent handles “clever” tasks that require rule understanding and dynamic composition. Hermes provides a natural‑language interface for orchestration.

Five‑Layer System Design

1. Interaction Layer : Hermes offers a dialogue entry point for querying, re‑ranking, and modifying images via natural language.

2. Orchestration Layer : A stable Python pipeline executes the deterministic steps; Hermes schedules and invokes Agent actions without sacrificing reliability.

3. Node Layer : Core nodes include query generalization, multi‑path recall, coarse ranking, fine ranking, fallback, and quality inspection.

4. MCP/Skill Layer : Each capability is encapsulated as an independent Skill callable from Python, Bash, or Hermes.

5. Atomic Capability Layer : Includes POI‑CLIP recall, aesthetic scoring, OCR, rotation detection, face/person detection, duplicate detection, and domain‑specific multimodal models.

Full Six‑Step Production Funnel

1. Data Preparation : Images are gathered from POI galleries, user‑submitted photos, and wall‑of‑photos.

2. Recall & Admission : Atomic capabilities filter out obviously unsuitable images.

3. Coarse Ranking : A composite score over relevance, distinctiveness, and aesthetics orders candidates; two additional LLM re‑rankings refine the list.

4. Fine Ranking (Agent‑Based) : The Agent interprets theme‑specific selection rules, combines classified candidates, and picks the three final images.

5. Fallback : If fewer than three images remain, higher‑level candidates are reused and a VLM selects a single fallback image based on the ARAD criteria (Aesthetics, Relevance, Authenticity, Diversity).

6. Quality Inspection & Release : Four dimensions—basic quality, visual duplication, theme relevance, and anti‑seasonality—are checked before writing results to the downstream table.

Key Technology 1: Agent‑Based Fine Ranking

The fine‑ranking stage moves beyond simple score thresholds. It splits aesthetic evaluation into three domains (general, advanced, domain‑specific) across fifteen dimensions, applying a two‑step process of gate‑keeping and compensation to ensure both baseline quality and highlight amplification.

Rule configurations are externalized into three RAG tables (custom, generic, default), allowing new themes to be added without code changes. When no configuration matches, the system generates dynamic query terms (e.g., “门头”, “环境”) via LLM.

The Agent then autonomously selects images by iteratively invoking tools to interpret rules, perform de‑duplication, and satisfy custom constraints, producing a stable, explainable combination.

Key Technology 2: Fallback & Quality Inspection

Fallback reuses higher‑level candidates to avoid recomputing from raw images, preserving computational resources and leveraging already‑validated results. The ARAD standard guides VLM‑driven selection, and only one fallback image is added to maintain a conservative quality baseline.

Quality inspection checks four aspects: basic image quality, visual duplication, thematic relevance, and seasonality, each with concrete heuristics (e.g., blur detection, duplicate angle filtering, seasonal tag mismatch).

Key Technology 3: Multimodal Knowledge Base

Because generic VLMs lack detailed business knowledge, a multimodal knowledge base stores “text description + image example + source evidence” for each POI. Text side captures local practices, generic methods, and seasonal facts; image side clusters candidates and validates consistency via VLM‑based image‑text matching.

This knowledge base enables two benefits: “select the right one from wrong ones” (disambiguating similar images) and “choose the best among correct ones” (optimizing for visual appeal).

Key Technology 4: Language‑Driven Production

Hermes exposes a dialogue interface that parses natural‑language requests, translates them into executable rules, triggers the appropriate Skills, and writes back results. The system ensures every Agent‑initiated action is observable, configurable, and rollback‑able, with full monitoring of inputs, intermediate states, and final outputs.

Engineering Practice

Real‑time production is achieved with Bash scripts, Python concurrency, and parallel Skill execution, shrinking per‑ranking latency from 24 hours to 30 minutes (48× speedup). Comprehensive ODPS‑based monitoring tracks POI counts, image counts per stage, success rates, and quality metrics, enabling rapid A/B testing and safe rollouts.

Operational challenges addressed include exponential retry storms (mitigated by exponential backoff with jitter), VLM numerical instability (model outputs raw dimensions, code aggregates), structured‑output validation (whitelist checks), compute capacity limits (token and model selection optimization), and session consistency across machines (consistent hashing and distributed state).

Conclusion & Outlook

The redesign demonstrates that deterministic pipelines and Agent reasoning can coexist, that structured multi‑dimensional scoring outperforms black‑box scores, and that clear separation of model perception and code computation yields robust production. Future work will integrate AIGC for image generation, advance self‑evolving Agent loops, and move the pipeline toward fully online KV stores for sub‑minute latency.