Lightweight Ontology Practice: Building AI Agents for Big Data Platform Operations

Big data platforms such as Hadoop, Hive, Spark, Flink, YARN, and Ranger generate hundreds of daily alerts and require cross-component troubleshooting that traditionally relies on expert experience, scripts, and manual document lookup. This creates challenges: knowledge is scattered across wikis and engineers' minds, troubleshooting chains span multiple systems taking hours, alert fatigue buries real signals, monitoring lacks semantic context, and each new cluster demands re-accumulating experience.

The article argues that ontology bridges the semantic gap for large language models by structuring operations concepts (slow task, DAG failure, memory overflow) and their causal relationships into a machine-reasonable knowledge base. Unlike pure RAG, which retrieves text fragments and cannot traverse structured chains like Application → DAG → Task → Container, ontology provides explicit relationship modeling. Unlike property graphs, OWL ontologies based on Description Logic support logical inference (e.g., transitive dependency discovery). Unlike unconstrained SQL agents that guess tables and columns, ontology adds a semantic mapping layer (mapping.yml) linking business concepts to physical schemas.

A full ontology platform offers semantic understanding, cross-component correlation, explainability (each reasoning step anchored to ontology rules), and reusable knowledge assets. However, it requires 1–2 weeks for deployment (server, database, network, permissions), 3–5 days to build a 50–100 object ontology (though a runnable POC can be done in one day with existing docs and DDLs), ongoing maintenance for version upgrades, and a steep learning curve for its GUI.

The lightweight solution extracts core capabilities as six Skills and an MCP service within the WorkBuddy "Ontology Modeling & Reasoning Full-Stack Expert" package, organized in four stages:

File & empty ontology management (ontology-files) : list, create, upload/download .ttl/.owl/mapping/.db files, status/flush/reload.

Content modeling CRUD (ontology-crud) : create/read/update/delete object types, properties, relationships, Actions, Logic rules; apply changes; visualize subgraphs/class diagrams via Mermaid; run reason_check for logical validation.

Build from documents or DDL : document-ontology-update-lite extracts from .md/.txt docs with user confirmation (supports incremental updates); ddl-ontology-build-lite deterministically maps CREATE TABLE statements to OWL/Turtle without LLM interpretation.

Publish & reasoning : ontology-lifecycle-lite publishes immutable, vectorized, AOR-queryable versions with version list and rollback;

aor-lite</strong> performs verifiable agentic reasoning (Scope, ObjectSet, relationship paths, read-only evidence tools) and can execute <code>data_execution

against physical data sources.

The practice loop uses a Hive operations ontology (tested on a single cluster with 10k-scale tasks): install the expert package (pre-configured internal endpoints), build ontology from docs/DDLs, upload mapping.yml to bind ontology objects to physical tables, publish the version, then ask natural-language questions. The agent parses intent via ontology semantics, traverses object relationships, generates SQL through mapping, queries read-only, and returns structured results with full traceability (object matches, rule hits, SQL). Images in the source illustrate mapping.yml template generation, data-model-guided ontology enrichment, and the agent's explicit "not covered" response when a component is missing, followed by guidance on how to monitor it.

Applicable scenarios: historical fault analysis and root cause (non-real-time), performance trend and metric queries, configuration compliance checks, knowledge retrieval and onboarding, cross-component cascade troubleshooting (e.g., Hive → YARN → HDFS). Not suitable for: millisecond-level real-time alert handling (1–3 s reasoning latency), automated write operations (restart, config changes), multi-heterogeneous-cluster faults (current ontology covers single-cluster component relations), or unmodeled new components/metrics (require ontology extension first).

Safeguards: agent replies "current ontology does not cover this scenario" instead of hallucinating; data_execution is read-only; every reasoning step is auditable. Production readiness checklist: role-based access control (read-only for operators, write for admins), audit logs for data_execution to prevent data leakage, stress testing for concurrency and latency, and a versioned ontology update process aligned with component upgrades.

The conclusion emphasizes that enterprise-grade agents succeed not by larger models but by controllable semantic boundaries: turning tacit operational experience into computable, reasonable, auditable explicit assets — moving from probabilistic retrieval to semantic constraint so every agent decision is traceable and verifiable.