Intelligent Duty Robot for Real‑Estate Data Job Monitoring and Automation

The real‑estate data platform faces complex, multi‑source, long‑process data tasks, making timely and accurate data delivery a major challenge. To address this, an intelligent duty robot was developed to ensure reliable, on‑time data services while significantly easing operational workload.

Background – 58房产 & 安居客 provide critical data services to end users, agents, and internal teams. Data originates from numerous sources (listings, user behavior, third‑party data) and passes through many processing layers, creating deep dependency chains where any delay or error can impact final output.

Solution – The robot follows a Sense‑Think‑Act architecture. Sense gathers job configurations, dependencies, current status, logs, and historical profiles. Think builds a job dependency graph, computes depths, identifies which jobs need monitoring, estimates leaf‑job completion times, and decides whether anomalies are self‑healing or require human intervention. Act triggers actions such as job retries via the scheduler or phone notifications to owners.

The dependency graph is constructed by iteratively adding data‑service jobs and their upstream dependencies, assigning parent‑child relationships, and calculating each job’s depth (root depth = 1, otherwise max(parent depth)+1). Monitoring logic traverses unfinished jobs at the current depth, checks parent completion, and adds eligible jobs to a watch list.

When anomalies are detected, the robot classifies them: self‑healing issues (e.g., temporary external resource failures) trigger retries or rescheduling; non‑self‑healing issues (e.g., SQL syntax errors) generate immediate alerts to responsible engineers. The system also estimates job finish times to detect overall pipeline delays and identifies the critical path causing the delay.

Effect – Prior to deployment, operators manually checked jobs during off‑hours, often missing delays. After launch, alerts are automatically sent only for jobs that affect final data delivery, reducing noise and operational burden. In three months, notification accuracy and coverage exceeded 98%.

Conclusion & Future Work – By leveraging task dependency graphs, anomaly analysis, and execution‑time estimation, the robot resolves monitoring blind spots, late‑delay detection, and lengthy diagnosis, ensuring stable, reliable data services while simplifying troubleshooting.