How to Transform a T+1 Dashboard into Real‑Time T+0 with MQ and MongoDB

Background

The team is building a user data dashboard that aggregates behavior data from many sources (food, daily life, recharge, travel, communications, logistics, medical, housing, sports, etc.). Because arbitrary queries, sorting, and statistics on raw tables are impractical, the previous approach fetched the previous day's data at midnight and stored it as JSON objects in MongoDB.

The initial design used a T+1 strategy to reduce real‑time computation and ensure data accuracy. However, users requested real‑time data, especially for end‑of‑month consumption verification, and waiting until the next month’s first day for reports was considered a poor experience.

Optimization Plan

Based on the prototype requirements and data presentation, an architecture of (MongoDB data service – API service – frontend page) was formed, still providing data with a T+1 strategy.

The goal is to let users efficiently browse their behavior data and receive concrete analysis and suggestions.

The original workflow scheduled a service at 1 am daily via a control‑panel console, processing incremental logs of user additions and modifications.

The optimization target is to change this to a real‑time model: every modification, deletion, or addition of user behavior data triggers a message‑queue notification, the service consumes the message immediately, and updates the MongoDB records instantly.

T+0 Service Overview Design

Core Function Implementation

1. After a user behavior record is saved, a MQ message is sent in real time, decoupling data persistence from dashboard generation.

2. An independent service consumes the MQ, aggregates dashboard data, generates user behavior reports, and pushes notification messages to users.

Data Service Generation Process

Sequence / Flow Diagram Explanation

1. Previously, an independent service performed daily midnight data calculations; now each user behavior change sends an MQ message.

2. The server program listens to the MQ; upon receiving data, it invokes the scheduling service for processing.

3. The scheduling service follows configured rules, starts the console service, pulls incremental data into memory, filters, sorts, merges it into target MongoDB documents, and saves them to the MongoDB cluster.

4. After processing, the scheduling service aggregates dashboard data, outputs user behavior reports, and pushes notification messages for user viewing.

Data Aggregation Process Explanation

All user‑behavior modules follow this rule, ultimately achieving real‑time T+0 data aggregation.