Designing a Full-Stack Credit Data System: From Ingestion to Real-Time Decision

1. The Core Problem of Credit Data Systems

Credit data systems aim to answer a dynamic decision question: "How risky is a given enterprise or individual right now?" The difficulty stems from three constraints: multiple heterogeneous data sources , the need for millisecond‑level response for fraud detection while also leveraging long‑term historical data, and the requirement for high recall and high precision to avoid costly mis‑judgments.

2. Six‑Layer Logical Architecture

The system is divided into six logical layers, each with clear responsibilities:

Application Layer : credit reports, API services, risk‑control dashboards, regulatory reports.

Decision Engine Layer : rule engine, strategy orchestration, explainable outputs.

Model Layer : credit scoring models, anti‑fraud models, graph‑based anomaly detection.

Feature Engineering Layer : offline, near‑real‑time, real‑time, and static features; feature storage and monitoring.

Processing Layer : split into a real‑time stream (Flink/Kafka) and an offline batch (Spark/Hive/Presto) sub‑layer.

Data Collection Layer : internal business data, external APIs, third‑party sources, message queues.

Data flows through these layers, forming a closed loop from raw data to final decision.

3. Data Collection Layer – Multi‑Source Heterogeneity

Typical data sources include:

Internal business data (credit records, transaction logs, repayment history) – synchronized via DB sync/CDC, T+0 latency.

Business‑registry & judicial data – API integration, T+1 latency.

Financial data (tax records, social security, financial statements) – file import/API, monthly updates.

Public sentiment data – web crawling/NLP extraction, real‑time.

Transaction behavior data – event tracking/Kafka, millisecond latency.

Graph relationship data – equity penetration, guarantee relations, fund flows – data purchase or self‑built, T+1 latency.

Incremental synchronization of multi‑source data (e.g., MySQL, PostgreSQL) is a common challenge; CDC solutions such as Debezium + Kafka , Canal , and DataX are used to capture binlog changes without impacting source databases. The key metric for this layer is "no data loss, no source‑DB impact".

4. Feature Engineering Layer – The Real Bottleneck

Feature quality determines the upper bound of machine‑learning performance. The article distinguishes four feature types:

Offline batch features (Spark, T+1/hourly) – e.g., average daily transaction amount over the past 30 days.

Near‑real‑time features (Flink, minute‑level) – e.g., login‑failure count in the last hour.

Real‑time features (Redis, second‑level) – e.g., cumulative transaction count for the current day.

Static features (DB lookup, T+0) – e.g., registered capital, legal‑person age.

A unified Feature Store is essential to avoid training‑serving skew; the same feature definition must be used for both offline training (Parquet/Hive) and online inference (Redis/HBase).

Graph features capture relational risk signals that traditional features miss. Common graph features include:

Degree centrality – number of neighbours (business breadth).

PageRank – importance score (core position in the network).

Community detection – whether an entity belongs to a high‑risk cluster.

Path features – shortest path or common neighbours to black‑list entities.

Typical graph databases: Neo4j (flexible queries, medium scale) or JanusGraph (large‑scale distributed).

5. Model Layer – From Scorecards to Deep Learning

The evolution path consists of three stages:

Expert scorecards – rule‑based logistic regression; strong interpretability, regulatory friendliness, but limited in capturing non‑linear risk.

Machine‑learning models – GBDT (XGBoost, LightGBM); higher accuracy and automatic feature crossing, but weaker interpretability (requires SHAP/LIME).

Deep learning & GNN – Transformers for sequential behavior, Graph Neural Networks for relational data; can capture complex patterns and collusive fraud, at the cost of large data requirements, high training cost, and still‑challenging explainability.

Anti‑fraud models are as critical as credit‑scoring models, differing in objective (detect malicious behavior vs predict repayment), time horizon (real‑time vs months), data features (real‑time behavior, device fingerprint vs historical financials), response speed (millisecond vs second), and update frequency (daily/weekly vs monthly/quarterly).

Regulatory compliance demands model explainability. Common techniques:

SHAP – global and local explanations, theoretically solid but computationally heavy.

LIME – fast, suitable for online per‑prediction explanations.

Feature importance – simple global view, not per‑prediction.

Decision‑path extraction – works for rule‑based or tree models, understandable by business users.

In practice, a SHAP + LIME combination is mainstream: SHAP for offline deep analysis, LIME for millisecond‑level online explanations.

6. Real‑Time Stream Processing Layer – Millisecond‑Level Challenges

The article compares Lambda (batch + stream) and Kappa (pure stream) architectures. For credit systems, a hybrid approach is recommended: use Kappa (Kafka + Flink) for real‑time features, batch processing for complex offline features, and a Lambda‑style dual‑track for core features. Key performance indicators for Flink‑based real‑time risk control:

End‑to‑end latency ≤ 100 ms (from transaction receipt to decision).

Throughput ≥ 100 k TPS (to handle peak load).

Exactly‑once processing (no duplicate or lost messages).

Failure recovery time ≤ 30 s (RTO).

7. Decision Engine Layer – Fusion of Rules and Models

The decision engine combines model outputs with business rules to produce final decisions. Typical workflow:

Request → Query real‑time (Redis) + offline (HBase) features → Model inference (score / fraud probability) → Rule validation (blacklist / whitelist / thresholds) → Decision output (approve / reject / manual review) → Explainability output (rejection code / SHAP contribution) → Full‑link audit log

Rule engine choice: Drools (open‑source, visual rule authoring, forward/backward reasoning, Spring Boot integration). Rules handle hard constraints (blacklists, regulatory limits), while models handle soft judgments (probabilistic risk ranking).

8. Privacy Computing – Breaking Data Silos

Data islands (banks, tax authorities, courts) hinder credit assessment. Privacy‑preserving techniques enable collaborative analysis without raw data sharing:

Federated Learning – models train on local data, only gradients are shared; mature, used for cross‑bank credit modeling.

Secure Multi‑Party Computation (MPC) – cryptographic joint computation; mature, for private data matching.

Trusted Execution Environment (TEE) – hardware secure enclave; relatively mature, for high‑security joint analysis.

Homomorphic Encryption – compute directly on ciphertext; early stage, for extreme security scenarios.

9. Full‑Stack Technology Stack

Data ingestion: Kafka, Debezium, Flume (alternatives: Canal, Pulsar).

Batch processing: Spark, Hive, Presto (alternative: Flink batch).

Stream processing: Apache Flink (alternative: Kafka Streams).

Feature store: Redis + HBase (alternative: Feast).

Graph DB: Neo4j, JanusGraph (alternative: TuGraph).

Model training: XGBoost, LightGBM, PyTorch (alternative: TensorFlow).

Model serving: Triton, Ray Serve (alternative: TorchServe).

Rule engine: Drools (alternative: Esper).

Privacy computing: FATE (federated learning) (alternative: MP‑SPDZ).

Containerization: Kubernetes + Docker.

Monitoring: Prometheus + Grafana + ELK.

Job scheduling: Airflow, DolphinScheduler.

10. Architecture Selection Framework

A decision matrix helps choose between batch‑first or real‑time‑first designs based on business scenario, latency tolerance, data scale, and team expertise. The article stresses that even "pure real‑time" anti‑fraud systems should retain offline batch capabilities for full‑history feature computation and model retraining. Overall, credit data systems stress the breadth of data collection, depth of feature engineering (including graph and temporal features), precision of real‑time computation, thickness of privacy compliance, and continuity of model governance. The optimal architecture evolves gradually, starting from the smallest viable system that meets current needs and iterating based on observability.