Data Stays, Model Moves: SDH PaaS 3.0's Federated Learning Productization

The previous article "Making Joint Analysis More Efficient: SecureID Turns Ciphertext Intersection into Plaintext Alignment" introduced how reusable SID mapping enables faster cross-domain data alignment. After data alignment, businesses further want to answer: which users are more likely to convert, and who should be targeted in the next round?

From Joint Analysis to Joint Modeling

When data collaboration moves from performance review to conversion prediction, the required cooperation changes. Predicting whether a user will purchase requires a label (purchase or not) held by the platform and complementary features held by the business. Relying on a single party's data may miss valuable signals. An analysis suite can review existing results — e.g., how many users were reached and how many purchased — but to predict which users are more likely to buy, the platform's purchase labels and the business's complementary features are combined through vertical federated learning. A prerequisite is that both parties share a set of common users so that the platform's labels can be matched with the business's features for the same individuals. After alignment, each party continues to use its own locally stored data.

How Both Parties Jointly Train a Model

Define Roles and Identify Common Users

In the EFLS federated learning framework, parties configure roles for training. In this example, the platform acts as Leader, providing the "user purchased" label and optionally local features; the business acts as Follower, providing complementary features. Sample alignment finds records corresponding to the same user across both datasets. Based on task configuration and cooperation authorization, the common sample set is determined, ensuring subsequent computation targets the same user cohort. After alignment, each party still uses its own data.

Local Computation and Exchange of Intermediate Results

During training, each party first computes locally on its own features, then exchanges protocol-required intermediate results. The Leader combines these results with the purchase label, computes the gap between predicted and actual outcomes, and sends back gradient information for model updates. Both parties update their model parameters accordingly and repeat until training converges. Communication throughout training carries only the intermediates and messages dictated by the algorithm and task configuration; raw features and labels never leave their local environments. EFLS supports encrypting forward-computation intermediates and applying differential privacy — adding controlled noise to reduce per-sample leakage risk. Which protections are enabled and their parameters are decided per task and protocol. After training, further authorization and approval govern who may use the model, which data may be used for prediction, and who receives the prediction results.

Launching and Running Federated Tasks on SDH

Confirm Participants and Usable Data

Before modeling, both parties confirm in SDH's collaboration space who participates, what business problem is addressed, and which data will be used. Data from sources such as ODPS and OSS is registered as data assets, and the fields available for the task are specified. Authorization is granular to the field level: for the business's features, it must be pre-approved which fields can be used for training and which for prediction. The modeler then selects task inputs from the authorized fields.

Configure Task, Complete Approval, and Use Results

Creating a training task involves selecting a template and configuring participants, input data, Leader/Follower roles, and training parameters. When prediction is needed, a prediction task is created specifying the model, input data, and result storage location. Prediction input data and result recipients must also have appropriate authorization. After submission, each party approves per the cooperation agreement, while SDH enforces budget and runtime limits. Once approved, SDH schedules the task to both execution environments. Each party prepares its data and begins computation under the same protocol. During execution, task status — preparing, running, or failed — can be monitored. Upon completion, the model, evaluation metrics, and other results are delivered to authorized parties after approval per the agreement. Run logs capture execution details; failures can be diagnosed using status and logs.

Ensuring Stable Large-Scale Joint Training

When samples reach tens of millions and features number in the hundreds, challenges extend to data preparation and runtime coordination. SDH optimizes the execution pipeline around memory and disk usage, asymmetric preparation speeds, task retries, and sample consistency to keep large-scale jobs stable.

Memory Pressure: Batched Sorting with Multi-Way Merge

Tens of millions of samples with hundreds of features can yield tens of gigabytes per party, exceeding single-machine memory if loaded at once. SDH first sorts data in batches, writes sorted batches to temporary files, then merges these files sequentially. This external sort with multi-way merge avoids requiring enough memory to hold the entire dataset.

Asymmetric Preparation Speeds: Separate Wait from Communication Timeout

Parties often have different data volumes and processing speeds; one may finish preparation while the other is still working. The system therefore distinguishes "waiting for peer preparation" from "communication timeout after handshake." The pre-training handshake exchanges messages to confirm both sides are ready. Starting the communication timeout too early would mistake normal data preparation for a network fault. Separating these phases allows correct diagnosis of why a task is waiting.

Disk Space: Timely Cleanup of Temporary Files and Caches

Data preparation, remote file reads, and retry attempts consume local disk. Accumulating temporary files and caches can cause out-of-space failures. SDH reserves disk space proportional to data scale, closes files promptly after reading, and cleans unused caches and temporary files. Data needed for retries is saved to locations that persist beyond container destruction.

Task Retries: Reuse Completed Shards to Reduce Rework

Large datasets are split into shards for batched processing. On restart, SDH checks the recorded generation state of each shard: whether it matches the current task's input and is complete. Shards that are already finished and validated are reused directly; only missing or incomplete shards are reprocessed, cutting retry time.

Sample Correspondence: Ensure Labels and Features Belong to the Same User

During joint training, the platform's purchase labels must align one-to-one with the business's features for the same user. Both parties follow agreed sharding, sorting, and reading rules to guarantee consistent sample order within each batch, preventing label-feature mismatch. SDH validates this by running Leader and Follower simultaneously in tests to verify correct collaboration.

Using Predictions to Support Business Decisions

The value of joint modeling is ultimately measured by business actions and outcomes. Prediction scores help identify audience differences, assisting in targeting decisions and marketing planning. Model usefulness is judged by combining offline and online evaluation. Offline, holdout data with known purchase outcomes tests prediction accuracy. Online, after using predictions for targeting, metrics such as purchase count and conversion rate are compared against the previous approach to assess real-world impact. SDH has achieved ISO/IEC 27001 and ISO/IEC 27701 dual certification, as well as the China Academy of Information and Communications Technology "Trusted Privacy Computing" federated learning basic capability evaluation.