Elastic Federated Learning Solution (EFLS): Architecture, Core Functions, and Technical Details

Project background: In the mobile internet era, privacy and data security reduce openness between apps, leading to information islands. Google introduced federated learning in 2016 to protect privacy. Alibaba's algorithm team and advertising team open‑sourced the Elastic Federated Learning Solution (EFLS) to bring federated learning to Alibaba Mama business scenarios.

Business application: EFLS targets large‑scale sparse advertising scenarios where media and e‑commerce parties cannot share raw user data. It enables joint click‑through‑rate prediction while preserving privacy, improving ROI for brands such as 珀莱雅, 卡姿兰, 薇诺娜, 花西子, 修正, etc.

Architecture and core functions: EFLS supports both vertical and horizontal federated learning. Version 0.1 focuses on two‑party vertical federated learning and consists of three modules—EFLS‑Data (sample set intersection), EFLS‑Train (federated training), and EFLS‑Console (web control panel). The solution provides a workflow‑driven visual UI, a Flink‑on‑K8s implementation for sample intersection, a lightweight client for quick testing, a C++‑based high‑performance training framework, and two novel algorithms (horizontal aggregation and hierarchical aggregation).

Sample set intersection: The process includes bucketizing samples, performing per‑bucket intersection via gRPC, and verifying checksums. For sensitive keys, EFLS implements a blind RSA‑based PSI protocol to prevent leakage.

Federated model training: In vertical federated learning, each party holds its own features and model parameters. The collaborator encrypts intermediate results, sends them to the leader, which computes gradients, encrypts them back, and updates its own model. EFLS‑Train builds on TensorFlow 1.15, offering a federated Dataset API, secure C++ gRPC communication, privacy encryption (including differential privacy), and a Keras‑like high‑level API.

CTR = efl.FederalModel()
CTR.input_fn(input_fn)
CTR.loss_fn(model_fn)
CTR.optimizer_fn(efl.optimizer_fn.optimizer_setter(tf.train.GradientDescentOptimizer(0.001)))
CTR.compile()
CTR.fit(efl.procedure_fn.train(), log_step=1, project_name="train")

Algorithm innovations: EFLS introduces a horizontal‑aggregation method that uses attention mechanisms to fuse media‑side feature vectors with e‑commerce embeddings, and a hierarchical‑aggregation method (AutoHERI) that automatically searches optimal multi‑level connections via neural architecture search.

Product console: A web console abstracts the federated workflow, allowing users to manage accounts, tasks, data, and permissions, thereby reducing development time.

Future roadmap: EFLS will add multi‑party support, auto‑elastic scaling, advanced privacy encryption, multi‑modal graph learning, and pre‑training models for ad recall and ranking, aiming to enhance performance under limited data conditions.