Federated Transfer Learning: Concepts, Examples, and Model Structures

Zhang San develops an autonomous drone and wants to build a sentiment analysis model for product reviews, but labeling data is costly. Observing a competitor's model, he learns that direct reuse fails due to different feature spaces, prompting the consideration of transfer learning.

His colleague suggests using domain adaptation (transfer learning) by labeling a small amount of target data and leveraging the competitor's large annotated source dataset, citing a domain adaptation method for sentiment classification.

Transfer learning aims to apply knowledge from a source task to a related target task, where differences may exist in feature space, label space, or prediction functions. In this context, a "domain" refers to the data distribution and a "task" to the learning objective.

The article defines transfer learning formally, describing source and target domains and tasks, and introduces federated transfer learning as a privacy‑preserving extension where data remain encrypted during collaborative training.

Two illustrative examples of transfer learning are presented. The first transfers the first seven layers of AlexNet (mid‑level visual features) to a new image classification network, improving performance on the Pascal VOC 2007 dataset. The second transfers knowledge from text classification to image classification using stacked auto‑encoders, aligning latent features of both modalities and employing a translator function to share label information.

Federated transfer learning model structures from references [3] and [4] are described. Both parties (source A and target B) possess private datasets and neural networks that map samples to latent representations. Shared co‑occurrence data and partially labeled target data enable alignment losses and regularization terms in a composite loss function comprising four components: supervised loss on labeled target data, alignment loss on co‑occurrence data, and regularization of model parameters.

The loss is optimized via back‑propagation, with gradients computed jointly over encrypted data from both parties. Encryption methods (e.g., homomorphic encryption) and secure transmission protocols (e.g., ABY, SPDZ) allow additive and multiplicative operations without revealing raw data, while polynomial approximations replace non‑linear activations to maintain compatibility with encrypted computation.

The article concludes by promising a detailed walkthrough of the federated transfer learning training process in the next installment.

References: [1] Blitzer et al., 2007; [2] Pan & Yang, 2009; [3] Liu et al., 2018; [4] Sharma et al., 2019; [5] Jing et al., 2019; [6] Dai et al., 2007; [7] Pan et al., 2010; [8] Lawrence & Platt, 2004; [9] Richardson & Domingos, 2006; [10] Davis & Domingos, 2009; [11] Oquab et al., 2014; [12] Krizhevsky et al., 2012; [13] Shu et al., 2015; [14] Chua et al., 2009; [15] Zhu et al., 2011; [16] Qi et al., 2011.