Transfer Learning: Concepts, Challenges, and Recent Research Highlights from CIKM 2017

Transfer learning (TL), also called domain adaptation, is a mainstream machine‑learning technique that transfers knowledge from a source domain with abundant labeled data to a target domain with limited labels.

The CIKM 2017 keynote by Prof. Yang Qiang (HKUST) highlighted how TL can be combined with recent deep‑learning advances to address the growing demand for data‑efficient AI solutions.

Why TL is hard? The main difficulty lies in learning a shared knowledge representation that adapts well across domains.

TL methods are typically grouped into four categories:

Instance‑based TL : Find source samples similar to the target and re‑weight them during joint training. Works well when source and target share overlapping regions.

Feature‑based TL : Assume common features exist between domains and map both domains into a shared feature space before applying conventional learning. Effective but prone to over‑fitting.

Parameter‑based TL : Transfer a pre‑trained model (e.g., ImageNet) to a new task, often fine‑tuning only the last layers. Leverages model similarity but may converge slowly.

Relation‑based TL : Transfer relational or logical structures learned in the source domain to the target domain.

Supervised TL: Feature Learning

Deep Adaptation Networks (DAN, Long et al., 2015) propose a fully‑shared TL model that learns a shared representation with a MK‑MMD loss to reduce domain discrepancy, training separate classifiers for source and target.

Yosinski et al. (NIPS 2014) analyze which deep‑network features are transferable, showing that lower‑level features are more generic while higher‑level features become domain‑specific, supporting the fully‑shared TL design.

Unsupervised Deep TL

These methods assume no labeled data in the target domain. They jointly optimize the source loss (with labels) and a domain‑difference loss, often using domain discriminators or adversarial training to learn domain‑invariant features.

Transitive Transfer Learning

When source and target domains are too distant, intermediate domains are introduced to bridge the gap. This approach, also called distant‑domain transfer learning, selects useful intermediate data via a reconstruction loss (often using autoencoders).

The overall loss comprises binary selection variables v, reconstruction losses for selected source and intermediate data, target reconstruction loss, and a regularizer encouraging the selection of many source/intermediate samples.

Side‑information can be incorporated by adding classification losses for source and target (J1) and a confidence‑based loss for intermediate data (J2). Optimization proceeds via block coordinate descent: fix v and update model parameters by back‑propagation, then fix parameters and update v.

The final model architecture integrates the shared feature extractor, domain‑invariant layers, and task‑specific classifiers as illustrated below:

— END —