Model‑Independent Learning: Multi‑Task Learning and Transfer Learning

Machine learning traditionally relies on large, task‑specific training sets, but many real‑world problems lack sufficient data or have distribution shifts between training and test data. To address these limitations, researchers explore model‑independent learning approaches that are not tied to a particular neural architecture.

Multi‑Task Learning (MTL) simultaneously learns several related tasks, allowing them to share knowledge through shared modules (often low‑level layers) while keeping task‑specific modules for specialized features. Four common sharing patterns are hard sharing, soft sharing, hierarchical sharing, and shared‑private sharing. The joint objective combines the losses of all tasks, typically with equal weights, and training proceeds by alternating task selection and gradient updates, followed optionally by task‑specific fine‑tuning.

MTL improves generalization because it effectively enlarges the training data, acts as a regularizer that prevents over‑fitting to a single task, and yields representations that are useful across tasks.

Transfer Learning (TL) tackles the scenario where a source domain has abundant labeled data while the target domain has limited or no labels. TL aims to transfer knowledge from the source to the target, often by re‑using pretrained deep networks. Two main TL types are inductive transfer learning (source and target share input space but may differ in output space) and transductive transfer learning (source has many labels, target has few or none, but both domains are visible during training).

Inductive TL can be performed by using pretrained features as inputs to a new classifier or by fine‑tuning parts of the pretrained network. The choice of which layers to transfer depends on the similarity between source and target tasks.

Transductive TL, especially domain adaptation, assumes the same input space but different data distributions. Common strategies include learning domain‑invariant representations via discrepancy measures (e.g., MMD, CMD) or adversarial training with a domain discriminator that tries to distinguish source from target features while the feature extractor learns to confuse it.

The adversarial objective balances a domain classification loss against the task loss, encouraging features that are both discriminative for the source task and indistinguishable across domains.

In summary, both multi‑task learning and transfer learning provide ways to reduce the reliance on massive task‑specific datasets by leveraging shared structures and knowledge across tasks or domains, bringing machine learning closer to the continual, data‑efficient learning observed in human cognition.