How to Overcome Data Scarcity in Machine Learning: Strategies and Techniques

Identifying assumptions is crucial for breaking assumptions — and breaking assumptions drives creativity and technological innovation.

Recently, ChatGPT has reignited enthusiasm for general AI, and AIGC is now a hot topic. However, large models depend heavily on abundant data, raising the question: what does a data architecture for machine learning and deep learning look like?

"Garbage in, garbage out" – data acquisition has become a bottleneck for many ML applications. Deep learning amplifies this issue because models with millions or billions of parameters require massive training data. Creating new large datasets for each task is time‑consuming and often infeasible, especially for rare‑disease detection.

How can a software system, particularly an ML‑driven one, address data scarcity?

Data scarcity is common in ML, especially in supervised learning, but also appears in unsupervised scenarios. We will focus on supervised, unsupervised, and semi‑supervised learning, ignoring reinforcement learning.

Data scarcity can be divided into two categories: (1) data is hard to obtain, leading to missing data; (2) existing data lack labels, preventing high‑quality datasets.

1. Data Missing

1.1 Dataset Reuse

Dataset reuse means applying an existing dataset to a new purpose, such as using ImageNet for image generation after its original classification use.

Reuse also includes transforming existing datasets, e.g., image inpainting that restores missing parts using pre‑existing data.

Sometimes a dataset is redefined to have no ML task at all.

1.2 Data Augmentation

Data augmentation artificially expands the training set by applying modifications, originally to prevent overfitting.

Typical augmentation in computer vision includes geometric transforms (flip, crop, scale, rotate) and photometric changes (color channel adjustments). For small or imbalanced datasets, augmentation improves generalization, and GAN‑based methods can generate new examples.

1.3 Multimodal Learning

Multimodal learning enriches inputs by providing multiple modalities (e.g., an image and its caption), reducing data requirements and improving generality.

When only a few labeled examples exist, combining images with richer semantic information (labels, attributes, natural‑language descriptions) yields better performance.

1.4 Curriculum Learning

Curriculum learning presents training examples in order of increasing difficulty, mimicking human teaching. Starting with easy samples helps the model learn broad concepts, then harder examples refine them, reducing the number of required examples.

1.5 Argument‑Based Machine Learning (ABML)

ABML leverages expert local knowledge to constrain the search space by iteratively adding if‑then rules, removing covered examples, and repeating until all data are explained.

When expert knowledge is available, ABML integrates strong priors and can be more meaningful than global explanations.

1.6 Multi‑Task Learning

Multi‑task learning trains several related tasks simultaneously, exploiting shared structure to improve generalization, especially when individual tasks lack large datasets.

Implementation typically uses hard parameter sharing (shared hidden layers) or soft sharing (task‑specific models with regularized parameter distances).

1.7 Transfer Learning

Transfer learning reuses a pre‑trained model from a related task, fine‑tuning it on a small target dataset, dramatically reducing the amount of task‑specific labeled data needed.

Examples include ImageNet‑pretrained models for medical imaging and BERT for NLP tasks, where the pre‑trained representations provide useful priors.

1.8 Meta‑Learning

Meta‑learning learns how to learn: it extracts meta‑knowledge from many tasks so that a model can quickly adapt to a new task with few examples.

Common approaches are metric‑based (e.g., nearest‑neighbor), optimization‑based (e.g., MAML), and model‑based (e.g., fast weights).

2. Label Missing

Often abundant data lack labels because unlabeled data are easier to collect.

2.1 Active Learning

Active learning queries the most informative unlabeled examples—typically those near the decision boundary—to obtain labels efficiently.

Generated examples still need labeling; methods like GANs can produce expressive samples but may be harder to interpret.

2.2 Semi‑Supervised Learning

Semi‑supervised learning combines labeled and unlabeled data, assuming smoothness, cluster, and manifold structures to reduce labeling requirements.

Smoothness: neighboring points likely share a label.

Cluster assumption: data form discrete clusters with consistent labels.

Manifold assumption: data lie on a low‑dimensional manifold where points on the same manifold share labels.

Unsupervised pre‑processing (e.g., autoencoders, PCA) can extract useful features to aid learning.

2.3 Data Programming

Data programming defines weak supervision functions (labeling functions) that noisily label data; the noisy labels are then denoised to create a training set.

2.4 Regularized Expectation

Regularized expectation incorporates prior knowledge about label proportions in data sub‑groups to generate noisy labels and estimate true label distributions.

2.5 Distant Supervision

Distant supervision automatically generates labeled training data by aligning existing knowledge bases with raw text, producing weakly labeled examples.

2.6 Side Supervision

Side supervision exploits auxiliary signals present in unrelated datasets (e.g., inferring gender from names using Wikipedia) to provide additional supervision without explicit labeling.