How Matching Networks Tackle Imbalance with Cosine Similarity and Attention

Mathematical Foundations and Transformations of Cosine Similarity

Cosine similarity is defined on the interval [-1, 1] and quantifies directional similarity by the cosine of the angle between two vectors, completely ignoring vector magnitude. A value of 1 indicates identical direction, 0 indicates orthogonality, and -1 indicates opposite direction, giving it strong geometric properties in high‑dimensional spaces.

In practice the similarity is often linearly mapped to the [0, 1] range for downstream processing: scaled = (cosine + 1) / 2 Alternatively, a softmax transformation can convert similarities into a probability distribution, where a temperature parameter controls distribution sharpness; lower temperatures produce more peaked distributions.

Exponential scaling further amplifies similarity differences, useful in scenarios where stronger discrimination between close and distant pairs is required.

Classification Bias Caused by Imbalanced Datasets

When the support set contains markedly different numbers of samples per class, the simple sum or average of similarities gives disproportionate weight to majority classes, causing systematic bias toward those classes. This issue is prevalent in medical diagnosis, anomaly detection, and long‑tail image classification, where minority classes are often under‑represented.

Solutions for Imbalance

Adaptive Sample Weight Allocation

Adaptive weighting abandons the equal‑treatment assumption and assigns each support sample a weight based on quality indicators such as similarity to the query, representativeness in feature space, or an auxiliary confidence network. This approach automatically adapts to varying data quality, improving robustness to noisy or outlier samples. However, it introduces extra parameters and training complexity, requiring a trade‑off between performance gain and computational cost.

Global Normalization of Distance Matrix

This method builds a full query‑support similarity matrix and normalizes it globally: row‑wise normalization forces each query’s similarity scores to sum to 1, while column‑wise normalization equalizes each support sample’s total contribution across all queries. The technique offers fine‑grained control of similarity weights, especially when support samples vary greatly in quality, but incurs higher computational cost that can be mitigated with block processing or approximation.

Incorporating Prior Distribution Knowledge

Using class‑level prior statistics to guide weight allocation directly boosts rare classes. Rare‑class support samples receive higher weight coefficients, compensating for their scarcity. The hyper‑parameter ϵ controls the strength of this boost; larger ϵ values intensify the effect for extremely rare categories. This method works well when the class distribution is known but is limited in environments with unknown or shifting distributions.

Hierarchical Multi‑Scale Matching

Hierarchical matching decomposes the classification task into multiple levels: a coarse‑grained stage identifies broad categories, followed by finer‑grained stages that resolve sub‑classes. This strategy is effective for data with natural hierarchies, such as biological taxonomies or product catalogs. The hierarchy can be predefined using domain knowledge or learned automatically via clustering.

Hybrid Multi‑Paradigm Architecture

Hybrid architectures combine metric‑learning with other meta‑learning paradigms. A typical design trains a feature extractor with gradient‑based optimization while retaining a metric‑based similarity computation for final decisions. Another variant integrates a meta‑network that generates task‑specific distance functions, leveraging both optimization flexibility and intuitive similarity scoring.

Dynamic Sample Selection via Attention

Attention‑driven Matching Networks employ learnable attention weights to dynamically determine sample importance, replacing fixed weighting schemes. In transformer‑based implementations, the query vector interacts with support samples through multi‑head attention, automatically learning task‑relevant weights. This yields high flexibility and strong performance on complex tasks, at the expense of increased computational overhead and training difficulty.

Conclusion

Matching Networks scale well with various optimization strategies. Fixed cosine similarity offers low‑latency inference, while learnable distance functions improve adaptability at higher training and inference costs. Probability‑based normalization mitigates imbalance with minimal overhead, and max‑similarity per class can eliminate sample‑count bias but may under‑utilize available information. Adaptive weighting excels with noisy data, hierarchical matching shines on structured domains, and attention‑based variants represent the current frontier despite their computational intensity.