Hierarchical Residual Network for Multi‑Granularity Classification (HRN) – CVPR 2022 Paper Overview

Motivation In supervised deep‑learning image recognition, constructing large‑scale, high‑quality labeled datasets requires expert knowledge and high‑quality images; annotators with different background knowledge may assign labels at different hierarchical granularities, leading to inconsistencies and high annotation costs.

Problem Statement The paper aims to increase the utilization of labeled data and reduce annotation cost by modeling hierarchical semantic structures and leveraging multi‑granularity labels.

Algorithm Introduction The proposed Hierarchical Residual Network (HRN) consists of three key components:

Construction of a label‑relation tree that encodes parent‑child and mutual‑exclusion relations among hierarchical labels.

A granularity‑specific feature extraction block (GSB) for each hierarchy level, extracting level‑specific features from a shared backbone (ResNet‑50).

Hierarchical residual connections that linearly combine coarse‑level features with fine‑level features, followed by a ReLU non‑linearity, enabling each child level to inherit attributes from its parent.

The network has two parallel output branches: a sigmoid‑based branch that predicts probabilities for all hierarchical labels according to the label‑relation tree, and a softmax branch that predicts mutually exclusive fine‑grained classes, forming a composite loss.

Composite Loss Function The loss combines a hierarchical probability classification loss (derived from marginal probabilities over the label‑relation tree) with a standard multi‑class cross‑entropy loss for leaf nodes, allowing selective weighting based on whether a sample is annotated at the leaf level.

Experiments The method is evaluated on three fine‑grained benchmarks (CUB‑200‑2011, FGVC‑Aircraft, Stanford Cars) with hierarchical label trees built from Wikipedia. Two experimental factors are simulated: (1) varying the proportion of leaf‑level samples re‑labeled to parent classes (0 %–90 %) to mimic expert knowledge variance, and (2) degrading image resolution to model quality loss. Ablation studies verify the contribution of GSB, linear combination, and ReLU. Additional experiments compare the composite loss against a traditional hierarchical loss, and against four state‑of‑the‑art hierarchical multi‑granularity methods (HMC‑LMLP, HMCN, Chang et al., C‑HMCNN). Results show consistent accuracy gains across all datasets and labeling ratios.

Visualization Grad‑CAM heatmaps illustrate how HRN focuses on hierarchical regions of interest, and comparative visualizations demonstrate superior localization over baseline methods.

Practical Impact The hierarchical classification algorithm has been deployed in Ant Group’s content‑security services, demonstrating real‑world applicability.

For the full paper and code, see CVPR 2022 paper and the GitHub repository .