How YOLO-Count Enables Precise Object Counting in Text-to-Image Generation

Research Motivation

State‑of‑the‑art text‑to‑image (T2I) generators such as Stable Diffusion XL produce high‑fidelity images but often fail to respect numeric constraints in prompts (e.g., “5 apples”). Traditional counting approaches—object detection or density‑map regression—are either non‑differentiable or biased when objects vary in size or sparsity, making them unsuitable for direct integration with T2I pipelines.

YOLO‑Count Architecture

YOLO‑Count extends the YOLO‑World framework with three technical innovations that enable accurate, differentiable object counting for open‑vocabulary categories.

Cardinality Map : The image is divided into a regular grid. Each cell predicts a scalar in the interval [0, 1] that estimates the probability of containing an object. Summing all cell values yields the total object count. Because each object contributes roughly one unit regardless of its scale, the design eliminates the size‑related bias inherent in density‑map methods.

Fully Differentiable, Open‑Vocabulary Design : The counting head is trained end‑to‑end together with the backbone and can be back‑propagated through any downstream T2I generator. This allows the counting error (predicted count − desired count) to be turned into a gradient that directly corrects the generative model’s latent representation, ensuring that the final image matches the requested quantity.

Hybrid Strong‑Weak Supervision : Training leverages a mixture of strong annotations (pixel‑accurate segmentation masks) and weak signals (single‑point clicks or scalar count labels). The strong data provide precise localization, while the weak data dramatically increase the amount of usable training material and improve generalisation to unseen categories.

Training Procedure

The model is optimized with a combined loss:

Loss = λ_strong * L_strong(mask) + λ_weak * L_weak(count/point)

where L_strong is a pixel‑wise segmentation loss (e.g., binary cross‑entropy) applied to fully annotated images, and L_weak is a regression loss (e.g., mean‑squared error) on the summed cardinality map for images that only provide a total count or a single point annotation. The weighting coefficients λ_strong and λ_weak balance the two supervision streams.

Integration with Text‑to‑Image Generation

During inference, a T2I model receives a textual prompt containing a numeric phrase. YOLO‑Count processes the intermediate latent image, predicts the current object count, and computes a gradient that nudges the latent representation toward the target count. Because the entire pipeline is differentiable, this guidance can be applied iteratively (e.g., via classifier‑free guidance) without breaking the generation process.

Experimental Evaluation

T2I Quantity Control : On a benchmark derived from Stable Diffusion XL, YOLO‑Count reduces the absolute count error between the prompt and the generated image by a large margin compared with the SDXL baseline and other control methods. The improvement holds for categories seen during training and for novel categories, demonstrating robust open‑vocabulary generalisation.

General Object Counting : On standard counting datasets (e.g., COCO‑Count, FSC147), YOLO‑Count achieves state‑of‑the‑art mean absolute error (MAE) and root‑mean‑square error (RMSE), confirming that the cardinality map is an effective regression target for generic counting tasks.

Qualitative examples show that prompts such as “5 apples” consistently produce exactly five apples while preserving the visual quality of the underlying T2I model.

Key Contributions

Introduces a novel Cardinality Map that provides unbiased, size‑invariant counting by aggregating per‑cell probabilities.

Demonstrates a fully differentiable, open‑vocabulary counting module that can be plugged into any T2I generator for precise quantity control.

Proposes a hybrid strong‑weak supervision scheme that reduces annotation cost while maintaining high counting accuracy.

Shows state‑of‑the‑art performance on both dedicated counting benchmarks and T2I quantity‑control tasks, bridging the gap between computer‑vision counting and controllable generative AI.