Can AI Really Judge Good Design? Findings from the Design Crit Study

1. AI Can Generate Images, But Can It Judge Quality?

Contra Labs released a study called Design Crit to address whether AI systems can evaluate the quality of a design, not just produce one.

2. Design Evaluation Is Multi‑Dimensional

Design judgment cannot be reduced to a single "which is better" question. Ten professional designers rated outputs from four cutting‑edge text‑to‑image models across nine real‑world design dimensions, such as overall preference, mood, visual hierarchy, color harmony, typography, color accuracy, spatial accuracy, and brief compliance.

3. Dataset Scale and Rating Procedure

The dataset contains 9 criteria, each with 80 prompts. For every prompt, five designers scored the four model outputs, yielding 1 600 ratings per criterion. Designers also performed pairwise comparisons between the four models and marked whether each image exhibited hallucinations. About 55% of images were clean, 35% showed mild hallucinations, and 10% had severe hallucinations.

4. Designer Consistency Signals

Consistency among designers is measurable. Agreement is higher on criteria that directly map to the brief (e.g., correct text rendering, spatial accuracy) and lower on purely subjective dimensions (e.g., color harmony). Overall, designers agreed with the majority opinion 74.1% of the time, well above the 50% random baseline.

5. How Current AI Judges Perform

Nine pretrained systems were evaluated, including three dedicated aesthetic scorers (HPSv2.1, PickScore‑v1, LAION‑Aesthetic‑V2) and six open‑source vision‑language models. The best system, HPSv2.1, achieved only 54.3% agreement with the majority designer opinion—just marginally better than random guessing and far below the human 74.1% benchmark.

6. Model Size Does Not Solve the Problem

Scaling up models (Qwen 3‑VL 4B, 8B, 32B) did not improve accuracy; performance stayed between 51% and 54%. Larger models were more stable with respect to image position but did not become more aligned with designers.

7. Training a Small Judge on Design Crit Data

A lightweight pairwise‑difference head was added to a frozen visual‑language encoder and trained solely on the Design Crit dataset. This model reached 0.611 agreement, closing roughly 46% of the gap between random (0.500) and the human upper bound (0.741). On difficult 3‑vs‑2 split cases, it matched the human upper limit (0.602 vs 0.600).

8. Why This Matters

Design Crit provides a criterion‑level decision layer for generative design systems, enabling task‑specific routing (e.g., choosing a model strong in typography for logos). It also offers fine‑grained supervision signals for training preference judges and reward models, moving beyond a single vague score.

9. Limitations

Small sample size: only five designers per prompt.

Each criterion uses independent prompts, so the same image is never evaluated on multiple dimensions.

All prompts are English, limiting cross‑language applicability.

The nine criteria, while extensive, omit accessibility, brand consistency, motion, and audience relevance.

10. Future Directions

Future work should increase the number of annotators per prompt, add multilingual prompts, and expand evaluation dimensions. Evaluating the same design across all criteria would reveal how designers trade off color, hierarchy, fidelity, and feel—information hidden by a single aggregate score.

11. Author’s Perspective

The study argues that design judgment is not a monolithic "good vs. bad" notion but a set of interlocking trade‑offs. A judgment layer that can articulate specific strengths and weaknesses is more valuable than a one‑click perfect generator.

12. Final Takeaway

AI can now generate designs, but it still cannot reliably distinguish good from bad design. However, the missing signal—design "taste"—is learnable from expert data, and incorporating such fine‑grained feedback is the next crucial step toward production‑grade AI‑assisted design.