PinchBench Reveals Real‑World Performance of LLMs on OpenClaw Tasks

Benchmark Overview

PinchBench is an evaluation suite that measures large language models (LLMs) on the OpenClaw “shrimp‑farming” scenario, turning each model into an autonomous digital worker that performs real‑world office tasks.

The benchmark records three key metrics for each model: success rate, execution time, and per‑invocation monetary cost.

Performance Rankings

Success Rate

Google Gemini‑3‑Flash‑Preview leads with a 95.1% success rate.

MiniMax minimax‑m2.1 and Kimi‑k2.5 follow closely with 93.6% and 93.4% respectively.

MiniMax minimax‑m2.5 achieves only 35.5%.

Speed

MiniMax minimax‑m2.5 records the fastest average completion time of 105.96 seconds.

Google Gemini‑2.0‑Flash and Meta LLaMA‑3.1‑70B follow closely, both around 106 seconds.

Models that require deeper reasoning tend to be slower.

Cost

OpenAI GPT‑5‑nano has the lowest per‑run cost at $0.03.

Google Gemini‑2.5‑Flash‑Lite follows at $0.05.

The top eight lightweight models stay under $0.20 per task, while heavyweight models can approach $1 per invocation.

Evaluation Methodology

PinchBench embeds each model in a proxy framework that simulates a full office environment. Identical task prompts are supplied to every model; the system timestamps execution, records token usage, and validates outputs against a predefined answer key.

All tasks are stored as formatted text files in a dedicated Git repository, ensuring reproducibility and version control: https://github.com/pinchbench/skill.

Test Suite Composition

The current suite contains 23 distinct tasks covering:

Basic conversational checks.

Administrative duties such as generating correctly formatted calendar entries.

Research tasks like fetching the latest stock prices or compiling conference listings.

Programming challenges, e.g., writing a weather‑query script with error handling or scaffolding a project directory.

Technical writing, including summarizing long documents or translating complex papers into child‑friendly explanations.

Interpersonal scenarios, such as drafting polite decline emails with alternative suggestions and testing long‑term memory of project details.

Scoring Mechanisms

Automated objective scoring : Checks for required files or specific function calls.

Judge‑based scoring : For subjective tasks (e.g., email tone), a secondary LLM (Claude Opus) applies a detailed rubric.

Hybrid scoring : Combines automated checks with judge evaluation to ensure both factual correctness and expressive quality.

Version Control and Reproducibility

Each test run records a unique cryptographic hash of the exact repository state, acting as a timestamped seal. Any change—even a single punctuation mark—in the task repository generates a new hash, guaranteeing that leaderboard scores can be traced back to the precise test set and scoring logic used.

Minor, non‑impactful edits (e.g., documentation tweaks or configuration adjustments) retain the current version label, while substantive changes to prompts, scoring scripts, or evaluation code trigger a new version and reset the ranking for affected runs.

Key Findings

Raw model size or headline performance does not directly translate to cost‑effective productivity in real office workflows. Lightweight models such as MiniMax‑m2.1, Kimi‑k2.5, as well as Chinese models GLM‑4.5‑air and Qwen3‑coder‑next, achieve a strong balance of success rate, speed, and cost, making them suitable for routine tasks. Heavyweight models, despite higher capabilities, often incur costs near $1 per invocation, rendering them economically impractical for everyday use.