Measuring Harness: How a $0.175/M DeepSeek Setup Beats Claude Opus 4.8 by 57×

AOE Tech Labs, the creator of Floatboat, has released three product innovations—Floatboat client, FloatIM, and FloatSchedule—each focused on real‑world office scenarios rather than UI novelty. The latest release puts the underlying technology, called Harness, on a public benchmark.

On August 7, Floatboat published a full evaluation of its Harness using the inexpensive DeepSeek‑V4‑Flash model (API price $0.14 / M input, $0.28 / M output, mixed price $0.175 / M). Across five third‑party benchmarks the Floatboat Harness beats Claude Opus 4.8, whose mixed price is $10 / M, making Opus 4.8 57.1 times more expensive while scoring lower on every metric.

The experiment isolates a single variable: the same DeepSeek‑V4‑Flash 0731 model is run with two different harnesses. Both sides use the model, DeepSeek’s official minimal‑mode harness (max tier, top_p = 0.95, temperature = 1.0), and Floatboat’s Evaluation Harness in an isolated sandbox with identical input parameters.

Results show the official DeepSeek harness scores of 54.4, 73.2, 82.7, 25.1 and 70.7, whereas the Floatboat harness scores rise to 67.25 and higher, winning all five tasks. Gains increase with task length: 1.9 % for short tasks, up to 23.6 % for the longest, demonstrating a monotonic increase in benefit as the execution chain grows.

To quantify this benefit the authors propose the Harness Leverage Ratio (HLR):

HLR = (gain from swapping Harness) / (gain from upgrading to a stronger model)

Using DeepSWE as an example, the score jump from 54.4 to 67.25 (12.85 points) divided by the 3.6‑point gap between DeepSeek’s baseline and Claude Opus 4.8 yields HLR > 1, meaning the harness improvement alone delivers more performance than moving to a model that costs 57 times more.

The analysis explains why long‑term tasks benefit most. Real work involves ambiguous, evolving goals and multi‑step state, so model drift accumulates over many iterations. A proactive Agent Loop must repeatedly surface hidden standards, correct drift, and verify intermediate results. The authors break the system into four layers—Runtime, Agent Loop, Tools, and Infra—each essential for handling long‑running tasks, and argue that only a fully controllable stack can reliably harness unpredictable large models.

Floatboat’s definition of a “Proactive Agent” emphasizes the ability to discover vague user intents, continuously explore, execute, observe, and self‑correct without explicit prompts, a capability that aligns with the observed larger HLR values for longer tasks.

The benchmark environment is deliberately stripped down for reproducibility, so actual client usage—integrating over 3,000 online services and multimodal models—should yield even higher scores. The cheap cost ($0.175 / M) eliminates the need to weigh expense, allowing agents to be used daily rather than merely demonstrated.

All numbers are reproducible via the published report (https://floatboat.ai/news/harness-benchmark). The authors encourage users to repeat the experiment, especially with long‑term tasks such as modifying a codebase or generating a complete report, to see the full leverage of the harness.

“Technically, only a fully controllable system can master highly unpredictable models and maximize model intelligence.” – AOE Tech Labs team