Jev's Dangerous Fallacy: Rational Judgments, Stupid Conclusions

The Phishing Benchmark That Sparked the Debate

A third-party test used 2,000 emails (1,000 phishing, 1,000 legitimate) to compare Jev and Claude Haiku 4.5. When asked directly "Is this a phishing email?", Jev scored 62.6% accuracy versus Claude's 81.3%.

Then the testers changed the method: instead of a direct verdict, they asked Jev to judge five simpler signals:

Sender domain vs. link domain anomaly

Suspicious hosting service

Request for login or verification

Urgency pressure

Impersonation traits

Jev returned probabilities for each signal, e.g.:

Domain anomaly probability      0.91
Suspicious hosting probability  0.74
Login request probability       0.88
Urgency probability             0.63
Impersonation probability       0.81

These probabilities were fed into a trained logistic regression classifier, yielding a final accuracy of 95.0% .

The 95% result belongs to the whole system: human-designed decomposition + Jev signal extraction + supervised aggregator + final classification. The clever part is not all from Jev.

Why Phishing Is Unusually Decomposable

Phishing signals are mostly static, observable features. The email does not react to being analyzed; a suspicious domain stays suspicious regardless of the model's conclusion. This makes the task close to classic feature engineering: find stable features, learn a good combination function, achieve high accuracy.

Many real decisions are not so friendly, especially when:

Factors interact strongly

Current choices change the future environment

Each action affects the next action space

Opponents observe and react

A Concrete Example: Buying a Coat

A Jev-style system judges five attributes of a coat:

Looks good?        0.92
Cheap?             0.88
Quality?           0.81
Comfortable?       0.90
Fits me?           0.86
Composite score = 0.874 → Buy

All judgments are reasonable. Then a new fact: you already own six nearly identical coats. The coat's intrinsic attributes haven't changed, but the decision should change because the marginal value to your wardrobe is near zero.

Add a promotion: "Buy 3, get 50% off." The system evaluates each coat separately, each time concluding "Buy." You end up with three coats, two never worn. The system optimized "Is this coat worth buying?" but the real goal is "How to allocate a limited budget across the whole purchase set for maximum long-term utility?"

Game Theory Perspective

When a merchant introduces "Second item half price," the local price of the second item drops, making each step look rational. But the global alternatives are:

Option A: Buy one, spend 500
Option B: Buy two, spend 750
Option C: Buy none, wait until needed

The merchant's mechanism design exploits local rationality to steer you toward their optimum, not yours. A decision system that only asks "Is this step a good deal?" can be manipulated.

Missing Causal Chains in Investment Analysis

An AI judges a company on five signals (industry growth, product quality, management, valuation, customer satisfaction), all scoring high. But the real risk lies in a causal chain: two clients provide 70% of revenue; one is building a replacement; the other is in talks with a competitor. Compressing the problem into independent probabilities erases that relational structure. The aggregator cannot recover relationships that were never in the input.

Agent Scenarios Amplify the Problem

In a research agent, each step asks "Continue searching?" Jev returns probabilities like 0.81, 0.76, 0.73 — each step seems justified because new information might appear. The agent searches 30 times, accumulates duplicate context, buries key evidence, and blows up token usage. The local question "Does one more search have value?" differs from the global objective "Maximize answer quality per search cost, time, and error risk."

Where Jev Shines: Judgment, Not Planning

Jev excels at well-bounded judgments:

Is this page relevant?
Is this source duplicate?
Does this evidence support the proposition?
Which skill should handle this request?
Does the result meet a clear criterion?

Planning requires maintaining long-term goals, history, resource constraints, action dependencies, future state transitions, opponent strategies, and opportunity costs — a single local probability cannot carry all that.

How to Interpret the Phishing Benchmark

In tasks that decompose well, breaking a complex judgment into multiple stable features and processing them with a dedicated aggregator can significantly improve results.

That conclusion is valuable on its own. It should not be stretched to "All complex reasoning can be decomposed into simple probability judgments." Phishing naturally suits feature engineering; investment, agent planning, negotiation, resource allocation, competition, and long-term strategy involve interactions, feedback loops, and state transitions where the decomposition itself is part of the reasoning. If the decomposition is wrong, every downstream model can be perfectly accurate yet the system fails.

The Real Benchmark for Agent Decision Layers

Testing only classification accuracy or probability calibration is insufficient. The critical tests are:

Overall task completion rate
Quality of the entire agent trajectory
Does it fall into local optima?
Do consecutive correct small decisions accumulate into a large error?
Does it survive opponent strategy shifts?
How does it handle conflicts between long-term goals and short-term signals?

Jev proves that many judgments previously done via LLM text generation can be compressed into cheap, fast probability decisions. But when we decompose a complex task into dozens of neat probabilities, we must ask: Do these probabilities retain the information needed for the final decision? If not, the system exhibits a bizarre failure mode: every local judgment withstands scrutiny, every number looks reasonable, every code rule executes correctly — yet the final conclusion is stupid.