From Vibe Coding to Vibe Research: Less Harness, More Human Guidance

Core Thesis: Harness Subtraction, Human Key Guidance

As model capability increases, the required harness (prompt engineering, complex rules, skills) should decrease. Overly complex prompts and rules interfere with SOTA LLMs (GPT-6 Astra, Opus 5.5, Fable 5.1, Gemini 4 Argon). The human role is to provide correct guidance at critical points: specify what to do, how much effort to spend, and how to verify correctness. This is the essence of human-in-the-loop value.

Boris Cherny's Minimal Prompt Example

Claude Code lead Boris Cherny shared a prompt given to Opus 5.5 to create an interactive website for an Acquired podcast episode. The prompt was remarkably simple: describe the goal, ask for excellence, suggest watercolor illustrations and historical photos in New Yorker/NYT style, mention OpenCV for image processing. The community reaction focused on the prompt's simplicity, confirming that SOTA models need only clear goals, effort budgets, and verification criteria.

Project Management: Cross-Session Collaboration via AGENTS.md

Context windows are limited (Codex 258k default, Claude Code 1M, Gemini 4 Argon 2M). At 50-80% utilization, models drop details and make errors. Cross-session collaboration is essential, anchored by rigorous project documentation.

Directory Structure

my-project/
├─ AGENTS.md          # Most critical!
├─ docs/              # Long-term docs
│  ├─ README.md       # Index
│  ├─ HANDOFF.md      # Cross-session entry point
│  ├─ DATASET.md      # Dataset description
│  └─ ENV.md          # Environment
├─ data/              # Symlinks to data disk
├─ experiments/       # Results (symlinks if large)
├─ notes/             # Drafts, decisions, brainstorming
├─ ref-papers/        # LaTeX sources from arXiv
├─ ref-repos/         # Reference implementations
└─ workspace/         # Actual development

AGENTS.md is mandatory. Human-in-the-loop creation and maintenance: when an agent repeats an error, add the correction to AGENTS.md. The author's AGENTS.md includes sections for directory ownership, environment/execution rules, testing philosophy (default no unit tests; verify only core changes with minimized reproduction; match published metrics for reproduction), and documentation discipline (HANDOFF.md 40-70 lines, delete anything not needed for handoff).

Thinking Effort Selection

Codex Ultra/Max Pitfalls

Ultra/Max generate excessive subagents that debate endlessly, nitpick meaningless edge cases, bloat the codebase, waste tokens, and corrupt context. Root cause: GPT's over-defensive, over-cautious nature; the main agent distrusts subagents and double-checks their work, wasting tokens and polluting context. Claude Code's Ultracode feature outperforms Codex significantly.

Decouple Planning and Execution

Use higher effort for planning, lower for execution. This means opening parallel sessions (not switching effort/model mid-conversation, which destroys KV/prompt cache and increases cost). Exception: latest Claude models (Fable 5.1, Opus 5.5) allow effort switching via /effort without breaking cache.

Anthropic Researcher Thariq's 370 Experiments

Thariq ran Fable 5.1 on Terminal-Bench 3.0 across low-to-max effort. Macro insight: effort signals desired compute allocation, correlating with task difficulty modeling. Effort does not raise intrinsic intelligence; it makes the model spend more cycles thinking and verifying. High effort helps tasks with many edge cases (legacy bug fixes, security, performance) but causes overthinking on simple tasks. Effort must match task difficulty and desired human involvement.

Thariq's Heuristic Workflow

Give Claude a spec; let it ask clarifying questions.

Implement with low effort.

Check direction; iterate with low effort if off-track.

Switch to high effort for verification and testing.

Testing & Code Review

Unit Test Bloat

SOTA LLMs (from GPT-5.6 Sol) autonomously write excessive tests, bloating code and wasting tokens. OpenAI docs note older models needed prompting to test; new models test automatically. Legacy "remember to run tests" instructions in AGENTS.md exacerbate this. Remove such stale instructions.

Three Rules from a Developer's AGENTS.md

Never write unit tests after code.

Strongly prefer end-to-end (E2E) tests as the sole test method; E2E must produce a verifiable, reproducible artifact.

If module testing is needed, first list all failure modes, then write code.

LLMs favor static unit tests (test piecewise) over system-level validation; E2E fits research scenarios better.

Endless Code Review

LLMs find endless issues, many false positives, because they don't know when to stop — they merely appease the prompt. Clean context, concise constrained prompts, predefined completion criteria, and deep architectural understanding help. Same problem applies to paper reviewing: best papers can be trashed.

Prompt & Skills Best Practices

Don't rely on GPT-written prompts (verbose, over-constrained, hurt readability, trap in local optima). Ask the model to include: harness should subtract (fewer constraints), progressive disclosure (root doc points to children, read on demand), avoid detailed checklists (over-specific guidance backfires), regularly audit AGENTS.md for obsolete lines (replace "read xxx.md before every edit" with "check database.md when modifying schema"), define completion criteria upfront (old hard constraints may cause premature stops).

Recommended Skill: Research-Paper-Writing-Skills

GitHub: Master-cai. Good skills are concise, like ad-hoc prompts, yet effective. Contrast with grill-me (sometimes over-questions, stalls when architecture unclear) and large auto-research skills (redundant, noisy, mediocre ideas/output).

Research Workflow: Survey & Idea Generation

Don't Trust LLM-Generated Ideas Directly

LLMs lack research taste; strong human-in-the-loop required.

Use Pro for Survey

Web Pro (especially GPT-6 Astra Pro) excels at research but lacks coding harness; Codex + weaker model beats Web Pro for code. Pro tends to rabbit-hole; adjust prompts, supplement with keyword searches. It often repeats the same few papers, missing others.

Idea Generation Process

Run 2-3 independent Pro surveys; organize outputs into Project source.

Deep-read selected papers; pick 2-3 most promising/combinable — relies on your taste and intuition (critical step).

Feed chosen papers + prior survey into Project source; final Q&A round (iterate until satisfied), discussing your understanding and attraction points.

Don't Trust GPT Novelty Judgments

GPT searches papers, reads only abstracts, not methods/contributions, and concludes "paper A did this, paper B did that, so insufficient." Human reading often reveals GPT's summary is wrong. Human-in-the-loop mandatory.

Benchmark Design Principles

A good benchmark must answer two layers of questions.

Layer 1: Three Top-Level Questions

(Referenced image illustrates: What capability? Why this capability? How to measure?)

Layer 2: Six Organizational Questions

Scope & Boundary: Core problem, task cohesion, explicit exclusions.

Taxonomy: Problem space split rationale; categories must distinguish meaningful capabilities/difficulties and guide data coverage and analysis.

Task & Design: How abstract capabilities map to concrete I/O, success criteria, data constraints; justification for each key design.

Verification & Diagnosis: Experiments must test the benchmark itself, not just models; distinguish capability gaps from data/interface/scoring issues.

Insight & Contribution: Reveal evidenced capability boundaries or failure patterns, not just rankings and "models still not good enough."

Reusability & Continuity: Can others independently run, verify, and continuously use this evaluation for cumulative progress?

Paper Reading Tips & Miscellaneous

Translate Papers to Chinese for Speed

Author now translates papers via Codex (GPT-6 Luna Medium sufficient): fetch arXiv LaTeX source, translate sentence-by-sentence, recompile to Chinese PDF (parallelizable via subagents). High quality, clear layout. Prevent "token" → "词元" by prompt tweak. Decouples reading from English practice.

Miscellaneous Tips

Codex Fast mode costs ~2-2.5x tokens for only 1.2-1.5x speed; evaluate trade-off.

Use ChatGPT Web Project feature: restart chat or branch if unsatisfied; avoid "is xxx better?" follow-ups (elicits sycophantic "You're right!").

Codex Side chat is the best feature (equivalent to Claude Code /btw).

LLM leaderboard: Artificial Analysis (authoritative).

Model strengths (personal experience): summarized in referenced image.