When Should AI Write Your Code? Karpathy’s One‑Question Framework

Andrej Karpathy uses a simple judgment framework to decide which line of code should be written by a human and which can be handed to AI. The core question is “Where does the thinking happen?” If the problem still requires a solution, the work stays in the engineer’s mind; AI can only suggest options but cannot reliably infer decisions that have not yet been made.

When a task no longer requires judgment, it becomes an execution problem where AI can provide the greatest leverage. Predictable tasks such as CRUD APIs, documentation, database migrations, repetitive refactoring, testing, and boilerplate code have clear success criteria and are suitable for AI. A practical rule is: if two senior engineers would independently produce similar implementations, the task is appropriate for AI; otherwise, human judgment is required.

Karpathy does not lock himself into a single AI workflow; he switches among writing code himself, using code completion, or delegating the entire task to an AI agent, depending on the level of judgment the task demands.

Developer rvaniaaa built a system that embodies Karpathy’s framework: humans make the judgment, AI performs the execution. The system continuously discovers new ideas on GitHub, extracts reusable AI workflows, and packages them as standardized skills that AI agents can consume without human intervention.

High‑Level Architecture

The architecture resembles a pipeline where each agent performs a single responsibility and passes a structured result to the next, keeping components simple and maintainable.

1. Scout: Find Repositories

Scout monitors GitHub Trending and runs keyword searches to generate a candidate list of repositories that might contain new AI workflows or agent architectures. It does not read code or assess quality.

Query:
("agent framework" OR langgraph OR mcp OR "multi-agent")

Filters:
stars:>100
pushed:>2026-06-01
language:Python
archived:false
sort:updated-desc

Each discovered repository is represented as a structured object before entering the model:

{
  "name": "awesome-mcp-server",
  "stars": 842,
  "language": "Python",
  "last_updated": "2026-07-09T14:32:00Z",
  "status": "QUEUED"
}

Input: GitHub Trending, GitHub Search<br/>Output: List of newly discovered repositories

2. Filter: Remove Noise

Most repositories are irrelevant to reusable AI workflows. Deterministic rules discard obvious noise such as CSS libraries, game projects, or datasets, saving token and compute resources for later LLM steps.

Filter applies a fixed rule set rather than an LLM classifier. An example decision:

{
  "repository": "awesome-mcp-server",
  "category": "AI Agent",
  "language": "Python",
  "decision": "KEEP",
  "reason": "Contains reusable agent workflows"
}

Rejected example:

{
  "repository": "css-animation-library",
  "category": "CSS",
  "language": "TypeScript",
  "decision": "REJECT",
  "reason": "Outside AI workflow scope"
}

The goal is not perfect judgment, only to eliminate obvious noise so that expensive LLM steps handle only promising candidates.

3. Reader: Extract Information

Reader is a critical stage. Instead of loading the entire source tree, it incrementally builds context by reading documentation first, stopping when enough information is gathered.

README
   ↓
 docs/
   ↓
 examples/
   ↓
 package.json
   ↓
 requirements.txt
   ↓
 source code

Good repositories explain the intent in README and docs before requiring code inspection.

Reader produces incremental context rather than dumping the whole repo into the model:

{
  "repository": "langgraph-agent",
  "context_loaded": [
    "README",
    "docs/workflows.md",
    "examples/basic_agent.py"
  ],
  "source_code_loaded": false,
  "decision_reason": "Workflow identified before code analysis."
}

4. Workflow Extractor: Pull Out Reusable Workflows

The core component extracts a standardized representation of a reusable workflow. If no workflow is found, the repository is dropped; otherwise, the workflow is emitted as a skill definition.

{
  "skill_name": "github-pr-reviewer",
  "goal": "Review GitHub pull requests before merge",
  "inputs": [
    "Pull Request URL",
    "Repository Context"
  ],
  "steps": [
    "Read changed files",
    "Identify risky modifications",
    "Check coding standards",
    "Generate review comments"
  ],
  "outputs": [
    "Review Summary",
    "Action Items"
  ],
  "failure_modes": [
    "Missing project context",
    "Large architectural refactor"
  ]
}

5. Skill Score: Rule‑Based Evaluation

Not every extracted workflow should become an AI skill. A deterministic rule set evaluates each workflow; any mandatory check that fails results in rejection.

{
  "skill_name": "github-pr-reviewer",
  "score": 0.94,
  "checks": {
    "readme": true,
    "examples": true,
    "min_steps": true,
    "reusable": true,
    "general_purpose": true
  },
  "decision": "PASS"
}

Result example:

Result

PASS

Confidence: 94%

6. Skill Generator: Create Standardized Skill Package

When a workflow passes scoring, it is transformed into a uniform skill package.

name: github-pr-reviewer
version: 1.0.0

description: Review GitHub pull requests and generate actionable feedback.

inputs:
  - github_pull_request
  - repository_context

steps:
  - Read changed files
  - Identify risky modifications
  - Verify coding standards
  - Generate review comments

outputs:
  - review_summary
  - review_comments

tags:
  - github
  - code-review
  - engineering

Additional supporting files are generated automatically:

github-pr-reviewer/
├── SKILL.md
├── examples.md
├── commands.md
├── metadata.json
└── tests.md

7. Reviewer: Human Validation

Reviewer, a separate agent, decides whether the skill is worthy of publication. The generating agent never judges its own output.

Would an experienced engineer install this Skill without editing it?

Answer only:

YES or NO

Explain your reasoning in one paragraph.

Point out any missing assumptions or unclear steps.

If NO, suggest the minimum changes required for approval.

Do not rewrite the entire Skill.

Typical review result:

{
  "decision": "YES",
  "confidence": 0.96,
  "reason": "The workflow is reusable, clearly documented and can be applied outside the original repository."
}

If the answer is NO, the pipeline stops; if YES, it proceeds to publishing.

8. Publisher: Automated Pull Request Creation

Approved skills are published via a fully automated GitHub Action pipeline that creates a branch, commits the skill package, opens a pull request, and assigns a reviewer.

GitHub Action
      ↓
Create Branch
      ↓
Commit Skill Package
      ↓
Open Pull Request
      ↓
Assign Reviewer

The generated PR contains all required files:

Title:
Add Skill: github-pr-reviewer

Files:
✓ SKILL.md
✓ examples.md
✓ commands.md
✓ metadata.json
✓ tests.md

Automation proposes changes; humans retain final approval, ensuring reliability.

Conclusion

The system demonstrates that AI can handle execution‑level tasks when human judgment has already defined the problem. The bottleneck remains the discovery and extraction of reusable workflows from the vast amount of code on GitHub, which this pipeline automates while keeping the final decision in human hands.