Specula Finds 382 Deep Bugs in 67 Projects, Reducing Formal Verification to Hours

Overview

Specula is an open‑source system that combines large‑language‑model coding agents (Claude Code, Codex, Copilot CLI, etc.) with automated TLA+ model generation and model checking. The tool reads a target system’s source code, documentation, tests, issues and commit history, synthesizes correctness invariants, builds a formal model, runs a model checker, and feeds any counterexample back into the real code as a reproducible test.

Core Principles

The workflow is driven by concrete artifacts: each stage produces evidence (code snippets, issue links, execution traces) that the next stage consumes and validates. Decisions are made based on these artifacts, ensuring traceability and preventing the agent from drifting away from the actual system.

Four‑Step Automated Process

Extract invariants from system evidence – Specula analyzes code, comments, documentation, tests, issues and historical fixes to derive protocol‑level and implementation‑level invariants. In a MongoDB case study, 87.35% of invariants cite code or comments, and 74.34% reference issues, PRs or security notices.

Generate models around high‑risk scenarios – Using the invariants, Specula extracts high‑risk execution scenarios from the same evidence sources, creates customized TLA+ models that retain relevant variables, actions and faults while abstracting away irrelevant details. For ScyllaDB Raft, this revealed a new bug that caused a read‑barrier to stall during configuration changes.

Validate models with real execution traces – The system is instrumented to collect concrete execution traces. Specula checks whether each trace is accepted by the TLA+ model; a divergence pinpoints inconsistencies between model and implementation.

Replay counterexamples in the real code – When a model‑checking run violates an invariant, Specula translates the counterexample into a deterministic event sequence, forces the same interleaving in the live system, and packages the reproduction as a test.

Self‑Evolving Closed Loops

Two interdependent loops ensure continuous improvement. The model‑code consistency loop uses trace validation to keep the model faithful to the implementation, while the bug‑reproduction loop feeds unreproducible counterexamples back into model refinement or invariant adjustment.

Empirical Results

As of August 19 2026, Specula has been applied to 67 open‑source projects—including MongoDB, Etcd, ScyllaDB, HashiCorp Raft, RabbitMQ, GCC libgomp and LLVM libomp—discovering 382 deep bugs that are reproducible in the actual code base.

One highlighted case is a five‑year‑old deadlock in GCC’s OpenMP runtime (libgomp). The bug occurs only when all other threads are parked at a barrier and a wake‑up path omits a “pending task” flag, leading to a permanent deadlock.

In a controlled experiment on five representative systems, Specula found 62 verified bugs, whereas the baseline Claude Code agent found 2 and a Claude Code + TLA+ setup found 3. The advantage stems from scenario‑aware modeling, model‑code consistency checks, and the closed‑loop bug‑reproduction mechanism.

Performance-wise, an end‑to‑end run on 48 projects took between 1.43 h and 9.86 h (median 3.69 h) with token costs ranging from $19 to $168 (median $57), turning months‑long formal specification efforts into a matter of hours.

Practical Usage

Specula supports multiple coding agents and can be launched with two commands on a target system, or customized with user‑defined scenarios and formal models.