Closing the Real-World Gap for Code Models: SEAlign Improves Software Agent Decision Quality

Recent advances in large code models and code agents achieve strong results on classic generation benchmarks, yet their performance drops dramatically when deployed in real software engineering environments. The authors argue that this gap stems from the fact that real engineering is a long‑running, context‑rich, iterative process requiring continuous decision‑making, not a single isolated coding problem.

Through analysis of failure trajectories, the paper identifies three dominant mismatches: insufficient instruction following, erroneous tool invocation, and repetitive loops that waste context and compute. These findings motivate the need for an alignment approach that goes beyond token‑level correctness.

SEAlign Framework

SEAlign addresses the problem in three stages:

Trajectory Data Collection : Agents are run in authentic software engineering settings, recording full decision trajectories and labeling each run as success or failure while excluding any test‑set repositories to avoid data leakage.

Trajectory Tree Construction & Key Action Identification : Shared prefixes among trajectories are merged into a tree; low‑quality samples (e.g., loops without progress, outlier paths) are filtered out, enabling focus on decisive decision nodes.

Preference‑Based Alignment Training : Using a Monte‑Carlo‑style node scoring, pairs of actions at the same prefix are compared, and the model is trained via preference learning to favor the action that leads to a significantly higher success probability.

This design treats software‑engineering ability as a series of critical decision points, aligning training signals with those points rather than uniformly optimizing every token.

Experimental Results

On the SWE‑Bench suite, SEAlign‑14B improves problem‑solving rates from 3.7% to 17.7% (SWE‑Bench‑Lite) and from 2.8% to 21.8% (SWE‑Bench‑Verified). It also reduces empty‑patch rates from 52.0% to 22.8% and stuck‑in‑loop rates from 27.8% to 15.6%.

For the HumanEvalFix repair benchmark, the baseline Qwen‑2.5‑Coder‑Instruct‑14B drops from 54.3% Pass@1 to 31.1% when used with an agent, whereas SEAlign‑14B rises from 52.4% (no agent) to 62.8% with the agent and cuts invalid‑patch rates to 10.4%.

Ablation studies show that removing fine‑grained key‑action optimization or the key‑action identification step degrades performance to as low as 5.3%, confirming the importance of these components.

Data‑scale experiments reveal a monotonic improvement: increasing training samples from 25% to 100% raises SWE‑Bench‑Lite success from 3.7% to 17.7%.

User‑Centric Evaluation

Five developers evaluated five simple application tasks (to‑do list, Snake game, weather app, Hacker News query, personalized homepage). SEAlign‑14B achieved higher scores on functionality (1.8 → 3.1), code quality (2.7 → 3.5), and aesthetics (2.0 → 3.2), indicating a noticeable improvement in perceived development experience.

Future Outlook

The authors conclude that the decisive capability of code models in real engineering lies not only in generating correct code but also in making sustained, correct decisions throughout the development workflow. SEAlign’s focus on trajectory alignment, tool usage, and process‑level training offers a practical path toward the engineering‑ready deployment of code agents.