Validating Agent Optimizations with Backtesting, Offline Experiments, and Question‑Level Rubrics

1. Experiment Plan: Dataset + Evaluator + Variable Mapping

When evaluation results are unsatisfactory, an experiment plan is created by selecting a dataset, checking the evaluator, and configuring variable mapping. Variable mapping links built‑in variables (input, output, trace) to dataset fields such as expected output, and custom variables can record version, experiment ID, or any KV pair.

2. Offline Experiment Platform (Deployed Inside the Intranet)

The platform connects to the AgentLoop cloud, binds an AgentSpace, sets the endpoint/VPC, and uses an AK/SK with FullAccess permissions. After configuration, a local execution code is generated to run tests against the internal Agent.

Connecting to a local Agent requires creating a connection (e.g., “Test1”). Because many agents are stateful, the default stateless call code can be replaced with a “create session → send message → get response” flow. A single‑question test (e.g., asking “Where am I?”) verifies connectivity.

3. Scheduled Runs and Launch History

Experiments can be scheduled (e.g., every 5 minutes) to turn regression testing into a routine. Launch History records each trigger (e.g., at minute 51 and 56) and keeps the cloud console and local platform synchronized.

4. Experiment Dashboard and Score Analysis

Each experiment generates a dashboard showing overall scores, trends, and per‑question details. In the demo, two questions received scores of 0.25 and 0.5 (out of 1). The dashboard supports drilling down by question, computing moving averages over configurable time windows, and spotting score drops.

5. Question‑Level Rubric

To avoid mixing evaluation criteria, Rubrics are defined per question. The process involves four steps:

Collect each question’s Rubric : copy the detailed scoring items, boundaries, and penalties.

Extend the dataset schema : add a rubric (text) column and store each question’s Rubric JSON.

Upgrade the evaluator : add a rubric variable so the evaluator scores according to the supplied Rubric and returns a JSON payload containing rubric_id, score, reason, and item_scores.

Bind Rubric in the experiment : configure the experiment to read the Rubric from the dataset entry.

After this refactor, each question is assessed against its own standard, making the scores actionable – low‑scoring questions point directly to the capability gaps that need improvement.

6. Takeaways

Combining backtesting, an offline intranet experiment platform, scheduled runs, a unified dashboard, and per‑question Rubrics creates a closed‑loop workflow that safely validates agent optimizations, prevents regressions, and provides clear guidance for the next round of improvements.