AI 研究arXiv cs.LG5 Oct, 12:00 pm SGTSIGNAL BC0B19

VERSE: A Verified Self-Evolving Optimizer for Agent Harnesses

ORIGINAL / VERSE: Verified Self-Evolving Optimizer for Agent Harnesses

Existing methods typically fix the optimizer's own tools and procedures, improving only the target agent's prompts, tools, and workflow. VERSE enables the optimizer to also improve how it diagnoses failures, develops edits, and tests effects, using execution-based verification, failure replay, and step perturbation to enhance optimization. On SWE-rebench in-distribution and cross-language out-of-distribution tasks, its best validation-selected harness reaches 42.3% and 37.7%

01 ABSTRACT

The authors study whether an optimizer can improve another agent more effectively by also improving how it diagnoses failures, develops edits, and tests their effects. In a controlled study, optimizer self-evolution fails to improve performance without execution-based verification but achieves the best result of that study when verification is available. Based on this, VERSE tests draft edits, replays failures, and perturbs suspected steps before submission while tracking fixes and regressions across rounds. The weights of the optimizer and executor models stay fixed; only the executor harness and the optimizer's own prompts, skills, tools, hooks, and notes are revised.

02 KEY FINDINGS

In the controlled study, optimizer self-evolution fails to improve performance without execution-based verification, but achieves the best result of that study when verification is available.
VERSE lets the optimizer test draft edits, replay failures, and perturb suspected steps before submission, while tracking fixes and regressions across rounds.
Under a shared protocol with disjoint training, validation, and test tasks, VERSE improves all four evaluated harness optimizers on held-out SWE-rebench tasks and newer out-of-distribution tasks in five languages.

03 RELEVANCE

This work suggests that agent harness optimization can come not only from modifying the target agent but also from improving the optimizer's own diagnosis and verification process; mechanisms such as execution-based verification, failure replay, and step perturbation may improve optimization and can transfer across multiple harnesses and language tasks without changing model weights.

04 EVIDENCE BOUNDARY

The evidence comes from the paper's controlled study, shared protocol, and tasks from SWE-rebench and five languages; reported results include accuracy of the best validation-selected harness, and do not show generalization to other task distributions, other models, other verification mechanisms, or real production environments. The weights of the optimizer and executor models stay fixed, so the conclusions do not directly apply to settings involving weight updates.

Return to the primary source

AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.CL · ARXIV CS.LG

Read original ↗