Reflective Prompt Optimization: Topic-Disjoint Multi-Seed Evaluation of Transferable Gains

01 ABSTRACT

Under a topic-disjoint split, reflective prompt optimization raises held-out citation reward from 0.702 to 0.781, and the gain replicates across three seeds. The authors find that nearly half of the winning prompt is a memorized answer key, yet ablation shows this content earns no reward even on the encoded training questions; the validation-minus-held-out gap is mainly small-sample selection under a noisy judge. The authors argue decomposed per-dimension feedback is the causal ingredient for transfer, and collapsing it to a single scalar reward makes held-out transfer negative.

02 KEY FINDINGS

1. Under topic-disjoint split, held-out citation reward improves by 0.079 (bootstrap 95% CI [0.051, 0.107]), with mean 0.107 across three seeds.
2. Nearly half of the winning prompt is a memorized answer key, but ablation shows it earns no reward even on encoded training questions, so the validation-minus-held-out gap is small-sample selection rather than memorization.
3. Decomposed per-dimension feedback is the causal ingredient for transferable gain; collapsing it to a single scalar reward makes held-out transfer negative.