TextualRL: Textual Reinforcement Learning for Context Management to Improve Agent Performance
ORIGINAL / TextualRL: Textual Reinforcement Learning for Context Management
TextualRL introduces a gradient-free reinforcement learning framework that treats multiple rollouts from the same task under the same context as the unit of policy critique, generates bounded textual patches through groupwise and cross-group review, and decides adoption via held-out validation rewards. It offers a new way to turn execution rewards into reusable context updates without fine-tuning model parameters, potentially improving LLM agents.
01 ABSTRACT
The paper states that LLM agents can adapt to task feedback by revising their textual context without changing model parameters, but turning execution rewards into reusable context updates is challenging. TextualRL is a gradient-free framework that uses a group of rollouts from the same task under the same context as the unit of policy critique; it combines Groupwise Policy Critique and Cross-Group Policy Update to form a bounded textual patch, with adoption determined by held-out validation rewards. The authors evaluate TextualRL with five target agents on six benchmarks and report average test accuracy improvements over the base model, but these are author-reported results that require independent replication.
02 KEY FINDINGS
- TextualRL uses a group of rollouts from the same task under the same context, rather than an isolated trajectory, as the unit of policy critique, contrasting successful and failed rollouts to extract revision signals.
- When outcomes are uniform, it draws on cross-task evidence to identify behaviors worth preserving or to formulate repair hypotheses, which are then reviewed against evidence pooled across groups.
- Reviewed proposals form a bounded textual patch, and adoption of the candidate context is determined by held-out validation rewards, establishing a reward-driven policy optimization loop.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.CL