World-Model Policy Arbiter Integrates Frozen Policies for Goal-Conditioned Reinforcement Learning
ORIGINAL / World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
This work introduces a test-time framework that, without retraining policies, combines multiple frozen goal-conditioned policies into a portfolio, selecting the best policy at each state by imagining futures in a world model. Its novelty lies in comparing heterogeneous policies via a shared value function and avoiding excessive switching. On 18 state-based OGBench datasets, it improves macro-average success from 44% to 58%, offering a plug-and-play policy integration approach
01 ABSTRACT
The paper notes that no single algorithm performs best across environments, goals, and task phases in offline goal-conditioned reinforcement learning. The authors propose World-Model Policy Arbiter (WMPA), which takes a set of frozen policies as input, rolls out imagined futures in a world model, scores them with a shared goal-conditioned value function, and executes the highest-scoring policy for a short commitment interval. Under the official OGBench protocol, WMPA improves the macro-average success rate from 44% to 58%, with significant gains on 12 datasets.
02 KEY FINDINGS
- WMPA takes a set of frozen goal-conditioned policies as input and requires no retraining or task-specific knowledge.
- It rolls out each policy's imagined future in a world model, scores them with a shared goal-conditioned value function, and executes the highest-scoring policy for a short commitment interval.
- On 18 state-based OGBench datasets, macro-average success improves from 44% to 58%, with significant gains on 12 datasets, including +33 percentage points on cube-double-play and +36 percentage points on scene-play.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.LG