Certifying Reward Alignment in Portfolio Reinforcement Learning

01 ABSTRACT

In financial reinforcement learning, an agent may accurately optimize an additive risk-adjusted reward yet implement a different financial preference. The authors develop a decision audit for terminal mean-variance equilibrium that bounds the gain from changing one action using actor, reward, and continuation-critic residuals. Experiments show the method reduces policy distance and one-step deviation gain, but joint reward calibration does not improve realized investment performance.

02 KEY FINDINGS

1. A certificate is proposed that bounds the gain from changing one action using actor, reward, and continuation-critic residuals, ignoring state-constant critic offsets and accommodating convex action constraints.
2. Across nine controlled markets and twenty seeds, a continuation-aware actor-critic method reduces policy distance and one-step deviation gain.
3. In backtests over 432 months with seven covariance estimators and 10, 30, and 49 US industry portfolios, joint reward calibration preserves fitted mean-variance targets but does not improve realized investment performance.