Continuous-Utility Direct Preference Optimization Improves Reasoning Strategy Selection

01 ABSTRACT

The work argues that LLM reasoning is often treated as a monolithic capability, and binary preference supervision fails to capture partial progress or fine-grained reasoning quality. The authors propose CU-DPO, which replaces binary labels with continuous utility scores and proves a Θ(K log K) sample-complexity improvement over binary preferences with convergence to an entropy-regularized utility-maximizing policy. They report strategy-selection accuracy improving from 35-46% to 68-78% across seven base models, downstream reasoning gains up to +6.6 points, and consistent gains on code generation and causal reasoning benchmarks.

02 KEY FINDINGS

1. CU-DPO replaces binary preferences with continuous utility scores, decomposing reasoning into a portfolio of K prompt-based cognitive strategies.
2. Theoretical analysis claims a Θ(K log K) sample-complexity improvement and convergence of DPO to an entropy-regularized utility-maximizing policy.
3. A two-stage pipeline first selects strategies via best-vs-all comparisons and then refines correct execution with margin-stratified pairs; the authors report 68-78% strategy-selection accuracy across seven base models and up to +6.6 downstream points.