QuantEval: A Benchmark for LLMs in the Compute–Implement–Validate Loop of Quantitative Finance

01 ABSTRACT

The paper introduces QuantEval, a benchmark for evaluating whether LLMs can complete the compute–implement–validate loop required in quantitative finance, containing 1,575 expert-validated instances across knowledge-based QA, quantitative mathematical reasoning, and quantitative strategy coding. The coding task uses a deterministic CTA-style backtesting framework to execute model-generated strategies and compare risk–return metrics against expert references. Evaluating 13 open-source and proprietary LLMs reveals substantial gaps relative to human experts, especially in multi-step reasoning and framework-constrained coding; supervised fine-tuning and reinforcement learning on domain-aligned data can improve quantitative reasoning while exposing trade-offs in factual recall.

02 KEY FINDINGS

1. QuantEval contains 1,575 expert-validated instances spanning three complementary dimensions: knowledge-based QA, quantitative mathematical reasoning, and quantitative strategy coding.
2. The coding task executes model-generated strategies through a deterministic CTA-style backtesting framework and compares risk–return metrics against expert references, going beyond syntax or template matching.
3. Evaluation of 13 open-source and proprietary LLMs shows substantial gaps relative to human experts, especially in multi-step reasoning and framework-constrained coding; supervised fine-tuning and reinforcement learning on domain-aligned data can improve quantitative reasoning but also expose trade-offs in factual recall.