CodeInsight: A Large-Scale Dataset and Benchmark for Modeling Iterative Problem-Solving in Programming

01 ABSTRACT

The research frames iterative problem-solving as a sequential modeling task: at each step, a solver receives feedback and decides how to revise, with the central question being whether performance improves, plateaus, or regresses. The authors curate CodeInsight, covering over 3 million submissions from 3,286 undergraduates in two introductory C++ courses over two academic years, including test-case-level outcomes, timestamps, and source code, and build a benchmark for multiple model classes. Experiments show that an adapted Recurrent State Space Model (RSSM) achieves the strongest predictive accuracy on three of four courses; the LLM predictor is less accurate but generates full submissions, and its coding proficiency is inversely related to predictive performance in this setting.

02 KEY FINDINGS

1. CodeInsight contains over 3 million submissions from 3,286 undergraduates across two introductory C++ courses in two academic years, with test-case-level outcomes, timestamps, and source code.
2. The benchmark compares parametric, sequential, and generative models under a shared calibration-and-scoring protocol, including an adapted Recurrent State Space Model (RSSM) and an LLM-based predictor.
3. The adapted RSSM achieves the highest predictive accuracy on three of four courses; the LLM predictor is less accurate but produces full submissions at each attempt, enabling direct failure-mode analysis.