System1-Arena: Which Decisions Can Be Left to Single-Pass System 1 Models?
ORIGINAL / System1-Arena: Which Decisions Can Be Left to System 1?
Novelty: a unified probability-scoring benchmark with 105 tasks and 138 closed-loop environments to directly compare System-1 models, open LLMs, and closed LLMs on decision-making. Utility: it shows where System-1 can replace fast LLMs in non-agentic settings and where closed-loop control still needs large models.
01 ABSTRACT
The study releases System1-Arena, a benchmark that gives all models the same decision requests and scores their option probabilities. The largest System-1 models match or exceed fast closed LLMs on non-agentic accuracy with better-calibrated probabilities at one-quarter of the latency, while a reasoning LLM is most accurate at about 70x the cost of a fast LLM. In closed-loop environments, large open LLMs succeed most often (up to 32.9%), and small System-1 models rarely succeed.
02 KEY FINDINGS
- The benchmark contains 92k questions across 105 tasks from 268 public sources, grouped into text classification and knowledge, applied decisions, evaluation and control, and agent and action.
- It evaluates 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models that return option probabilities in a single forward pass.
- In non-agentic settings, the largest System-1 models match or exceed fast closed LLMs with better-calibrated probabilities at about one-quarter of the latency; a reasoning LLM is most accurate at about 70x the cost of a fast LLM.
- In closed-loop environments, large open LLMs succeed most often (up to 32.9%) while small System-1 models rarely succeed; in real-time mode the world evolves while the model decides.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.CL · ARXIV CS.LG