Agent Planning Benchmark: Diagnosing LLM Planning Shortfalls and Tool Robustness
ORIGINAL / Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents
This study introduces the Agent Planning Benchmark (APB), which separates planning from execution and uses 4,209 multimodal cases across 22 domains and five settings to reveal systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement across 12 MLLMs, while validating that APB-guided refinement improves downstream execution metrics.
01 ABSTRACT
Fact: The study proposes APB, a planning-specific diagnostic benchmark with 4,209 multimodal cases across 22 domains and five settings, evaluates 12 MLLMs, and validates on 200 ToolSandbox and 200 τ²-bench tasks that APB-guided refinement improves plan correctness, plan grade, and execution metrics. Author's view: APB serves as an upstream diagnostic complement to execution benchmarks, helping distinguish planning from execution failures.
02 KEY FINDINGS
- APB contains 4,209 multimodal cases across 22 domains and five planning settings, specifically diagnosing planning capabilities rather than end-to-end success.
- It reveals systematic weaknesses in long-horizon planning, tool-noise robustness, calibrated refusal, and inference-time refinement across 12 MLLMs.
- On 200 ToolSandbox and 200 τ²-bench tasks, APB-guided refinement improves plan correctness, plan grade, and execution metrics.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.CL