AI 研究arXiv cs.CL8 Oct, 12:00 pm SGTSIGNAL 4A6722

Knowledge Boundary Probing and Demand-Guided Intervention for LLM-Based Power System Code Generation

ORIGINAL / Knowledge boundary probing and demand-guided intervention for LLM-based power system code generation

This study addresses first-pass failures in locally deployed LLM-generated power-system simulation code, proposing a deployment-time workflow that requires no weight updates: documentation-driven L0-L3 probes build per-model API profiles, query-side demand estimation selects layered API evidence before generation, and execution feedback routes targeted repair. Its value lies in turning common API hallucination and parameter misuse into a diagnosable and calibratable engineeri

01 ABSTRACT

Fact: The authors release PowerCodeBench, a frozen 2,000-task suite for pandapower, and propose a deployment-time workflow combining L0-L3 probing, query-side demand estimation, and execution-feedback repair without weight updates. Fact: Across ten open-weight LLMs (1.5B-480B) and four mid-tier APIs, the validation-enabled workflow raises scalar-match accuracy by 32-56 percentage points after up to three repair rounds relative to an unassisted first pass; selective injection approaches the full-layer reference using 41% of prompt tokens. Author view: such probing and demand-guided intervention can support diagnosis, model comparison, documentation allocation, and backend calibration, motivating staged onboarding from broad retrieval at cold start to calibrated selective injection.

02 KEY FINDINGS

Releases PowerCodeBench, a frozen 2,000-task benchmark for pandapower, with documentation-driven L0-L3 probes that produce per-model API profiles.
Deployment-time workflow requires no weight updates: a query-side demand estimator selects layered API evidence, and execution feedback routes targeted repair.
Across ten open-weight LLMs (1.5B-480B) and four mid-tier APIs, the validation-enabled workflow raises scalar-match accuracy by 32-56 percentage points after up to three repair rounds; 70B-120B open-weight models reach the four-vendor mid-tier accuracy range under matched no-tool conditions.

03 RELEVANCE

Power-system simulation code generation often demands on-premise deployment and is highly sensitive to API-call correctness and result-table handling. This work localizes first-pass failures to the API knowledge boundary and provides an actionable deployment-time diagnosis and intervention path, helping evaluate models, allocate documentation, and calibrate backends without weight fine-tuning, while offering a practical on-premise serving envelope for throughput, latency, energy, and allocated GPU memory.

04 EVIDENCE BOUNDARY

Evidence boundaries are explicit: results come from the authors' PowerCodeBench and specific pandapower tasks and should not be directly extrapolated to other libraries or domains; accuracy gains are measured under up to three repair rounds and validation-enabled conditions, so the benefit without validation needs separate evaluation; OpenDSS and PyPSA are only reported as round-0 pilots, constituting early motivational evidence; all comparisons are limited to ten open-weight models and four mid-tier APIs, not covering closed frontier models or broader toolchains; engineering review covers only model-item pairs passing numerical checks under both workflows, so the 88% versus 66% confirmation rates should not be generalized to all generated outputs.

Return to the primary source

AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.CL

Read original ↗