Early Tool Calling in Spoken Dialogue Systems
ORIGINAL / Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems
Introduces a synthetic spoken-dialogue pipeline that provides earliest-actionable-point supervision and trains an omni-modal language model for chunk-level causal decisions, enabling tool calls before the user finishes speaking.
01 ABSTRACT
The paper argues that latency-sensitive spoken agents should call tools as soon as enough information is heard, rather than waiting for turn completion. The authors construct a synthetic spoken dialogue dataset with earliest-actionable-point annotations and train an omni-modal model for causal chunk-level decisions. Experiments show tool-state macro-F1 of 0.48 and 0.47 on held-out dialogues and an external benchmark, with mean lead over a turn-end cascade of 1.8-2.0 seconds and 1.1 seconds for correct calls, respectively.
02 KEY FINDINGS
- Constructs a synthetic spoken dialogue dataset of 26,694 dialogues (786 hours) across 48 domains with earliest-actionable-point supervision.
- Trains an omni-modal language model for causal chunk-level tool-state decisions and partial-transcript generation.
- Achieves tool-state macro-F1 of 0.48 on held-out dialogues and 0.47 on an external benchmark.
- For correct calls, achieves mean lead over a turn-end cascade of 1.8-2.0 seconds in-domain and 1.1 seconds on the external benchmark.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.CL