Can Structured Disambiguation Improve Text-to-SQL Accuracy?

01 ABSTRACT

The paper argues that as text-to-SQL systems move toward real-world deployment, ambiguity in user questions becomes a primary error source; such ambiguities are often subtle, domain- or data-specific, and can silently cause outputs to deviate from true intent, while traditional conversational clarification is inefficient, cognitively demanding, and misaligned with real workflows. The authors propose structured disambiguation, resolving ambiguity through explicit, constrained interactions rather than free-form dialogue. They construct ARCS, claimed as the first text-to-SQL benchmark featuring naturally occurring, unconstrained ambiguities over real-world databases, with complete annotations of all valid ambiguity points, interpretations, and SQL queries; experiments show the task remains challenging under ambiguity, with gpt-6-sol achieving only 51% end-to-end execution accuracy and no open-source model exceeding 27%.

02 KEY FINDINGS

1. Proposes structured disambiguation: explicit, constrained interactions replace free-form dialogue clarification to resolve ambiguity in text-to-SQL.
2. Builds the ARCS benchmark: over real-world databases, annotating naturally occurring ambiguity points, corresponding interpretations, and SQL queries.
3. Experiments show text-to-SQL remains challenging under ambiguity: gpt-6-sol reaches 51% end-to-end execution accuracy, and no open-source model exceeds 27%.