Fundamental Challenges in Evaluating Text2SQL Solutions and Detecting Their Limitations

In this work, we dive into the fundamental challenges of evaluating Text2SQL solutions and highlight potential failure causes and the potential risks of relying on aggregate metrics in existing benchmarks. We identify two largely unaddressed limitations in current open benchmarks: (1) data quality issues in the evaluation data mainly attributed to the lack of capturing the probabilistic nature of translating a natural language description into a structured query (e.g., NL ambiguity), and (2) the bias that using different match functions as approximations for SQL equivalence can introduce. To put both limitations into context, we propose a unified taxonomy over all Text2SQL limitations that can lead to both prediction and evaluation errors. We then motivate the taxonomy by providing a survey of Text2SQL limitations using state-of-the-art Text2SQL solutions and benchmarks. We describe causes of limitations with real-world examples and propose potential mitigation solutions for each of the categories in the taxonomy. We conclude by highlighting the open challenges when deploying such mitigation strategies or trying to automatically apply the taxonomy across categories.

† University of Waterloo