🦫 BEAVER: An Enterprise Benchmark for Text-to-SQL

Peter Baile Chen1, Devin Yang1, Weiyue Li2, Fabian Wenz1,3, Yi Zhang4, Nesime Tatbul1,5, Michael Cafarella1, Çağatay Demiralp1,6, Michael Stonebraker1
1MIT, 2Harvard University, 3TU Munich, 4Greenshoe, Inc., 5Intel, 6AWS AI Labs
arXiv Dataset Code
Overview Leaderboard

BEAVER is a large-scale enterprise text-to-SQL dataset containing 9128 queries spanning 812 tables across 19 diverse domains. Of these, 7978 queries are publicly released, while the remaining portion is held out as a private test set. Queries and databases were collected from private organizations.
To facilitate fine-grained evaluation and analysis, we provide

  • annotations for five subtasks: multi-table retrieval, join key detection, column mapping, domain knowledge extraction, and query decomposition
  • three categories of queries: complex queries without domain knowledge, domain-specific queries with minimal complexity, and domain-specific complex queries

Example data

Representative BEAVER tasks with question, SQL, and subtask annotations.

db source

Question

SQL

Multi-Table retrieval

Tables used in SQL

Join Keys

Connections among used tables

Column Mapping

Mapping from question phrases to table columns(s)

Domain knowledge

Domain-specific predictates used in SQL

    Subquery Decomposition

    Decomposition of SQL into simpler sub-queries

    Citation

    If you find our data, code, or the paper helpful, please cite the paper: 

    @article{chen2024beaver,
title={BEAVER: an enterprise benchmark for text-to-sql},
  author={Chen, Peter Baile and Wenz, Fabian and Zhang, Yi and Yang, Devin and Choi, Justin and Tatbul, Nesime and Cafarella, Michael and Demiralp, {\c{C}}a{\u{g}}atay and Stonebraker, Michael},
  journal={arXiv preprint arXiv:2409.02038},
  year={2024}
}