Benchmarks
| TREND | DATASET | BEST METHOD | PAPER TITLE | PAPER | CODE | COMPARE |
|---|---|---|---|---|---|---|
|
Datasets
Greatest papers with code
SeqGenSQL -- A Robust Sequence Generation Model for Structured Query Language
We explore using T5 (Raffel et al. (2019)) to directly translate natural language questions into SQL statements.
SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
Existing state-of-the-art approaches rely on reinforcement learning to reward the decoder when it generates any of the equivalent serializations.
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
A significant amount of the world's knowledge is stored in relational databases.
Ranked #7 on
Code Generation
on WikiSQL
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
ACL 2020
• facebookresearch/tabert
• 
Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks.
Ranked #1 on
Semantic Parsing
on WikiTableQuestions
Semantic Evaluation for Text-to-SQL with Distilled Test Suites
We propose test suite accuracy to approximate semantic accuracy for Text-to-SQL models.
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
We define a new complex and cross-domain semantic parsing and text-to-SQL task where different complex SQL queries and databases appear in train and test sets.
Ranked #4 on
Semantic Parsing
on spider
Improving Text-to-SQL Evaluation Methodology
ACL 2018
• jkkummerfeld/text2sql-data
• 
Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work.
Ranked #1 on
SQL Parsing
on ATIS
RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
ACL 2020
• Microsoft/rat-sql
•
The generalization challenge lies in (a) encoding the database relations in an accessible way for the semantic parser, and (b) modeling alignment between database columns and their mentions in a given query.
Ranked #3 on
Semantic Parsing
on spider
Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation
We present a neural approach called IRNet for complex and cross-domain Text-to-SQL.
CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
IJCNLP 2019
• ryanzhumich/editsql
•
We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems.