TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Dialogue State Tracking	CoSQL	T5-3B + PICARD	question match accuracy	54.6	# 2
Dialogue State Tracking	CoSQL	T5-3B + PICARD	interaction match accuracy	23.7	# 3
Text-To-SQL	spider	T5-3B + PICARD	Exact Match Accuracy (Dev)	75.5	# 2
Text-To-SQL	spider	T5-3B + PICARD	Exact Match Accuracy (Test)	71.9	# 2
Semantic Parsing	spider	T5-3B + PICARD	Accuracy	71.9	# 5

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/picard-parsing-incrementally-for-constrained/dialogue-state-tracking-on-cosql)](https://paperswithcode.com/sota/dialogue-state-tracking-on-cosql?p=picard-parsing-incrementally-for-constrained)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/picard-parsing-incrementally-for-constrained/text-to-sql-on-spider)](https://paperswithcode.com/sota/text-to-sql-on-spider?p=picard-parsing-incrementally-for-constrained)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/picard-parsing-incrementally-for-constrained/semantic-parsing-on-spider)](https://paperswithcode.com/sota/semantic-parsing-on-spider?p=picard-parsing-incrementally-for-constrained)`

PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

EMNLP 2021 · Torsten Scholak, Nathan Schucher, Dzmitry Bahdanau ·

Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. We propose PICARD (code and trained models available at https://github.com/ElementAI/picard), a method for constraining auto-regressive decoders of language models through incremental parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state-of-the-art solutions.

PDF Abstract EMNLP 2021 PDF EMNLP 2021 Abstract

Code

Add Remove Mark official

ElementAI/picard official

302

servicenow/picard

302

Tasks

Add Remove

Dialogue State Tracking

Semantic Parsing

Text-To-SQL

Translation

valid

Datasets

SPIDER

Spider Spider-Realistic

CoSQL

Results from the Paper

Edit

Ranked #2 on Dialogue State Tracking on CoSQL

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Dialogue State Tracking	CoSQL	T5-3B + PICARD	question match accuracy	54.6	# 2	Compare
Dialogue State Tracking	CoSQL	T5-3B + PICARD	interaction match accuracy	23.7	# 3	Compare
Text-To-SQL	spider	T5-3B + PICARD	Exact Match Accuracy (Dev)	75.5	# 2	Compare
Text-To-SQL	spider	T5-3B + PICARD	Exact Match Accuracy (Test)	71.9	# 2	Compare
Semantic Parsing	spider	T5-3B + PICARD	Accuracy	71.9	# 5	Compare

Methods

Add Remove

Adafactor • Attention Dropout • BPE • Dense Connections • Dropout • GELU • GLU • Inverse Square Root Schedule • Layer Normalization • Linear Layer • Multi-Head Attention • PICARD • Residual Connection • Scaled Dot-Product Attention • SentencePiece • Softmax • T5

Edit Social Preview

PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove