PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

10 Sep 2021  ·  Torsten Scholak, Nathan Schucher, Dzmitry Bahdanau ·

Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable... We propose PICARD (code and trained models available at https://github.com/ElementAI/picard), a method for constraining auto-regressive decoders of language models through incremental parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state-of-the-art solutions. read more

PDF Abstract


Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Dialogue State Tracking CoSQL T5-3B + PICARD question match accuracy 54.6 # 1
interaction match accuracy 23.7 # 1
Text-To-Sql spider T5-3B + PICARD Accuracy (Dev) 75.5 # 1
Accuracy (Test) 71.9 # 1
Semantic Parsing spider T5-3B + PICARD Accuracy 71.9 # 1