Reranking for Neural Semantic Parsing

ACL 2019  ·  Pengcheng Yin, Graham Neubig ·

Semantic parsing considers the task of transducing natural language (NL) utterances into machine executable meaning representations (MRs). While neural network-based semantic parsers have achieved impressive improvements over previous methods, results are still far from perfect, and cursory manual inspection can easily identify obvious problems such as lack of adequacy or coherence of the generated MRs. This paper presents a simple approach to quickly iterate and improve the performance of an existing neural semantic parser by reranking an n-best list of predicted MRs, using features that are designed to fix observed problems with baseline models. We implement our reranker in a competitive neural semantic parser and test on four semantic parsing (GEO, ATIS) and Python code generation (Django, CoNaLa) tasks, improving the strong baseline parser by up to 5.7{\%} absolute in BLEU (CoNaLa) and 2.9{\%} in accuracy (Django), outperforming the best published neural parser results on all four datasets.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Code Generation CoNaLa Reranker BLEU 30.11 # 8
Exact Match Accuracy 2.8 # 3
Code Generation CoNaLa-Ext Reranker BLEU 19.85 # 5
Code Generation Django Reranker Accuracy 80.2 # 4

Methods


No methods listed for this paper. Add relevant methods here