Neural Language Modeling by Jointly Learning Syntax and Lexicon

We propose a neural language model capable of unsupervised syntactic structure induction. The model leverages the structure information to form better semantic representations and better language modeling. Standard recurrent neural networks are limited by their structure and fail to efficiently use syntactic information. On the other hand, tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation. In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network. Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word/character-level language model tasks.

PDF Abstract ICLR 2018 PDF ICLR 2018 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Constituency Grammar Induction PTB Diagnostic ECG Database PRPN (tuned) Max F1 (WSJ) 47.9 # 15
Mean F1 (WSJ) 47.3 # 21

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Source Paper Compare
Constituency Grammar Induction PTB Diagnostic ECG Database PRPN Max F1 (WSJ) 38.1 # 16

Methods


No methods listed for this paper. Add relevant methods here