Semi-Supervised Sequence Modeling with Cross-View Training

Unsupervised representation learning algorithms such as word2vec and ELMo improve the accuracy of many supervised NLP models, mainly because they can take advantage of large amounts of unlabeled text. However, the supervised models only learn from task-specific labeled data during the main training phase. We therefore propose Cross-View Training (CVT), a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. On labeled examples, standard supervised learning is used. On unlabeled examples, CVT teaches auxiliary prediction modules that see restricted views of the input (e.g., only part of a sentence) to match the predictions of the full model seeing the whole input. Since the auxiliary modules and the full model share intermediate representations, this in turn improves the full model. Moreover, we show that CVT is particularly effective when combined with multi-task learning. We evaluate CVT on five sequence tagging tasks, machine translation, and dependency parsing, achieving state-of-the-art results.

PDF Abstract EMNLP 2018 PDF EMNLP 2018 Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
CCG Supertagging CCGbank CVT + Multi-task + Large Accuracy 96.1 # 3
Named Entity Recognition (NER) CoNLL 2003 (English) CVT + Multi-Task + Large F1 92.61 # 36
Named Entity Recognition (NER) CoNLL 2003 (English) CVT + Multi-Task F1 92.61 # 36
Machine Translation IWSLT2015 English-Vietnamese CVT BLEU 29.6 # 6
Named Entity Recognition (NER) Ontonotes v5 (English) CVT + Multi-Task + Large F1 88.81 # 17
Part-Of-Speech Tagging Penn Treebank CVT + Multi-task Accuracy 97.76 # 5
Dependency Parsing Penn Treebank CVT + Multi-Task UAS 96.61 # 8
LAS 95.02 # 7