Search Results

Unified Language Model Pre-training for Natural Language Understanding and Generation

microsoft/unilm NeurIPS 2019

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.

Ranked #2 on Generative Question Answering on CoQA (using extra training data)

Abstractive Text Summarization Document Summarization +5

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

microsoft/unilm 28 Feb 2020

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).

Ranked #3 on Question Generation on SQuAD1.1 (using extra training data)

Abstractive Text Summarization Language Modelling +2

LayoutLM: Pre-training of Text and Layout for Document Image Understanding

microsoft/unilm 31 Dec 2019

In this paper, we propose the \textbf{LayoutLM} to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents.

Document Image Classification Document Layout Analysis +1

LayoutReader: Pre-training of Text and Layout for Reading Order Detection

microsoft/unilm 26 Aug 2021

Reading order detection is the cornerstone to understanding visually-rich documents (e. g., receipts and forms).

Optical Character Recognition

TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models

microsoft/unilm 21 Sep 2021

Existing approaches for text recognition are usually built based on CNN for image understanding and RNN for char-level text generation.

Language Modelling Optical Character Recognition +1

Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering

microsoft/unilm 23 Oct 2020

Coupled with the availability of large scale datasets, deep learning architectures have enabled rapid progress on the Question Answering task.

Cross-Lingual Question Answering Data Augmentation +1

Discriminative Adversarial Search for Abstractive Summarization

microsoft/unilm ICML 2020

We introduce a novel approach for sequence decoding, Discriminative Adversarial Search (DAS), which has the desirable properties of alleviating the effects of exposure bias without requiring external metrics.

Abstractive Text Summarization Domain Adaptation

Pseudo-Masked Language Models for Unified Language Model Pre-Training

microsoft/unilm ICML 2020

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM).

Language Modelling Natural Language Understanding

LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

microsoft/unilm ACL 2021

Pre-training of text and layout has proved effective in a variety of visually-rich document understanding tasks due to its effective model architecture and the advantage of large-scale unlabeled scanned/digital-born documents.

Document Image Classification Language Modelling +1

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

microsoft/unilm NeurIPS 2020

The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher).

Cannot find the paper you are looking for? You can Submit a new open access paper.