LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding

30 May 2023  ·  Yi Tu, Ya Guo, Huan Chen, Jinyang Tang ·

Visually-rich Document Understanding (VrDU) has attracted much research attention over the past years. Pre-trained models on a large number of document images with transformer-based backbones have led to significant performance gains in this field. The major challenge is how to fusion the different modalities (text, layout, and image) of the documents in a unified model with different pre-training tasks. This paper focuses on improving text-layout interactions and proposes a novel multi-modal pre-training model, LayoutMask. LayoutMask uses local 1D position, instead of global 1D position, as layout input and has two pre-training objectives: (1) Masked Language Modeling: predicting masked tokens with two novel masking strategies; (2) Masked Position Modeling: predicting masked 2D positions to improve layout representation learning. LayoutMask can enhance the interactions between text and layout modalities in a unified model and produce adaptive and robust multi-modal representations for downstream tasks. Experimental results show that our proposed method can achieve state-of-the-art results on a wide variety of VrDU problems, including form understanding, receipt understanding, and document image classification.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Key Information Extraction CORD LayoutMask (base) F1 96.99 # 4
Key Information Extraction CORD LayoutMask (large) F1 97.19 # 3
Named Entity Recognition (NER) CORD-r LayoutMask F1 81.84 # 4
Semantic entity labeling FUNSD LayoutMask (large) F1 93.20 # 1
Semantic entity labeling FUNSD LayoutMask (base) F1 92.91 # 3
Named Entity Recognition (NER) FUNSD-r LayoutMask F1 77.10 # 4


No methods listed for this paper. Add relevant methods here