Simple Applications of BERT for Ad Hoc Document Retrieval

26 Mar 2019 Wei Yang Haotian Zhang Jimmy Lin

Following recent successes in applying BERT to question answering, we explore simple applications to ad hoc document retrieval. This required confronting the challenge posed by documents that are typically longer than the length of input BERT was designed to handle... (read more)

PDF Abstract

Datasets


  Add Datasets introduced or used in this paper

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Ad-Hoc Information Retrieval TREC Robust04 BERT FT(Microblog) MAP 0.3278 # 2
P@20 0.4287 # 5

Methods used in the Paper


METHOD TYPE
Residual Connection
Skip Connections
Attention Dropout
Regularization
Linear Warmup With Linear Decay
Learning Rate Schedules
Weight Decay
Regularization
GELU
Activation Functions
Dense Connections
Feedforward Networks
Adam
Stochastic Optimization
WordPiece
Subword Segmentation
Softmax
Output Functions
Dropout
Regularization
Multi-Head Attention
Attention Modules
Layer Normalization
Normalization
Scaled Dot-Product Attention
Attention Mechanisms
BERT
Language Models