TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Named Entity Recognition (NER)	BC2GM	PubMedBERT uncased	F1	84.52	# 10
Question Answering	BioASQ	PubMedBERT uncased	Accuracy	87.56	# 5
Sentence Similarity	BIOSSES	PubMedBERT uncased	Pearson Correlation	92.3	# 3
Text Classification	BLURB	PubMedBERT (uncased; abstracts)	F1	82.32	# 3
Question Answering	BLURB	PubMedBERT (uncased; abstracts)	Accuracy	71.7	# 3
Relation Extraction	ChemProt	PubMedBERT uncased	Micro F1	77.24	# 2
Relation Extraction	DDI	PubMedBERT uncased	Micro F1	82.36	# 2
Drug–drug Interaction Extraction	DDI extraction 2013 corpus	PubMedBERT	F1	0.8236	# 3
Drug–drug Interaction Extraction	DDI extraction 2013 corpus	PubMedBERT	Micro F1	82.36	# 3
Participant Intervention Comparison Outcome Extraction	EBM-NLP	PubMedBERT uncased	F1	73.38	# 2
PICO	EBM PICO	PubMedBERT uncased	Macro F1 word level	73.38	# 3
Relation Extraction	GAD	PubMedBERT uncased	Micro F1	82.34	# 2
Document Classification	HOC	PubMedBERT uncased	Micro F1	82.32	# 3
Named Entity Recognition (NER)	JNLPBA	PubMedBERT uncased	F1	79.1	# 9
Named Entity Recognition (NER)	NCBI-disease	PubMedBERT uncased	F1	87.82	# 17
Question Answering	PubMedQA	PubMedBERT uncased	Accuracy	55.84	# 23

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/relation-extraction-on-chemprot)](https://paperswithcode.com/sota/relation-extraction-on-chemprot?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/relation-extraction-on-ddi)](https://paperswithcode.com/sota/relation-extraction-on-ddi?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/participant-intervention-comparison-outcome)](https://paperswithcode.com/sota/participant-intervention-comparison-outcome?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/relation-extraction-on-gad)](https://paperswithcode.com/sota/relation-extraction-on-gad?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/sentence-similarity-on-biosses)](https://paperswithcode.com/sota/sentence-similarity-on-biosses?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/text-classification-on-blurb)](https://paperswithcode.com/sota/text-classification-on-blurb?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/question-answering-on-blurb)](https://paperswithcode.com/sota/question-answering-on-blurb?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/drug-drug-interaction-extraction-on-ddi)](https://paperswithcode.com/sota/drug-drug-interaction-extraction-on-ddi?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/pico-on-ebm-pico)](https://paperswithcode.com/sota/pico-on-ebm-pico?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/document-classification-on-hoc)](https://paperswithcode.com/sota/document-classification-on-hoc?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/question-answering-on-bioasq)](https://paperswithcode.com/sota/question-answering-on-bioasq?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/named-entity-recognition-ner-on-jnlpba)](https://paperswithcode.com/sota/named-entity-recognition-ner-on-jnlpba?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/named-entity-recognition-on-bc2gm)](https://paperswithcode.com/sota/named-entity-recognition-on-bc2gm?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/named-entity-recognition-ner-on-ncbi-disease)](https://paperswithcode.com/sota/named-entity-recognition-ner-on-ncbi-disease?p=domain-specific-language-model-pretraining)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/domain-specific-language-model-pretraining/question-answering-on-pubmedqa)](https://paperswithcode.com/sota/question-answering-on-pubmedqa?p=domain-specific-language-model-pretraining)`

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

31 Jul 2020 · Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon ·

Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly-available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition (NER). To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB.

PDF Abstract

Code

Add Remove Mark official

bionlu-coling2024/biomed-ner-intent…

Tasks

Add Remove

Continual Pretraining

Document Classification

Drug–drug Interaction Extraction

Language Modelling

named-entity-recognition

Named Entity Recognition

Named Entity Recognition (NER)

NER

Participant Intervention Comparison Outcome Extraction

PICO

Question Answering

Relation Extraction

Sentence Similarity

Text Classification

Datasets

Introduced in the Paper:

BLURB

Used in the Paper:

GLUE

SuperGLUE

BookCorpus

BioASQ NCBI Disease

PubMedQA BLUE

DDI

BIOSSES EBM-NLP

HOC JNLPBA ChemProt BC2GM GAD

Results from the Paper

Edit

Ranked #2 on Participant Intervention Comparison Outcome Extraction on EBM-NLP (using extra training data)

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Named Entity Recognition (NER)	BC2GM	PubMedBERT uncased	F1	84.52	# 10	Compare
Question Answering	BioASQ	PubMedBERT uncased	Accuracy	87.56	# 5	Compare
Sentence Similarity	BIOSSES	PubMedBERT uncased	Pearson Correlation	92.3	# 3	Compare
Text Classification	BLURB	PubMedBERT (uncased; abstracts)	F1	82.32	# 3	Compare
Question Answering	BLURB	PubMedBERT (uncased; abstracts)	Accuracy	71.7	# 3	Compare
Relation Extraction	ChemProt	PubMedBERT uncased	Micro F1	77.24	# 2	Compare
Relation Extraction	DDI	PubMedBERT uncased	Micro F1	82.36	# 2	Compare
Drug–drug Interaction Extraction	DDI extraction 2013 corpus	PubMedBERT	F1	0.8236	# 3	Compare
Drug–drug Interaction Extraction	DDI extraction 2013 corpus	PubMedBERT	Micro F1	82.36	# 3	Compare
Participant Intervention Comparison Outcome Extraction	EBM-NLP	PubMedBERT uncased	F1	73.38	# 2	Compare
PICO	EBM PICO	PubMedBERT uncased	Macro F1 word level	73.38	# 3	Compare
Relation Extraction	GAD	PubMedBERT uncased	Micro F1	82.34	# 2	Compare
Document Classification	HOC	PubMedBERT uncased	Micro F1	82.32	# 3	Compare
Named Entity Recognition (NER)	JNLPBA	PubMedBERT uncased	F1	79.1	# 9	Compare
Named Entity Recognition (NER)	NCBI-disease	PubMedBERT uncased	F1	87.82	# 17	Compare
Question Answering	PubMedQA	PubMedBERT uncased	Accuracy	55.84	# 23	Compare

Methods

Add Remove

Adam • Attention Dropout • BERT • Dense Connections • Dropout • GELU • Layer Normalization • Linear Layer • Linear Warmup With Linear Decay • Multi-Head Attention • Residual Connection • Scaled Dot-Product Attention • Softmax • Weight Decay • WordPiece

Edit Social Preview

Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove