TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Natural Language Understanding	LexGLUE	BERT	ECtHR Task A	71.4 / 64.0	# 1
Natural Language Understanding	LexGLUE	BERT	ECtHR Task B	87.6 / 77.8	# 4
Natural Language Understanding	LexGLUE	BERT	SCOTUS	70.5 / 60.9	# 6
Natural Language Understanding	LexGLUE	BERT	EUR-LEX	71.6 / 55.6	# 5
Natural Language Understanding	LexGLUE	BERT	LEDGAR	87.7 / 82.2	# 5
Natural Language Understanding	LexGLUE	BERT	UNFAIR-ToS	87.5 / 81.0	# 5
Natural Language Understanding	LexGLUE	BERT	CaseHOLD	70.7	# 6
Natural Language Understanding	LexGLUE	CaseLaw-BERT	ECtHR Task A	71.2 / 64.2	# 2
Natural Language Understanding	LexGLUE	CaseLaw-BERT	ECtHR Task B	88.0 / 77.5	# 1
Natural Language Understanding	LexGLUE	CaseLaw-BERT	SCOTUS	76.4 / 66.2	# 8
Natural Language Understanding	LexGLUE	CaseLaw-BERT	EUR-LEX	71.0 / 55.9	# 6
Natural Language Understanding	LexGLUE	CaseLaw-BERT	LEDGAR	88.0 / 82.3	# 1
Natural Language Understanding	LexGLUE	CaseLaw-BERT	UNFAIR-ToS	88.3 / 81.0	# 1
Natural Language Understanding	LexGLUE	CaseLaw-BERT	CaseHOLD	75.6	# 1
Natural Language Understanding	LexGLUE	Legal-BERT	ECtHR Task A	71.2 / 64.6	# 2
Natural Language Understanding	LexGLUE	Legal-BERT	ECtHR Task B	88.0 / 77.2	# 1
Natural Language Understanding	LexGLUE	Legal-BERT	SCOTUS	76.2 / 65.8	# 1
Natural Language Understanding	LexGLUE	Legal-BERT	EUR-LEX	72.2 / 56.2	# 1
Natural Language Understanding	LexGLUE	Legal-BERT	LEDGAR	88.1 / 82.7	# 8
Natural Language Understanding	LexGLUE	Legal-BERT	UNFAIR-ToS	88.6 / 82.3	# 7
Natural Language Understanding	LexGLUE	Legal-BERT	CaseHOLD	75.1	# 2
Natural Language Understanding	LexGLUE	BigBird	ECtHR Task A	70.5 / 63.8	# 4
Natural Language Understanding	LexGLUE	BigBird	ECtHR Task B	88.1 / 76.6	# 8
Natural Language Understanding	LexGLUE	BigBird	SCOTUS	71.7 / 61.4	# 4
Natural Language Understanding	LexGLUE	BigBird	EUR-LEX	71.8 / 56.6	# 3
Natural Language Understanding	LexGLUE	BigBird	LEDGAR	87.7 / 82.1	# 5
Natural Language Understanding	LexGLUE	BigBird	UNFAIR-ToS	87.7 / 80.2	# 2
Natural Language Understanding	LexGLUE	BigBird	CaseHOLD	70.4	# 7
Natural Language Understanding	LexGLUE	Longformer	ECtHR Task A	69.6 / 62.4	# 5
Natural Language Understanding	LexGLUE	Longformer	ECtHR Task B	88.0 / 77.8	# 1
Natural Language Understanding	LexGLUE	Longformer	SCOTUS	72.2 / 62.5	# 3
Natural Language Understanding	LexGLUE	Longformer	EUR-LEX	71.9 / 56.7	# 2
Natural Language Understanding	LexGLUE	Longformer	LEDGAR	87.7 / 82.3	# 5
Natural Language Understanding	LexGLUE	Longformer	UNFAIR-ToS	87.7 / 80.1	# 2
Natural Language Understanding	LexGLUE	Longformer	CaseHOLD	72.0	# 4
Natural Language Understanding	LexGLUE	DeBERTa	ECtHR Task A	69.1 / 61.2	# 7
Natural Language Understanding	LexGLUE	DeBERTa	ECtHR Task B	87.4 / 77.3	# 5
Natural Language Understanding	LexGLUE	DeBERTa	SCOTUS	70.0 / 60.0	# 7
Natural Language Understanding	LexGLUE	DeBERTa	EUR-LEX	72.3 / 57.2	# 8
Natural Language Understanding	LexGLUE	DeBERTa	LEDGAR	87.9 / 82.0	# 3
Natural Language Understanding	LexGLUE	DeBERTa	UNFAIR-ToS	87.2 / 78.8	# 6
Natural Language Understanding	LexGLUE	DeBERTa	CaseHOLD	72.1	# 3
Natural Language Understanding	LexGLUE	RoBERTa	ECtHR Task A	69.5 / 60.7	# 6
Natural Language Understanding	LexGLUE	RoBERTa	ECtHR Task B	87.2 / 77.3	# 6
Natural Language Understanding	LexGLUE	RoBERTa	SCOTUS	70.8 / 61.2	# 5
Natural Language Understanding	LexGLUE	RoBERTa	EUR-LEX	71.8 / 57.5	# 3
Natural Language Understanding	LexGLUE	RoBERTa	LEDGAR	87.9 / 82.1	# 3
Natural Language Understanding	LexGLUE	RoBERTa	UNFAIR-ToS	87.7 / 81.5	# 2
Natural Language Understanding	LexGLUE	RoBERTa	CaseHOLD	71.7	# 5

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/lexglue-a-benchmark-dataset-for-legal/natural-language-understanding-on-lexglue)](https://paperswithcode.com/sota/natural-language-understanding-on-lexglue?p=lexglue-a-benchmark-dataset-for-legal)`

LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

ACL 2022 · Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, Nikolaos Aletras ·

Laws and their interpretations, legal arguments and agreements\ are typically expressed in writing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) technologies can be a valuable tool to support legal practitioners in these endeavors. Their usefulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.

PDF Abstract ACL 2022 PDF ACL 2022 Abstract

Code

Add Remove Mark official

coastalcph/lex-glue official

↳ Quickstart in

Colab

193

Tasks

Add Remove

Multi-class Classification

Multi-Label Classification

Multiple Choice Question Answering (MCQA)

Natural Language Understanding

Open-Ended Question Answering

Datasets

Introduced in the Paper:

LexGLUE

Used in the Paper:

GLUE

SuperGLUE

ECHR ECtHR CaseHOLD

Results from the Paper

Edit

Ranked #1 on Natural Language Understanding on LexGLUE

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Natural Language Understanding	LexGLUE	BERT	ECtHR Task A	71.4 / 64.0	# 1	Compare
			ECtHR Task B	87.6 / 77.8	# 4	Compare
			SCOTUS	70.5 / 60.9	# 6	Compare
			EUR-LEX	71.6 / 55.6	# 5	Compare
			LEDGAR	87.7 / 82.2	# 5	Compare
			UNFAIR-ToS	87.5 / 81.0	# 5	Compare
			CaseHOLD	70.7	# 6	Compare
Natural Language Understanding	LexGLUE	CaseLaw-BERT	ECtHR Task A	71.2 / 64.2	# 2	Compare
			ECtHR Task B	88.0 / 77.5	# 1	Compare
			SCOTUS	76.4 / 66.2	# 8	Compare
			EUR-LEX	71.0 / 55.9	# 6	Compare
			LEDGAR	88.0 / 82.3	# 1	Compare
			UNFAIR-ToS	88.3 / 81.0	# 1	Compare
			CaseHOLD	75.6	# 1	Compare
Natural Language Understanding	LexGLUE	Legal-BERT	ECtHR Task A	71.2 / 64.6	# 2	Compare
			ECtHR Task B	88.0 / 77.2	# 1	Compare
			SCOTUS	76.2 / 65.8	# 1	Compare
			EUR-LEX	72.2 / 56.2	# 1	Compare
			LEDGAR	88.1 / 82.7	# 8	Compare
			UNFAIR-ToS	88.6 / 82.3	# 7	Compare
			CaseHOLD	75.1	# 2	Compare
Natural Language Understanding	LexGLUE	BigBird	ECtHR Task A	70.5 / 63.8	# 4	Compare
			ECtHR Task B	88.1 / 76.6	# 8	Compare
			SCOTUS	71.7 / 61.4	# 4	Compare
			EUR-LEX	71.8 / 56.6	# 3	Compare
			LEDGAR	87.7 / 82.1	# 5	Compare
			UNFAIR-ToS	87.7 / 80.2	# 2	Compare
			CaseHOLD	70.4	# 7	Compare
Natural Language Understanding	LexGLUE	Longformer	ECtHR Task A	69.6 / 62.4	# 5	Compare
			ECtHR Task B	88.0 / 77.8	# 1	Compare
			SCOTUS	72.2 / 62.5	# 3	Compare
			EUR-LEX	71.9 / 56.7	# 2	Compare
			LEDGAR	87.7 / 82.3	# 5	Compare
			UNFAIR-ToS	87.7 / 80.1	# 2	Compare
			CaseHOLD	72.0	# 4	Compare
Natural Language Understanding	LexGLUE	DeBERTa	ECtHR Task A	69.1 / 61.2	# 7	Compare
			ECtHR Task B	87.4 / 77.3	# 5	Compare
			SCOTUS	70.0 / 60.0	# 7	Compare
			EUR-LEX	72.3 / 57.2	# 8	Compare
			LEDGAR	87.9 / 82.0	# 3	Compare
			UNFAIR-ToS	87.2 / 78.8	# 6	Compare
			CaseHOLD	72.1	# 3	Compare
Natural Language Understanding	LexGLUE	RoBERTa	ECtHR Task A	69.5 / 60.7	# 6	Compare
			ECtHR Task B	87.2 / 77.3	# 6	Compare
			SCOTUS	70.8 / 61.2	# 5	Compare
			EUR-LEX	71.8 / 57.5	# 3	Compare
			LEDGAR	87.9 / 82.1	# 3	Compare
			UNFAIR-ToS	87.7 / 81.5	# 2	Compare
			CaseHOLD	71.7	# 5	Compare

Methods

Add Remove

No methods listed for this paper. Add relevant methods here

Edit Social Preview

LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove