A Statutory Article Retrieval Dataset in French

ACL 2022  ·  Antoine Louis, Gerasimos Spanakis ·

Statutory article retrieval is the task of automatically retrieving law articles relevant to a legal question. While recent advances in natural language processing have sparked considerable interest in many legal tasks, statutory article retrieval remains primarily untouched due to the scarcity of large-scale and high-quality annotated datasets. To address this bottleneck, we introduce the Belgian Statutory Article Retrieval Dataset (BSARD), which consists of 1,100+ French native legal questions labeled by experienced jurists with relevant articles from a corpus of 22,600+ Belgian law articles. Using BSARD, we benchmark several state-of-the-art retrieval approaches, including lexical and dense architectures, both in zero-shot and supervised setups. We find that fine-tuned dense retrieval models significantly outperform other systems. Our best performing baseline achieves 74.8% R@100, which is promising for the feasibility of the task and indicates there is still room for improvement. By the specificity of the domain and addressed task, BSARD presents a unique challenge problem for future research on legal information retrieval. Our dataset and source code are publicly available.

PDF Abstract ACL 2022 PDF ACL 2022 Abstract


Introduced in the Paper:


Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Information Retrieval BSARD Two-tower Bi-Encoder (RoBERTa) Recall@100 74.78 # 1
Recall@200 78.04 # 2
Recall@500 83.39 # 2
Information Retrieval BSARD Siamese Bi-Encoder (RoBERTa) Recall@100 71.63 # 2
Recall@200 78.38 # 1
Recall@500 83.77 # 1
Information Retrieval BSARD BM25 Recall@100 51.33 # 3
Recall@200 56.78 # 3
Recall@500 64.71 # 3