TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Variable Disambiguation	SV-Ident	sentence-transformers/distiluse-base-multilingual-cased-v1	mAP@10	18.93	# 1
Variable Disambiguation	SV-Ident	SPARTA	mAP@10	11.27	# 3
Variable Disambiguation	SV-Ident	BM25	mAP@10	9.43	# 4
Variable Disambiguation	SV-Ident	Sentence-T5	mAP@10	13.59	# 2
Variable Detection	SV-Ident	SsciBERT	F1	66.1	# 1
Variable Detection	SV-Ident	Sentence-T5	F1	60.17	# 2

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/overview-of-the-sv-ident-2022-shared-task-on/variable-disambiguation-on-sv-ident)](https://paperswithcode.com/sota/variable-disambiguation-on-sv-ident?p=overview-of-the-sv-ident-2022-shared-task-on)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/overview-of-the-sv-ident-2022-shared-task-on/variable-detection-on-sv-ident)](https://paperswithcode.com/sota/variable-detection-on-sv-ident?p=overview-of-the-sv-ident-2022-shared-task-on)`

Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications

sdp (COLING) 2022 · Tornike Tsereteli, Yavuz Selim Kartal, Simone Paolo Ponzetto, Andrea Zielinski, Kai Eckert, Philipp Mayr ·

In this paper, we provide an overview of the SV-Ident shared task as part of the 3rd Workshop on Scholarly Document Processing (SDP) at COLING 2022. In the shared task, participants were provided with a sentence and a vocabulary of variables, and asked to identify which variables, if any, are mentioned in individual sentences from scholarly documents in full text. Two teams made a total of 9 submissions to the shared task leaderboard. While none of the teams improve on the baseline systems, we still draw insights from their submissions. Furthermore, we provide a detailed evaluation. Data and baselines for our shared task are freely available at https://github.com/vadis-project/sv-ident

PDF Abstract sdp (COLING) 2022 PDF sdp (COLING) 2022 Abstract

Code

Add Remove Mark official

vadis-project/sv-ident official

Tasks

Add Remove

Sentence

Variable Detection

Variable Disambiguation

Datasets

Introduced in the Paper:

SV-Ident

Results from the Paper

Edit

Ranked #1 on Variable Disambiguation on SV-Ident

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Variable Disambiguation	SV-Ident	sentence-transformers/distiluse-base-multilingual-cased-v1	mAP@10	18.93	# 1	Compare
Variable Disambiguation	SV-Ident	SPARTA	mAP@10	11.27	# 3	Compare
Variable Disambiguation	SV-Ident	BM25	mAP@10	9.43	# 4	Compare
Variable Disambiguation	SV-Ident	Sentence-T5	mAP@10	13.59	# 2	Compare
Variable Detection	SV-Ident	SsciBERT	F1	66.1	# 1	Compare
Variable Detection	SV-Ident	Sentence-T5	F1	60.17	# 2	Compare

Methods

Add Remove

No methods listed for this paper. Add relevant methods here

Edit Social Preview

Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove