Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications

In this paper, we provide an overview of the SV-Ident shared task as part of the 3rd Workshop on Scholarly Document Processing (SDP) at COLING 2022. In the shared task, participants were provided with a sentence and a vocabulary of variables, and asked to identify which variables, if any, are mentioned in individual sentences from scholarly documents in full text. Two teams made a total of 9 submissions to the shared task leaderboard. While none of the teams improve on the baseline systems, we still draw insights from their submissions. Furthermore, we provide a detailed evaluation. Data and baselines for our shared task are freely available at https://github.com/vadis-project/sv-ident

PDF Abstract sdp (COLING) 2022 PDF sdp (COLING) 2022 Abstract


Introduced in the Paper:


Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Variable Disambiguation SV-Ident sentence-transformers/distiluse-base-multilingual-cased-v1 mAP@10 18.93 # 1
Variable Disambiguation SV-Ident SPARTA mAP@10 11.27 # 3
Variable Disambiguation SV-Ident BM25 mAP@10 9.43 # 4
Variable Disambiguation SV-Ident Sentence-T5 mAP@10 13.59 # 2
Variable Detection SV-Ident SsciBERT F1 66.1 # 1
Variable Detection SV-Ident Sentence-T5 F1 60.17 # 2


No methods listed for this paper. Add relevant methods here