Search Results for author: Vatsal Raina

Found 19 papers, 8 papers with code

Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons

no code implementations9 May 2024 Adian Liusie, Vatsal Raina, Yassir Fathullah, Mark Gales

When Gaussian experts are used one can derive simple closed-form solutions for the optimal candidate ranking, as well as expressions for selecting which comparisons should be made to maximize the probability of this ranking.

Question Difficulty Ranking for Multiple-Choice Reading Comprehension

no code implementations16 Apr 2024 Vatsal Raina, Mark Gales

Additionally, zero-shot comparative assessment is more effective at difficulty ranking than the absolute assessment and even the task transfer approaches at question difficulty ranking with a Spearman's correlation of 40. 4%.

Multiple-choice Reading Comprehension

An Information-Theoretic Approach to Analyze NLP Classification Tasks

1 code implementation1 Feb 2024 Luran Wang, Mark Gales, Vatsal Raina

This work provides an information-theoretic framework to analyse the influence of inputs for text classification tasks.

Multiple-choice Reading Comprehension +4

Structural-Based Uncertainty in Deep Learning Across Anatomical Scales: Analysis in White Matter Lesion Segmentation

1 code implementation15 Nov 2023 Nataliia Molchanova, Vatsal Raina, Andrey Malinin, Francesco La Rosa, Adrien Depeursinge, Mark Gales, Cristina Granziera, Henning Muller, Mara Graziani, Meritxell Bach Cuadra

The results from a multi-centric MRI dataset of 334 patients demonstrate that our proposed measures more effectively capture model errors at the lesion and patient scales compared to measures that average voxel-scale uncertainty values.

Lesion Segmentation Uncertainty Quantification

Assessing Distractors in Multiple-Choice Tests

no code implementations8 Nov 2023 Vatsal Raina, Adian Liusie, Mark Gales

Specifically, we define quality in terms of the incorrectness, plausibility and diversity of the distractor options.

Multiple-choice Reading Comprehension

Analyzing Multiple-Choice Reading and Listening Comprehension Tests

no code implementations3 Jul 2023 Vatsal Raina, Adian Liusie, Mark Gales

Multiple-choice reading and listening comprehension tests are an important part of language assessment.

Multiple-choice Reading Comprehension +1

CUED at ProbSum 2023: Hierarchical Ensemble of Summarization Models

1 code implementation8 Jun 2023 Potsawee Manakul, Yassir Fathullah, Adian Liusie, Vyas Raina, Vatsal Raina, Mark Gales

In this paper, we consider the challenge of summarizing patients' medical progress notes in a limited data setting.

Tackling Bias in the Dice Similarity Coefficient: Introducing nDSC for White Matter Lesion Segmentation

1 code implementation10 Feb 2023 Vatsal Raina, Nataliia Molchanova, Mara Graziani, Andrey Malinin, Henning Muller, Meritxell Bach Cuadra, Mark Gales

This work describes a detailed analysis of the recently proposed normalised Dice Similarity Coefficient (nDSC) for binary segmentation tasks as an adaptation of DSC which scales the precision at a fixed recall rate to tackle this bias.

Lesion Segmentation Segmentation

World Knowledge in Multiple Choice Reading Comprehension

1 code implementation13 Nov 2022 Adian Liusie, Vatsal Raina, Mark Gales

Two metrics are described: the expected number of options, which measures whether a passage-free system can identify the answer a question using world knowledge; and the contextual mutual information, which measures the importance of context for a given question.

General Knowledge Multiple-choice +2

Multiple-Choice Question Generation: Towards an Automated Assessment Framework

no code implementations23 Sep 2022 Vatsal Raina, Mark Gales

Applying n-gram based approaches is challenging for this form of system as the reference set is unlikely to capture the full range of possible questions and answer options.

Multiple-choice Question Generation +2

Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks

3 code implementations15 Jul 2021 Andrey Malinin, Neil Band, Ganshin, Alexander, German Chesnokov, Yarin Gal, Mark J. F. Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Roginskiy, Denis, Mariya Shmatova, Panos Tigas, Boris Yangel

However, many tasks of practical interest have different modalities, such as tabular data, audio, text, or sensor data, which offer significant challenges involving regression and discrete or continuous structured prediction.

Image Classification Machine Translation +5

An Initial Investigation of Non-Native Spoken Question-Answering

no code implementations9 Jul 2021 Vatsal Raina, Mark J. F. Gales

The SQA task considered in this paper is to extract the answer from a candidate$\text{'}$s spoken response to a question in a prompt-response style language assessment test.

Question Answering Reading Comprehension +1

Complementary Systems for Off-Topic Spoken Response Detection

no code implementations WS 2020 Vatsal Raina, Mark Gales, Kate Knill

This paper examines one form of spoken language assessment; whether the response from the candidate is relevant to the prompt provided.

Data Augmentation

Cannot find the paper you are looking for? You can Submit a new open access paper.