no code implementations • Findings (ACL) 2022 • Vatsal Raina, Mark Gales
In terms of an MRC system this means that the system is required to have an idea of the uncertainty in the predicted answer.
Ranked #8 on Reading Comprehension on ReClor
no code implementations • NAACL (BEA) 2022 • Yiting Lu, Stefano Bannò, Mark Gales
Due to a lack of end-to-end training data, SGEC is often implemented as a cascaded, modular system, consisting of speech recognition, disfluency removal, and grammatical error correction (GEC).
1 code implementation • 28 Feb 2024 • Akash Gupta, Ivaxi Sheth, Vyas Raina, Mark Gales, Mario Fritz
With the recent emergence of powerful instruction-tuned large language models (LLMs), various helpful conversational Artificial Intelligence (AI) systems have been deployed across many applications.
no code implementations • 21 Feb 2024 • Vyas Raina, Adian Liusie, Mark Gales
Large Language Models (LLMs) are powerful zero-shot assessors and are increasingly used in real-world situations such as for written exams or benchmarking systems.
1 code implementation • 1 Feb 2024 • Luran Wang, Mark Gales, Vatsal Raina
This work provides an information-theoretic framework to analyse the influence of inputs for text classification tasks.
1 code implementation • 15 Nov 2023 • Nataliia Molchanova, Vatsal Raina, Andrey Malinin, Francesco La Rosa, Adrien Depeursinge, Mark Gales, Cristina Granziera, Henning Muller, Mara Graziani, Meritxell Bach Cuadra
The results from a multi-centric MRI dataset of 172 patients demonstrate that our proposed measures more effectively capture model errors at the lesion and patient scales compared to measures that average voxel-scale uncertainty values.
no code implementations • 8 Nov 2023 • Vatsal Raina, Adian Liusie, Mark Gales
Specifically, we define quality in terms of the incorrectness, plausibility and diversity of the distractor options.
no code implementations • 22 Sep 2023 • Asma Farajidizaji, Vatsal Raina, Mark Gales
We also find greater drops in semantic and lexical similarity between the source and target texts with greater shifts in the readability.
1 code implementation • 12 Sep 2023 • Vyas Raina, Mark Gales
Minimum Bayes' Risk (MBR) decoding can be used to combine system outputs in a manner that encourages better alignment with the final assessment criterion.
no code implementations • 9 Jul 2023 • Rao Ma, Mengjie Qian, Potsawee Manakul, Mark Gales, Kate Knill
In this paper we investigate using ChatGPT, a generative LLM, for ASR error correction.
no code implementations • 3 Jul 2023 • Vatsal Raina, Adian Liusie, Mark Gales
Multiple-choice reading and listening comprehension tests are an important part of language assessment.
1 code implementation • 21 Jun 2023 • Vyas Raina, Mark Gales
Adversarial attack research in natural language processing (NLP) has made significant progress in designing powerful attack methods and defence approaches.
1 code implementation • 8 Jun 2023 • Potsawee Manakul, Yassir Fathullah, Adian Liusie, Vyas Raina, Vatsal Raina, Mark Gales
In this paper, we consider the challenge of summarizing patients' medical progress notes in a limited data setting.
no code implementations • 17 May 2023 • Yassir Fathullah, Guoxuan Xia, Mark Gales
Efficiently and reliably estimating uncertainty is an important objective in deep learning.
no code implementations • 2 May 2023 • Vyas Raina, Mark Gales
In this work, adversarial attacks for NMT systems are explored from an output perception perspective.
no code implementations • 3 Apr 2023 • Tian Huey Teh, Vivian Hu, Devang S Ram Mohan, Zack Hodari, Christopher G. R. Wallis, Tomás Gomez Ibarrondo, Alexandra Torresquintero, James Leoni, Mark Gales, Simon King
Generating expressive speech with rich and varied prosody continues to be a challenge for Text-to-Speech.
1 code implementation • 10 Feb 2023 • Vatsal Raina, Nataliia Molchanova, Mara Graziani, Andrey Malinin, Henning Muller, Meritxell Bach Cuadra, Mark Gales
This work describes a detailed analysis of the recently proposed normalised Dice Similarity Coefficient (nDSC) for binary segmentation tasks as an adaptation of DSC which scales the precision at a fixed recall rate to tackle this bias.
1 code implementation • 30 Jan 2023 • Vyas Raina, Mark Gales
We propose a deep-learning-based detector to identify the adversarially attackable and robust samples in an unseen dataset for an unseen target model.
1 code implementation • 13 Nov 2022 • Adian Liusie, Vatsal Raina, Mark Gales
Two metrics are described: the expected number of options, which measures whether a passage-free system can identify the answer a question using world knowledge; and the contextual mutual information, which measures the importance of context for a given question.
1 code implementation • 9 Nov 2022 • Nataliia Molchanova, Vatsal Raina, Andrey Malinin, Francesco La Rosa, Henning Muller, Mark Gales, Cristina Granziera, Mara Graziani, Meritxell Bach Cuadra
This paper focuses on the uncertainty estimation for white matter lesions (WML) segmentation in magnetic resonance imaging (MRI).
no code implementations • 6 Nov 2022 • Qingyun Dou, Mark Gales
Attention forcing has been introduced to address the mismatch, guiding the model with the generated back-history and reference attention.
no code implementations • 6 Nov 2022 • Qingyun Dou, Mark Gales
A deliberation network consists of multiple standard sequence-to-sequence models, each one conditioned on the initial input and the output of the previous model.
no code implementations • 23 Sep 2022 • Vatsal Raina, Mark Gales
Applying n-gram based approaches is challenging for this form of system as the reference set is unlikely to capture the full range of possible questions and answer options.
no code implementations • 19 Aug 2022 • Vyas Raina, Mark Gales
When considering the application of GEC systems to automated language assessment, the aim of an adversary could be to cheat by making a small change to a grammatically incorrect input sentence that conceals the errors from a GEC system, such that no edits are found and the candidate is unjustly awarded a perfect fluency score.
1 code implementation • NAACL 2022 • Vyas Raina, Mark Gales
Many popular image adversarial detection approaches are able to identify adversarial examples from embedding feature spaces, whilst in the NLP domain existing state of the art detection approaches solely focus on input text features, without consideration of model embedding spaces.
1 code implementation • NeurIPS 2021 • Max Ryabinin, Andrey Malinin, Mark Gales
\emph{Ensemble Distribution Distillation} is an approach that allows a single model to efficiently capture both the predictive performance and uncertainty estimates of an ensemble.
no code implementations • 13 Jan 2021 • Xixin Wu, Mark Gales
It is shown that well calibrated ensemble members will not necessarily yield a well calibrated ensemble prediction, and if the ensemble prediction is well calibrated its performance cannot exceed that of the average performance of the calibrated ensemble members.
no code implementations • 4 Dec 2020 • Potsawee Manakul, Mark Gales
Our approach consists of two steps: (1) Filtering redundant or less informative sentences in the transcription using the attention of a hierarchical model; (2) Applying a state-of-the-art text summarisation system (BART) fine-tuned on the Podcast data using a sequence-level reward function.
no code implementations • 24 Nov 2020 • Yassir Fathullah, Mark Gales, Andrey Malinin
It is, however, more challenging than the standard tasks investigated for distillation as the prediction of any grammatical correction to a word will be highly dependent on both the input sequence and the generated output history for the word.
no code implementations • WS 2020 • Vatsal Raina, Mark Gales, Kate Knill
This paper examines one form of spoken language assessment; whether the response from the candidate is relevant to the prompt provided.
1 code implementation • 20 Jun 2020 • Andrey Malinin, Sergey Chervontsev, Ivan Provilkov, Mark Gales
Prior Networks are a recently developed class of models which yield interpretable measures of uncertainty and have been shown to outperform state-of-the-art ensemble approaches on a range of tasks.
no code implementations • ICLR 2021 • Andrey Malinin, Mark Gales
Uncertainty estimation is important for ensuring safety and robustness of AI systems.
2 code implementations • 25 Oct 2019 • Alexandros Kastanos, Anton Ragni, Mark Gales
This paper examines this limited resource scenario for confidence estimation, a measure commonly used to assess transcription reliability.
Automatic Speech Recognition Automatic Speech Recognition (ASR) +1
1 code implementation • NeurIPS 2019 • Andrey Malinin, Mark Gales
Second, taking advantage of this new training criterion, this paper investigates using Prior Networks to detect adversarial attacks and proposes a generalized form of adversarial training.
1 code implementation • ICLR 2020 • Andrey Malinin, Bruno Mlodozeniec, Mark Gales
The properties of EnD$^2$ are investigated on both an artificial dataset, and on the CIFAR-10, CIFAR-100 and TinyImageNet datasets, where it is shown that EnD$^2$ can approach the classification performance of an ensemble, and outperforms both standard DNNs and Ensemble Distillation on the tasks of misclassification and out-of-distribution input detection.
no code implementations • 6 Dec 2018 • Andrey Malinin, Mark Gales
In this work, Prior Networks are applied to adversarial attack detection using measures of uncertainty in a similar fashion to Monte-Carlo Dropout.
no code implementations • 30 Oct 2018 • Anton Ragni, Qiujia Li, Mark Gales, Yu Wang
These errors are not accounted for by the standard confidence estimation schemes and are hard to rectify in the upstream and downstream processing.
4 code implementations • 30 Oct 2018 • Qiujia Li, Preben Ness, Anton Ragni, Mark Gales
The standard approach to mitigate errors made by an automatic speech recognition system is to use confidence scores associated with each predicted word.
Automatic Speech Recognition Automatic Speech Recognition (ASR) +3
1 code implementation • NeurIPS 2018 • Andrey Malinin, Mark Gales
Experiments on synthetic and MNIST and CIFAR-10 data show that unlike previous non-Bayesian methods PNs are able to distinguish between data and distributional uncertainty.
no code implementations • 1 Feb 2018 • Yu Wang, Xie Chen, Mark Gales, Anton Ragni, Jeremy Wong
As the combination approaches become more complicated the difference between the phonetic and graphemic systems further decreases.
Automatic Speech Recognition Automatic Speech Recognition (ASR) +1
no code implementations • 18 Aug 2017 • Xie Chen, Xunying Liu, Anton Ragni, Yu Wang, Mark Gales
Instead of using a recurrent unit to capture the complete future word contexts, a feedforward unit is used to model a finite number of succeeding, future, words.
no code implementations • ACL 2017 • Andrey Malinin, Anton Ragni, Kate Knill, Mark Gales
On experiments conducted on data from the Business Language Testing Service (BULATS), the proposed approach is found to outperform GPs and DNNs with MCD in uncertainty-based rejection whilst achieving comparable grading performance.