Search Results for author:
Found 34 papers, 14 papers with code
Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering
The field of visual question answering (VQA) has recently seen a surge in research focused on providing explanations for predicted answers.
Exploring the Long-Term Generalization of Counting Behavior in RNNs
Despite this and some positive empirical results for LSTMs on Dyck-1 languages, our experimental results show that LSTMs fail to learn correct counting behavior for sequences that are significantly longer than in the training data.
Belief Revision based Caption Re-ranker with Visual Semantic Information
In this work, we focus on improving the captions generated by image-caption generation systems.
Evaluation of Fake News Detection with Knowledge-Enhanced Language Models
The predominant state-of-the-art approaches are based on fine-tuning PLMs on labelled fake news datasets.
Numerical reasoning in machine reading comprehension tasks: are we there yet?
The current standings of these models in the DROP leaderboard, over standard metrics, suggest that the models have achieved near-human performance.
BERTGEN: Multi-task Generation through BERT
We present BERTGEN, a novel generative, decoder-only model which extends BERT by fusing multimodal and multilingual pretrained models VL-BERT and M-BERT, respectively.
A call for better unit testing for invariant risk minimisation
In this paper we present a controlled study on the linearized IRM framework (IRMv1) introduced in Arjovsky et al. (2020).
Discrete Reasoning Templates for Natural Language Understanding
Reasoning about information from multiple parts of a passage to derive an answer is an open challenge for reading-comprehension models.
MultiSubs: A Large-scale Multimodal and Multilingual Dataset
The dataset will benefit research on visual grounding of words especially in the context of free-form sentences, and can be obtained from https://doi. org/10. 5281/zenodo. 5034604 under a Creative Commons licence.
Ranked #1 on
Multimodal Text Prediction
on MultiSubs
Multimodal Lexical Translation
Multimodal Text Prediction
+1
Exploiting Multimodal Reinforcement Learning for Simultaneous Machine Translation
This paper addresses the problem of simultaneous machine translation (SiMT) by exploring two main concepts: (a) adaptive policies to learn a good trade-off between high translation quality and low latency; and (b) visual information to support this process by providing additional (visual) contextual information which may be available before the textual input is produced.
Cross-lingual Visual Pre-training for Multimodal Machine Translation
Pre-trained language models have been shown to improve performance in many natural language tasks substantially.
MSVD-Turkish: A Comprehensive Multimodal Dataset for Integrated Vision and Language Research in Turkish
We hope that the MSVD-Turkish dataset and the results reported in this work will lead to better video captioning and multimodal machine translation models for Turkish and other morphology rich and agglutinative languages.
Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale
Automatic evaluation of language generation systems is a well-studied problem in Natural Language Processing.
Simultaneous Machine Translation with Visual Context
Simultaneous machine translation (SiMT) aims to translate a continuous input text stream into another language with the lowest latency and highest quality possible.
A Tweet-based Dataset for Company-Level Stock Return Prediction
Public opinion influences events, especially related to stock market movement, in which a subtle hint can influence the local outcome of the market.
Deep Copycat Networks for Text-to-Text Generation
Most text-to-text generation tasks, for example text summarisation and text simplification, require copying words from the input to the output.
Imperial College London Submission to VATEX Video Captioning Task
This paper describes the Imperial College London team's submission to the 2019' VATEX video captioning challenge, where we first explore two sequence-to-sequence models, namely a recurrent (GRU) model and a transformer model, which generate captions from the I3D action features.
On Model Stability as a Function of Random Seed
In this paper, we focus on quantifying model stability as a function of random seed by investigating the effects of the induced randomness on model performance and the robustness of the model in general.
Probing Representations Learned by Multimodal Recurrent and Transformer Models
In this paper, we present a meta-study assessing the representational quality of models where the training signal is obtained from different modalities, in particular, language modeling, image features prediction, and both textual and multimodal machine translation.
Predicting Actions to Help Predict Translations
The question we ask ourselves is whether visual features can support the translation process, in particular, given that this is a dataset extracted from videos, we focus on the translation of actions, which we believe are poorly captured in current static image-text datasets currently used for multimodal translation.
WMDO: Fluency-based Word Mover's Distance for Machine Translation Evaluation
We propose WMDO, a metric based on distance between distributions in the semantic vector space.
VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions
It estimates the faithfulness of a generated caption with respect to the content of the actual image, based on the semantic similarity between labels of objects depicted in images and words in the description.
Distilling Translations with Visual Awareness
Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient.
Ranked #2 on
Multimodal Machine Translation
on Multi30K
(Meteor (EN-FR) metric)
Model Explanations under Calibration
Explaining and interpreting the decisions of recommender systems are becoming extremely relevant both, for improving predictive performance, and providing valid explanations to users.
Grounded Word Sense Translation
Recent work on visually grounded language learning has focused on broader applications of grounded representations, such as visual question answering and multimodal machine translation.
Grounded language learning
Multimodal Machine Translation
+3
Probing the Need for Visual Context in Multimodal Machine Translation
Current work on multimodal machine translation (MMT) has suggested that the visual modality is either unnecessary or only marginally beneficial.
Predicting Language Recovery after Stroke with Convolutional Networks on Stitched MRI
One third of stroke survivors have language difficulties.
Learning from Multiview Correlations in Open-Domain Videos
This shows it is possible to learn reliable representations across disparate, unaligned and noisy modalities, and encourages using the proposed approach on larger datasets.
End-to-end Image Captioning Exploits Multimodal Distributional Similarity
We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn `distributional similarity' in a multimodal feature space by mapping a test image to similar training images in this space and generating a caption from the same space.
Defoiling Foiled Image Captions
We address the task of detecting foiled image captions, i. e. identifying whether a caption contains a word that has been deliberately replaced by a semantically similar word, thus rendering it inaccurate with respect to the image being described.
Object Counts! Bringing Explicit Detections Back into Image Captioning
The use of explicit object detectors as an intermediate step to image captioning - which used to constitute an essential stage in early work - is often bypassed in the currently dominant end-to-end approaches, where the language model is conditioned directly on a mid-level image embedding.
What is image captioning made of?
We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn ‘distributional similarity’ in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space.
