Search Results for author:
Found 21 papers, 5 papers with code
MultiSubs: A Large-scale Multimodal and Multilingual Dataset
The dataset will benefit research on visual grounding of words especially in the context of free-form sentences, and can be obtained from https://doi. org/10. 5281/zenodo. 5034604 under a Creative Commons licence.
Ranked #1 on
Multimodal Text Prediction
on MultiSubs
Multimodal Lexical Translation
Multimodal Text Prediction
+2
Transformer-based Cascaded Multimodal Speech Translation
Upon conducting extensive experiments, we found that (i) the explored visual integration schemes often harm the translation performance for the transformer and additive deliberation, but considerably improve the cascade deliberation; (ii) the transformer and cascade deliberation integrate the visual modality better than the additive deliberation, as shown by the incongruence analysis.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Imperial College London Submission to VATEX Video Captioning Task
This paper describes the Imperial College London team's submission to the 2019' VATEX video captioning challenge, where we first explore two sequence-to-sequence models, namely a recurrent (GRU) model and a transformer model, which generate captions from the I3D action features.
Phrase Localization Without Paired Training Examples
Localizing phrases in images is an important part of image understanding and can be useful in many applications that require mappings between textual and visual information.
Predicting Actions to Help Predict Translations
The question we ask ourselves is whether visual features can support the translation process, in particular, given that this is a dataset extracted from videos, we focus on the translation of actions, which we believe are poorly captured in current static image-text datasets currently used for multimodal translation.
VIFIDEL: Evaluating the Visual Fidelity of Image Descriptions
It estimates the faithfulness of a generated caption with respect to the content of the actual image, based on the semantic similarity between labels of objects depicted in images and words in the description.
End-to-end Image Captioning Exploits Distributional Similarity in Multimodal Space
We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn {`}distributional similarity{'} in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space.
End-to-end Image Captioning Exploits Multimodal Distributional Similarity
We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn `distributional similarity' in a multimodal feature space by mapping a test image to similar training images in this space and generating a caption from the same space.
Defoiling Foiled Image Captions
We address the task of detecting foiled image captions, i. e. identifying whether a caption contains a word that has been deliberately replaced by a semantically similar word, thus rendering it inaccurate with respect to the image being described.
Object Counts! Bringing Explicit Detections Back into Image Captioning
The use of explicit object detectors as an intermediate step to image captioning - which used to constitute an essential stage in early work - is often bypassed in the currently dominant end-to-end approaches, where the language model is conditioned directly on a mid-level image embedding.
Visual and Semantic Knowledge Transfer for Large Scale Semi-supervised Object Detection
This is done by modeling the differences between the two on categories with both image-level and bounding box annotations, and transferring this information to convert classifiers to detectors for categories without bounding box annotations.
What is image captioning made of?
We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn ‘distributional similarity’ in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space.
Large Scale Semi-Supervised Object Detection Using Visual and Semantic Knowledge Transfer
This is done by modeling the differences between the two on categories with both image-level and bounding box annotations, and transferring this information to convert classifiers to detectors for categories without bounding box annotations.
Cross-validating Image Description Datasets and Evaluation Metrics
The task of automatically generating sentential descriptions of image content has become increasingly popular in recent years, resulting in the development of large-scale image description datasets and the proposal of various metrics for evaluating image description generation systems.
