Search Results for author:
Found 50 papers, 16 papers with code
CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization
One challenge in evaluating visual question answering (VQA) models in the cross-dataset adaptation setting is that the distribution shifts are multi-modal, making it difficult to identify if it is the shifts in visual or language features that play a key role.
Connecting Vision and Language with Video Localized Narratives
We propose Video Localized Narratives, a new form of multimodal video annotations connecting vision and language.
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting
Through extensive human evaluation on EditBench, we find that object-masking during training leads to across-the-board improvements in text-image alignment -- such that Imagen Editor is preferred over DALL-E 2 and Stable Diffusion -- and, as a cohort, these models are better at object-rendering than text-rendering, and handle material/color/size attributes better than count/shape attributes.
Improving Robust Generalization by Direct PAC-Bayesian Bound Minimization
Recent research in robust optimization has shown an overfitting-like phenomenon in which models trained against adversarial attacks exhibit higher robustness on the training set compared to the test set.
PaLI: A Jointly-Scaled Multilingual Language-Image Model
PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages.
Ranked #1 on
Image Captioning
on nocaps out-of-domain
MaXM: Towards Multilingual Visual Question Answering
Visual Question Answering (VQA) has been primarily studied through the lens of the English language.
PreSTU: Pre-Training for Scene-Text Understanding
The ability to recognize and reason about text embedded in visual inputs is often lacking in vision-and-language (V&L) models, perhaps because V&L pre-training methods have often failed to include such an ability as their training objective.
Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset
Research in massively multilingual image captioning has been severely hampered by a lack of high-quality evaluation datasets.
All You May Need for VQA are Image Captions
Visual Question Answering (VQA) has benefited from increasingly sophisticated models, but has not enjoyed the same level of engagement in terms of data creation.
End-to-end Dense Video Captioning as Sequence Generation
Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event.
Ranked #2 on
Dense Video Captioning
on ViTT
(CIDEr metric, using extra
training data)
PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks
With the increasing abundance of pretrained models in recent years, the problem of selecting the best pretrained checkpoint for a particular downstream classification task has been gaining increased attention.
COSMic: A Coherence-Aware Generation Metric for Image Descriptions
Developers of text generation models rely on automated evaluation metrics as a stand-in for slow and expensive manual evaluations.
H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
We describe an efficient hierarchical method to compute attention in the Transformer architecture.
Ranked #1 on
Language Modelling
on One Billion Word
(Validation perplexity metric)
Bridging the Gap Between Practice and PAC-Bayes Theory in Few-Shot Meta-Learning
Despite recent advances in its theoretical understanding, there still remains a significant gap in the ability of existing PAC-Bayesian theories on meta-learning to explain performance improvements in the few-shot learning setting, where the number of training examples in the target tasks is severely limited.
2.5D Visual Relationship Detection
To enable progress on this task, we create a new dataset consisting of 220k human-annotated 2. 5D relationships among 512K objects from 11K images.
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pre-training.
Ranked #9 on
Image Captioning
on nocaps-val-out-domain
Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval
Most existing image retrieval systems use text queries as a way for the user to express what they are looking for.
Understanding Guided Image Captioning Performance across Domains
Image captioning models generally lack the capability to take into account user interest, and usually default to global descriptions that try to balance readability, informativeness, and information overload.
Multimodal Pretraining for Dense Video Captioning
First, we construct and release a new dense video captioning dataset, Video Timeline Tags (ViTT), featuring a variety of instructional videos together with time-stamped annotations.
Ranked #1 on
Dense Video Captioning
on YouCook2
(ROUGE-L metric, using extra
training data)
Improving Text Generation Evaluation with Batch Centering and Tempered Word Mover Distance
Recent advances in automatic evaluation metrics for text have shown that deep contextualized word representations, such as those generated by BERT encoders, are helpful for designing metrics that correlate well with human judgements.
TeaForN: Teacher-Forcing with N-grams
Sequence generation models trained with teacher-forcing suffer from issues related to exposure bias and lack of differentiability across timesteps.
Attention that does not Explain Away
Models based on the Transformer architecture have achieved better accuracy than the ones based on competing architectures for a large set of tasks.
Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Models
Training large-scale image captioning (IC) models demands access to a rich and diverse set of training examples, gathered from the wild, often from noisy alt-text data.
Cross-modal Coherence Modeling for Caption Generation
We use coherence relations inspired by computational models of discourse to study the information needs and goals of image captioning.
Multi-Image Summarization: Textual Summary from a Set of Cohesive Images
Multi-sentence summarization is a well studied problem in NLP, while generating image descriptions for a single image is a well studied problem in Computer Vision.
Clue: Cross-modal Coherence Modeling for Caption Generation
We use coherence relations inspired by computational models of discourse to study the information needs and goals of image captioning.
Cross-modal Language Generation using Pivot Stabilization for Web-scale Language Coverage
Cross-modal language generation tasks such as image captioning are directly hurt in their ability to support non-English languages by the trend of data-hungry models combined with the lack of non-English annotations.
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube
Pretraining from unlabelled web videos has quickly become the de-facto means of achieving high performance on many video understanding tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Connecting Vision and Language with Localized Narratives
We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing.
Ranked #2 on
Image Captioning
on Localized Narratives
Reinforcing an Image Caption Generator Using Off-Line Human Feedback
Human ratings are currently the most accurate way to assess the quality of an image captioning model, yet most often the only used outcome of an expensive human rating evaluation is a few overall statistics over the evaluation dataset.
A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions
Instructional videos get high-traffic on video sharing platforms, and prior work suggests that providing time-stamped, subtask annotations (e. g., "heat the oil in the pan") improves user experiences.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks.
Ranked #1 on
Natural Language Inference
on QNLI
Multi-stage Pretraining for Abstractive Summarization
Neural models for abstractive summarization tend to achieve the best performance in the presence of highly specialized, summarization specific modeling add-ons such as pointer-generator, coverage-modeling, and inferencetime heuristics.
Quality Estimation for Image Captions Based on Large-scale Human Evaluations
Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild.
Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering
Object detection plays an important role in current solutions to vision and language tasks like image captioning and visual question answering.
Ranked #4 on
Visual Question Answering
on VizWiz 2018
Informative Image Captioning with External Sources of Information
An image caption should fluently present the essential information in a given image, including informative, fine-grained entity mentions and the manner in which these entities interact.
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
We present a new dataset of image caption annotations, Conceptual Captions, which contains an order of magnitude more images than the MS-COCO dataset (Lin et al., 2014) and represents a wider variety of both images and image caption styles.
SHAPED: Shared-Private Encoder-Decoder for Text Style Adaptation
Supervised training of abstractive language generation models results in learning conditional probabilities over language sequences based on the supervised training signal.
Cold-Start Reinforcement Learning with Softmax Policy Gradient
Policy-gradient approaches to reinforcement learning have two common and undesirable overhead procedures, namely warm-start training and sample variance reduction.
Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
We introduce a new multi-modal task for computer systems, posed as a combined vision-language comprehension challenge: identifying the most suitable text describing a scene, given several similar options.
Multilingual Word Embeddings using Multigraphs
We present a family of neural-network--inspired models for computing continuous word representations, specifically designed to exploit both monolingual and multilingual text.
Building Large Machine Reading-Comprehension Datasets using Paragraph Vectors
We present a dual contribution to the task of machine reading-comprehension: a technique for creating large-sized machine-comprehension (MC) datasets using paragraph-vector models; and a novel, hybrid neural-network architecture that combines the representation power of recurrent neural networks with the discriminative power of fully-connected multi-layered networks.
Morpho-syntactic Lexicon Generation Using Graph-based Semi-supervised Learning
Morpho-syntactic lexicons provide information about the morphological and syntactic roles of words in a language.
