Search Results for author:
Found 35 papers, 21 papers with code
Improving Pre-trained Vision-and-Language Embeddings for Phrase Grounding
Phrase grounding aims to map textual phrases to their associated image regions, which can be a prerequisite for multimodal reasoning and can benefit tasks requiring identifying objects based on language.
MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models
In this paper, we introduce a multimodal retrieval-augmented generation benchmark, MRAG-Bench, in which we systematically identify and categorize scenarios where visually augmented knowledge is better than textual knowledge, for instance, more images from varying viewpoints.
Unlocking Exocentric Video-Language Data for Egocentric Video Representation Learning
We present EMBED (Egocentric Models Built with Exocentric Data), a method designed to transform exocentric video-language data for egocentric video representation learning.
Re-ReST: Reflection-Reinforced Self-Training for Language Agents
Finetuning language agents with reasoning-action trajectories is effective, but obtaining these trajectories from human annotations or stronger models is costly and sometimes impractical.
Matryoshka Query Transformer for Large Vision-Language Models
This raises the question: can we achieve flexibility in the number of visual tokens to suit different tasks and computational resources?
Medical Vision-Language Pre-Training for Brain Abnormalities
Vision-language models have become increasingly powerful for tasks that require an understanding of both visual and linguistic elements, bridging the gap between these modalities.
VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
Our work also highlights the critical balance between faithfulness and coverage of model outputs, and encourages future works to address hallucinations in LVLMs while keeping their outputs informative.
ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos
Multimodal counterfactual reasoning is a vital yet challenging ability for AI systems.
Gender Biases in Automatic Evaluation Metrics for Image Captioning
Model-based evaluation metrics (e. g., CLIPScore and GPTScore) have demonstrated decent correlations with human judgments in various language generation tasks.
Masked Path Modeling for Vision-and-Language Navigation
In this paper, we introduce a masked path modeling (MPM) objective, which pretrains an agent using self-collected data for downstream navigation tasks.
Generalized Decoding for Pixel, Image, and Language
We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly.
Ranked #4 on
Instance Segmentation
on ADE20K val
(using extra training data)
Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Vision-language (VL) pre-training has recently received considerable attention.
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Dev
FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation
The speaker-follower models have proven to be effective in vision-and-language navigation, where a speaker model is used to synthesize new instructions to augment the training data for a follower navigation model.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
In this paper, we instead focus on better utilizing the \textit{implicit knowledge} stored in pre-trained language models.
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks.
Ranked #20 on
Cross-Modal Retrieval
on COCO 2014
(using extra training data)
RefSum: Refactoring Neural Summarization
Although some recent works show potential complementarity among different state-of-the-art systems, few works try to investigate this problem in text summarization.
ExplainaBoard: An Explainable Leaderboard for NLP
In this paper, we present a new conceptualization and implementation of NLP evaluation: the ExplainaBoard, which in addition to inheriting the functionality of the standard leaderboard, also allows researchers to (i) diagnose strengths and weaknesses of a single system (e. g.~what is the best-performing system bad at?)
Word Alignment by Fine-tuning Embeddings on Parallel Corpora
In addition, we demonstrate that we are able to train multilingual word aligners that can obtain robust performance on different language pairs.
GSum: A General Framework for Guided Neural Abstractive Summarization
Neural abstractive summarization models are flexible and can produce coherent summaries, but they are sometimes unfaithful and can be difficult to control.
CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems
In this paper, we perform an in-depth analysis of characteristics of different datasets and investigate the performance of different summarization models under a cross-dataset setting, in which a summarizer trained on one corpus will be evaluated on a range of out-of-domain corpora.
TICO-19: the Translation Initiative for Covid-19
Further, the team is converting the test and development data into translation memories (TMXs) that can be used by localizers from and to any of the languages.
A Deep Reinforced Model for Zero-Shot Cross-Lingual Summarization with Bilingual Semantic Similarity Rewards
The model uses reinforcement learning to directly optimize a bilingual semantic similarity metric between the summaries generated in a target language and gold summaries in a source language.
Dynamic Data Selection and Weighting for Iterative Back-Translation
Back-translation has proven to be an effective method to utilize monolingual data in neural machine translation (NMT), and iteratively conducting back-translation can further improve the model performance.
Domain Differential Adaptation for Neural Machine Translation
We then use these learned domain differentials to adapt models for the target task accordingly.
Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks
Learning general representations of text is a fundamental problem for many natural language understanding (NLU) tasks.
Unsupervised Domain Adaptation for Neural Machine Translation with Domain-Aware Feature Embeddings
The recent success of neural machine translation models relies on the availability of high quality, in-domain data.
Information Aggregation for Multi-Head Attention with Routing-by-Agreement
Multi-head attention is appealing for its ability to jointly extract different types of information from multiple representation subspaces.
compare-mt: A Tool for Holistic Comparison of Language Generation Systems
In this paper, we describe compare-mt, a tool for holistic analysis and comparison of the results of systems for language generation tasks such as machine translation.
Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement
With the promising progress of deep neural networks, layer aggregation has been used to fuse information across layers in various fields, such as computer vision and machine translation.
Exploiting Deep Representations for Neural Machine Translation
Advanced neural machine translation (NMT) models generally implement encoder and decoder as multiple layers, which allows systems to model complex functions and capture complicated linguistic structures.
Unsupervised Bilingual Lexicon Induction via Latent Variable Models
Bilingual lexicon extraction has been studied for decades and most previous methods have relied on parallel corpora or bilingual dictionaries.
SkipNet: Learning Dynamic Routing in Convolutional Networks
While deeper convolutional networks are needed to achieve maximum accuracy in visual perception tasks, for many inputs shallower networks are sufficient.
Metric Learning-based Generative Adversarial Network
Generative Adversarial Networks (GANs), as a framework for estimating generative models via an adversarial process, have attracted huge attention and have proven to be powerful in a variety of tasks.
Dynamic Oracle for Neural Machine Translation in Decoding Phase
However, there are certain limitations in Scheduled Sampling and we propose two dynamic oracle-based methods to improve it.
Capturing User and Product Information for Document Level Sentiment Analysis with Deep Memory Network
Document-level sentiment classification is a fundamental problem which aims to predict a user{'}s overall sentiment about a product in a document.
Ranked #8 on
Sentiment Analysis
on User and product information
