Search Results for author:
Found 269 papers, 177 papers with code
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder.
Ranked #1 on
Visual Question Answering (VQA)
on VizWiz 2018
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made.
Ranked #1 on
Few-Shot Text Classification
on RAFT
Unifying Vision, Text, and Layout for Universal Document Processing
UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation.
Ranked #5 on
Visual Question Answering (VQA)
on InfographicVQA
(using extra training data)
Any-to-Any Generation via Composable Diffusion
We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities.
Ranked #7 on
Audio Generation
on AudioCaps
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding
Specifically, the task involves multi-hop questions that require reasoning over image-caption pairs to identify the grounded visual object being referred to and then predicting a span from the news body text to answer the question.
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
In this work, we investigate the visual reasoning capabilities and social biases of different text-to-image models, covering both multimodal transformer language models and diffusion models.
Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle.
Ranked #24 on
Visual Question Answering (VQA)
on MSRVTT-QA
(using extra training data)
Revealing Single Frame Bias for Video-and-Language Learning
Training an effective video-and-language model intuitively requires multiple frames as model inputs.
Ranked #5 on
Video Retrieval
on SSv2-template retrieval
(using extra training data)
Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension
In this work, we present several effective adversaries and automated data augmentation policy search methods with the goal of making reading comprehension models more robust to adversarial evaluation, but also improving generalization to the source domain as well as new domains and languages.
Robustness Gym: Unifying the NLP Evaluation Landscape
Despite impressive performance on standard benchmarks, deep neural networks are often brittle when deployed in real-world systems.
Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
Inspired by how humans summarize long documents, we propose an accurate and fast summarization model that first selects salient sentences and then rewrites them abstractively (i. e., compresses and paraphrases) to generate a concise overall summary.
Ranked #7 on
Text Summarization
on CNN / Daily Mail (Anonymized)
PaperRobot: Incremental Draft Generation of Scientific Ideas
We present a PaperRobot who performs as an automatic research assistant by (1) conducting deep understanding of a large collection of human-written papers in a target domain and constructing comprehensive background knowledge graphs (KGs); (2) creating new ideas by predicting links from the background KGs, by combining graph attention and contextual text attention; (3) incrementally writing some key elements of a new paper based on memory-attention networks: from the input title along with predicted related entities to generate a paper abstract, from the abstract to generate conclusion and future work, and finally from future work to generate a title for a follow-on paper.
Adversarial NLI: A New Benchmark for Natural Language Understanding
We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure.
How Much Can CLIP Benefit Vision-and-Language Tasks?
Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world.
Ranked #4 on
Vision and Language Navigation
on RxR
(using extra training data)
Unifying Vision-and-Language Tasks via Text Generation
On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models.
Ranked #3 on
Image Captioning
on nocaps val
StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation
Hence, we first propose the task of story continuation, where the generated visual story is conditioned on a source image, allowing for better generalization to narratives with new characters.
Ranked #2 on
Story Continuation
on FlintstonesSV
MAttNet: Modular Attention Network for Referring Expression Comprehension
In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression.
Generalized Referring Expression Segmentation
Referring Expression
+1
Multi-Target Embodied Question Answering
To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module.
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w. r. t.
Ranked #12 on
Highlight Detection
on QVHighlights
Detecting Moments and Highlights in Videos via Natural Language Queries
Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w. r. t.
Ranked #6 on
Video Grounding
on QVHighlights
End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures
We present a novel end-to-end neural model to extract entities and relations between them.
Ranked #1 on
Relation Extraction
on ACE 2005
(Sentence Encoder metric)
Fine-grained Image Captioning with CLIP Reward
Toward more descriptive and distinctive caption generation, we propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function.
Ranked #26 on
Image Captioning
on COCO Captions
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
LST saves 69% of the memory costs to fine-tune the whole network, while other methods only save 26% of that in similar parameter usages (hence, 2. 7x more memory savings).
VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Our results demonstrate that training the adapter with the weight-sharing technique (4. 18% of total parameters for image-text tasks and 3. 39% for video-text tasks) can match the performance of fine-tuning the entire model.
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
We find that the main reason hindering this exploration is the large divergence in magnitude and distributions between the visually-grounded language datasets and pure-language corpora.
MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph.
Ranked #5 on
Video Captioning
on ActivityNet Captions
Self-Chained Image-Language Model for Video Localization and Question Answering
SeViLA framework consists of two modules: Localizer and Answerer, where both are parameter-efficiently fine-tuned from BLIP-2.
Ranked #3 on
Zero-Shot Video Question Answer
on IntentQA
(using extra training data)
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
Large Language Models (LLMs) still struggle with natural language reasoning tasks.
TVQA: Localized, Compositional Video Question Answering
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks.
Ranked #4 on
Video Question Answering
on SUTD-TrafficQA
Improving and Simplifying Pattern Exploiting Training
However, PET uses task-specific unlabeled data.
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval
The queries are also labeled with query types that indicate whether each of them is more related to video or subtitle or both, allowing for in-depth analysis of the dataset and the methods that built on top of it.
Ranked #2 on
Video Retrieval
on TVR
Scaling Data Generation in Vision-and-Language Navigation
Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents.
GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models
Providing natural language instructions in prompts is a useful new paradigm for improving task performance of large language models in a zero-shot setting.
Commonsense for Generative Multi-Hop Question Answering Tasks
We instead focus on a more challenging multi-hop generative task (NarrativeQA), which requires the model to reason, gather, and synthesize disjoint pieces of information within the context to generate an answer.
Ranked #6 on
Question Answering
on NarrativeQA
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
Next, we apply semi-supervised learning (via back-translation) on these dropped-out environments to generate new paths and instructions.
Ranked #1 on
Vision-Language Navigation
on Room2Room
TVQA+: Spatio-Temporal Grounding for Video Question Answering
We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos.
Ranked #6 on
Video Question Answering
on TVQA
TVLT: Textless Vision-Language Transformer
In this work, we present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and-language representation learning with minimal modality-specific design, and do not use text-specific modules such as tokenization or automatic speech recognition (ASR).
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction.
TIES-Merging: Resolving Interference When Merging Models
To address this, we propose our method, TRIM, ELECT SIGN & MERGE (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign.
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
Large vision-language models (LVLMs) have shown remarkable abilities in understanding visual information with human languages.
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
Second, since the traditional evaluation metrics (e. g., BLEU) often fall short in evaluating the quality of generated questions, we propose a QA-based evaluation method which measures the QG model's ability to mimic human annotators in generating QA training data.
VindLU: A Recipe for Effective Video-and-Language Pretraining
Furthermore, our model also obtains state-of-the-art video question-answering results on ActivityNet-QA, MSRVTT-QA, MSRVTT-MC and TVQA.
Ranked #2 on
Video Retrieval
on Condensed Movies
(using extra training data)
FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
With the availability of the fast influence functions, we demonstrate their usefulness in four applications.
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task.
Hierarchical Video-Moment Retrieval and Step-Captioning
Our hierarchical benchmark consists of video retrieval, moment retrieval, and two novel moment segmentation and step captioning tasks.
Vision Transformers are Parameter-Efficient Audio-Visual Learners
To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT.
Ranked #4 on
Audio-visual Question Answering
on MUSIC-AVQA
Combining Fact Extraction and Verification with Neural Semantic Matching Networks
The increasing concern with misinformation has stimulated research efforts on automatic fact checking.
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e. g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations.
Ranked #10 on
Action Recognition
on Diving-48
Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization
We first show that teacher LLMs can indeed intervene on student reasoning to improve their performance.
Expressing Visual Relationships via Language
To push forward the research in this direction, we first introduce a new language-guided image editing dataset that contains a large number of real image pairs with corresponding editing instructions.
Shortcut-Stacked Sentence Encoders for Multi-Domain Inference
We present a simple sequential sentence encoder for multi-domain natural language inference.
Ranked #62 on
Natural Language Inference
on SNLI
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale
In this work, we give general guidelines on system design for MRS by proposing a simple yet effective pipeline system with special consideration on hierarchical semantic retrieval at both paragraph and sentence level, and their potential effects on the downstream task.
Ranked #44 on
Question Answering
on HotpotQA
A Simple LLM Framework for Long-Range Video Question-Answering
Furthermore, we show that a specialized prompt that asks the LLM first to summarize the noisy short-term visual captions and then answer a given input question leads to a significant LVQA performance boost.
Ranked #1 on
Zero-Shot Video Question Answer
on NExT-GQA
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
We train a multi-modal teacher model on a video-text dataset, and then transfer its knowledge to a student language model with a text dataset.
Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
This finding raises questions about how past work relies on Causal Tracing to select which model layers to edit.
What is More Likely to Happen Next? Video-and-Language Future Event Prediction
Given a video with aligned dialogue, people can often infer what is more likely to happen next.
EmailSum: Abstractive Email Thread Summarization
Furthermore, we find that widely used automatic evaluation metrics (ROUGE, BERTScore) are weakly correlated with human judgments on this email thread summarization task.
Ranked #1 on
Email Thread Summarization
on EmailSum (short)
Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks, however, they have issues like (a) High Memory Usage, due to duplication of the network layers into multiple copies as experts; and (b) Redundancy in Experts, as common learning-based routing policies suffer from representational collapse.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Despite recent improvements in abstractive summarization, most current approaches generate summaries that are not factually consistent with the source document, severely restricting their trust and usage in real-world applications.
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
Through two kinds of simulation tests involving text and tabular data, we evaluate five explanations methods: (1) LIME, (2) Anchor, (3) Decision Boundary, (4) a Prototype model, and (5) a Composite approach that combines explanations from each method.
Continual and Multi-Task Architecture Search
Architecture search is the process of automatically learning the neural model or cell structure that best suits the given task.
The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
In this paper, we present the surprising conclusion that current language models often generalize relatively well from easy to hard data, even performing as well as "oracle" models trained on hard data.
GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
As Large Language Models (LLMs) are integrated into critical real-world applications, their strategic and logical reasoning abilities are increasingly crucial.
Unified Coarse-to-Fine Alignment for Video-Text Retrieval
Specifically, our model captures the cross-modal similarity information at different granularity levels.
Ranked #11 on
Video Retrieval
on MSR-VTT
Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts
In light of this, we reintroduce this dense design into the generalist models, to facilitate the development of generalist models with high segmentation quality.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present FactPEGASUS, an abstractive summarization model that addresses the problem of factuality during pre-training and fine-tuning: (1) We augment the sentence selection strategy of PEGASUS's (Zhang et al., 2020) pre-training objective to create pseudo-summaries that are both important and factual; (2) We introduce three complementary components for fine-tuning.
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA
Moreover, our model is also comprised of dual-level attention (word/object and frame level), multi-head self/cross-integration for different sources (video and dense captions), and gates which pass more relevant information to the classifier.
Punny Captions: Witty Wordplay in Image Descriptions
Wit is a form of rich interaction that is often grounded in a specific situation (e. g., a comment in response to an event).
A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
The speaker generates referring expressions, the listener comprehends referring expressions, and the reinforcer introduces a reward function to guide sampling of more discriminative expressions.
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
We present Perceiver-VL, a vision-and-language framework that efficiently handles high-dimensional multimodal inputs such as long videos and text.
An Empirical Study of Multimodal Model Merging
In this paper, we expand on this concept to a multimodal setup by merging transformers trained on different modalities.
EnvEdit: Environment Editing for Vision-and-Language Navigation
Training on these edit-augmented environments prevents the agent from overfitting to existing environments and helps generalize better to new, unseen environments.
Ranked #2 on
Vision and Language Navigation
on RxR
(using extra training data)
ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound
We introduce an audiovisual method for long-range text-to-video retrieval.
What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
Analysis reveals that: (1) high human disagreement exists in a noticeable amount of examples in these datasets; (2) the state-of-the-art models lack the ability to recover the distribution over human labels; (3) models achieve near-perfect accuracy on the subset of data with a high level of human agreement, whereas they can barely beat a random guess on the data with low levels of human agreement, which compose most of the common errors made by state-of-the-art models on the evaluation sets.
Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning
We introduce distributed NLI, a new NLU task with a goal to predict the distribution of human judgements for natural language inference.
Improving Generation and Evaluation of Visual Stories via Semantic Consistency
Therefore, we also provide an exploration of evaluation metrics for the model, focused on aspects of the generated frames such as the presence/quality of generated characters, the relevance to captions, and the diversity of the generated images.
Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks
Experimentally, we show that even state-of-the-art model editing methods such as ROME struggle to truly delete factual information from models like GPT-J, as our whitebox and blackbox attacks can recover "deleted" information from an edited model 38% of the time.
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs
In this paper, we discuss approaches to detecting when models have beliefs about the world, and we improve on methods for updating model beliefs to be more truthful, with a focus on methods based on learned optimizers or hypernetworks.
MLP Architectures for Vision-and-Language Modeling: An Empirical Study
Based on this, we ask an even bolder question: can we have an all-MLP architecture for VL modeling, where both VL fusion and the vision encoder are replaced with MLPs?
Paxion: Patching Action Knowledge in Video-Language Foundation Models
Action knowledge involves the understanding of textual, visual, and temporal aspects of actions.
Ranked #19 on
Video Question Answering
on NExT-QA
(using extra training data)
MTVR: Multilingual Moment Retrieval in Videos
We introduce mTVR, a large-scale multilingual video moment retrieval dataset, containing 218K English and Chinese queries from 21. 8K TV show video clips.
Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization
Prior work in this domain has shown that there is ample room for improvement in the generated image sequence in terms of visual quality, consistency and relevance.
Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization
Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image sequence (or visual story).
Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters
We also show improvements in a transfer-only setup on the DUC-2004 dataset.
Explore, Propose, and Assemble: An Interpretable Model for Multi-Hop Reading Comprehension
Multi-hop reading comprehension requires the model to explore and connect relevant information from multiple sentences/documents in order to answer the question about the context.
Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline
Aligning sentences in a reference summary with their counterparts in source documents was shown as a useful auxiliary summarization task, notably for generating training data for salience detection.
Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees
We demonstrate that SP-Search effectively represents the generative process behind human summaries using modules that are typically faithful to their intended behavior.
Polite Dialogue Generation Without Parallel Data
We present three weakly-supervised models that can generate diverse polite (or rude) dialogue responses without parallel data.
Evaluating the Factual Consistency of Large Language Models Through News Summarization
To generate summaries that are factually inconsistent, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent.
MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models
Experiments on seven widely-used commonsense and math reasoning benchmarks show that MAGDi improves the reasoning capabilities of smaller models, outperforming several methods that distill from a single teacher and multiple teachers.
ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness
Multi-step reasoning ability is fundamental to many natural language tasks, yet it is unclear what constitutes a good reasoning chain and how to evaluate them.
RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios
We introduce RESIN-11, a new schema-guided event extraction&prediction framework that can be applied to a large variety of newsworthy scenarios.
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
In this paper, we propose LayoutBench, a diagnostic benchmark for layout-guided image generation that examines four categories of spatial control skills: number, position, size, and shape.
Ranked #1 on
Layout-to-Image Generation
on LayoutBench
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
Despite the efficiency of PEFT methods, the size of expert models can make it onerous to retrieve expert models per query over high-latency networks like the Internet or serve multiple experts on a single GPU.
Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning
Multi-hop QA requires a model to connect multiple pieces of evidence scattered in a long context to answer the question.
D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning
There are two dominant approaches: (1) geometry-based data selection for maximizing data diversity in the coreset, and (2) functions that assign difficulty scores to samples based on training dynamics.
Game-Based Video-Context Dialogue
Current dialogue systems focus more on textual and speech context knowledge and are usually based on two speakers.
PRover: Proof Generation for Interpretable Reasoning over Rules
First, PROVER generates proofs with an accuracy of 87%, while retaining or improving performance on the QA task, compared to RuleTakers (up to 6% improvement on zero-shot evaluation).
Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?
We provide code for the experiments in this paper at https://github. com/peterbhase/LAS-NL-Explanations
ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization
To help save this endangered language, we introduce ChrEn, a Cherokee-English parallel dataset, to facilitate machine translation research between Cherokee and English.
Cultural Vocal Bursts Intensity Prediction
Language Modelling
+5
ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback
The quantitative evaluation demonstrates that our backbone translation models achieve state-of-the-art translation performance and our quality estimation well correlates with both BLEU and human judgment.
CREMA: Multimodal Compositional Video Reasoning via Efficient Modular Adaptation and Fusion
Furthermore, we propose a fusion module designed to compress multimodal queries, maintaining computational efficiency in the LLM while combining additional modalities.
Ranked #1 on
Question Answering
on SQA3D
Diagnosing the Environment Bias in Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations.
The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
In this paper, we study several under-explored dimensions of FI explanations, providing conceptual and empirical improvements for this form of explanation.
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization
Second, to alleviate the temporal misalignment issue, our method incorporates an entropy minimization-based constrained attention loss, to encourage the model to automatically focus on the correct caption from a pool of candidate ASR captions.
Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
An increasing number of vision-language tasks can be handled with little to no training, i. e., in a zero and few-shot manner, by marrying large language models (LLMs) to vision encoders, resulting in large vision-language models (LVLMs).
ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
By analyzing this model, we investigate which words in the question are indicative of the modality.
MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies
Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Q relative to the data distribution P -- that is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood estimation (MLE).
Merging by Matching Models in Task Parameter Subspaces
Model merging aims to cheaply combine individual task-specific models into a single multitask model.
Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences
However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated.
What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment
We propose an end-to-end, domain-independent neural encoder-aligner-decoder model for selective generation, i. e., the joint task of content selection and surface realization.
Adversarial Over-Sensitivity and Over-Stability Strategies for Dialogue Models
We present two categories of model-agnostic adversarial strategies that reveal the weaknesses of several generative, task-oriented dialogue models: Should-Not-Change strategies that evaluate over-sensitivity to small and semantics-preserving edits, as well as Should-Change strategies that test if a model is over-stable against subtle yet semantics-changing modifications.
Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization
Though extractive summarization is less prone to the common unfaithfulness issues of abstractive summaries, does that mean extractive is equal to faithful?
Multimodal Representation Learning by Alternating Unimodal Adaptation
This optimization process is controlled by a gradient modification mechanism to prevent the shared head from losing previously acquired information.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
In order to carefully control important properties of the data and explanations, we introduce a synthetic dataset for experiments, and we also make use of three existing datasets with explanations: e-SNLI, TACRED, and SemEval.
CoSIm: Commonsense Reasoning for Counterfactual Scene Imagination
Next, a counterfactual imagined scene change (in textual form) is applied, and the model has to predict the new response to the initial question based on this scene change.
Faithfulness-Aware Decoding Strategies for Abstractive Summarization
We present a systematic study of the effect of generation techniques such as beam search and nucleus sampling on faithfulness in abstractive summarization.
ReGAL: Refactoring Programs to Discover Generalizable Abstractions
While large language models (LLMs) are increasingly being used for program synthesis, they lack the global view needed to develop useful abstractions; they generally predict programs one at a time, often repeating the same functionality.
Data Augmentation for Abstractive Query-Focused Multi-Document Summarization
The progress in Query-focused Multi-Document Summarization (QMDS) has been limited by the lack of sufficient largescale high-quality training datasets.
Parsing Speech: A Neural Approach to Integrating Lexical and Acoustic-Prosodic Information
In conversational speech, the acoustic signal provides cues that help listeners disambiguate difficult parses.
Finding a Balanced Degree of Automation for Summary Evaluation
In this work, we propose flexible semiautomatic to automatic summary evaluation metrics, following the Pyramid human evaluation method.
Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization
On several syntactic and semantic probing tasks, we demonstrate the emergent structural information in the role vectors and improved syntactic interpretability in the TPR layer outputs.
Continuous Language Generative Flow
Recent years have witnessed various types of generative models for natural language generation (NLG), especially RNNs or transformer based sequence-to-sequence models, as well as variational autoencoder (VAE) and generative adversarial network (GAN) based models.
iFacetSum: Coreference-based Interactive Faceted Summarization for Multi-Document Exploration
We introduce iFacetSum, a web application for exploring topical document sets.
Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation
Conducting a manual evaluation is considered an essential part of summary evaluation methodology.
Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA
After adversarial training, the baseline's performance improves but is still limited on the adversarial evaluation.
Towards Robustifying NLI Models Against Lexical Dataset Biases
While deep learning models are making fast progress on the task of Natural Language Inference, recent studies have also shown that these models achieve high accuracy by exploiting several dataset biases, and without deep understanding of the language semantics.
ConjNLI: Natural Language Inference Over Conjunctive Sentences
Reasoning about conjuncts in conjunctive sentences is important for a deeper understanding of conjunctions in English and also how their usages and semantics differ from conjunctive and disjunctive boolean logic.
ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning
Recent commonsense-reasoning tasks are typically discriminative in nature, where a model answers a multiple-choice question for a certain context.
Proposition-Level Clustering for Multi-Document Summarization
Text clustering methods were traditionally incorporated into multi-document summarization (MDS) as a means for coping with considerable information repetition.
Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning
In this work, we study pre-trained language models that generate explanation graphs in an end-to-end manner and analyze their ability to learn the structural constraints and semantics of such graphs.
Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings
Transformers generalize to novel compositions of structures and entities after being trained on a complex dataset, but easily overfit on datasets of insufficient complexity.
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
We propose a neural sequence-to-sequence model for direction following, a task that is essential to realizing effective autonomous agents.
SafeCity: Understanding Diverse Forms of Sexual Harassment Personal Stories
With the recent rise of #MeToo, an increasing number of personal stories about sexual harassment and sexual abuse have been shared online.
Analyzing Compositionality-Sensitivity of NLI Models
Therefore, we propose a compositionality-sensitivity testing setup that analyzes models on natural examples from existing datasets that cannot be solved via lexical features alone (i. e., on which a bag-of-words model gives a high probability to one wrong label), hence revealing the models' actual compositionality awareness.
On Curriculum Learning for Commonsense Reasoning
Hence, we examine the effect of a human-like easy-to-difficult curriculum during finetuning of language models for commonsense reasoning tasks.
GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution
Learning from multimodal data has become a popular research topic in recent years.
Evaluating and Improving Factuality in Multimodal Abstractive Summarization
Current metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judgment, but they do not account for the vision modality and thus are not adequate for vision-and-language summarization.
Non-Sequential Graph Script Induction via Multimedia Grounding
To automate the induction of such graph scripts for given tasks, we propose to take advantage of loosely aligned videos of people performing the tasks.
Soft Self-Consistency Improves Language Model Agents
Current "sample and select" methods such as self-consistency (SC) rely on majority voting to score answers.
FixMyPose: Pose Correctional Captioning and Retrieval
During the correctional-captioning task, models must generate descriptions of how to move from the current to target pose image, whereas in the retrieval task, models should select the correct target pose given the initial pose and correctional description.
CAISE: Conversational Agent for Image Search and Editing
To our knowledge, this is the first dataset that provides conversational image search and editing annotations, where the agent holds a grounded conversation with users and helps them to search and edit images according to their requests.
Generating Summaries with Controllable Readability Levels
Readability refers to how easily a reader can understand a written text.
The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions
For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve.
Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks
Motivated by the failure of a Transformer model on the SCAN compositionality challenge (Lake and Baroni, 2018), which requires parsing a command into actions, we propose two auxiliary sequence prediction tasks that track the progress of function and argument semantics, as additional training supervision.
Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks
Motivated by the failure of a Transformer model on the SCAN compositionality challenge (Lake and Baroni, 2018), which requires parsing a command into actions, we propose two auxiliary sequence prediction tasks as additional training supervision.
GENE: Global Event Network Embedding
Current methods for event representation ignore related events in a corpus-level global context.
Debiasing Multimodal Models via Causal Information Minimization
In this paper, we study bias arising from confounders in a causal graph for multimodal data and examine a novel approach that leverages causally-motivated information minimization to learn the confounder representations.
Automatically Learning Data Augmentation Policies for Dialogue Tasks
Automatic data augmentation (AutoAugment) (Cubuk et al., 2019) searches for optimal perturbation policies via a controller trained using performance rewards of a sampled policy on the target task, hence reducing data-level model bias.
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning
In order to jointly learn from all proof graphs and exploit the correlations between multiple proofs for a question, we pose this task as a set generation problem over structured output spaces where each proof is represented as a directed graph.
Extending Multi-Document Summarization Evaluation to the Interactive Setting
In this paper, we develop an end-to-end evaluation framework for interactive summarization, focusing on expansion-based interaction, which considers the accumulating information along a user session.
CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations
Empirically, on the Room-Across-Room dataset, we show that our multilingual agent gets large improvements in all metrics over the strong baseline model when generalizing to unseen environments with the cross-lingual language representation and the environment-agnostic visual representation.
SETSum: Summarization and Visualization of Student Evaluations of Teaching
Ten university professors from diverse departments serve as evaluators of the system and all agree that SETSum helps them interpret SET results more efficiently; and 6 out of 10 instructors prefer our system over the standard static PDF report (while the remaining 4 would like to have both).
Exclusive Supermask Subnetwork Training for Continual Learning
Although there is no forgetting, the performance of SupSup is sub-optimal because fixed weights restrict its representational power.
DAM: Dynamic Adapter Merging for Continual Video QA Learning
Our DAM model outperforms prior state-of-the-art continual learning approaches by 9. 1% while exhibiting 1. 9% less forgetting on 6 VidQA datasets spanning various domains.
How can NLP Help Revitalize Endangered Languages? A Case Study and Roadmap for the Cherokee Language
We hope that our work serves not only to inform the NLP community about Cherokee, but also to provide inspiration for future work on endangered languages in general.
Efficient Few-Shot Fine-Tuning for Opinion Summarization
In the same vein, we pre-train the adapters in a query-based manner on customer reviews and then fine-tune them on annotated datasets.
Interactive Query-Assisted Summarization via Deep Reinforcement Learning
Interactive summarization is a task that facilitates user-guided exploration of information within a document set.
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations
We observe that (1) GPT-3 explanations are as grammatical as human explanations regardless of the hardness of the test samples, (2) for easy examples, GPT-3 generates highly supportive explanations but human explanations are more generalizable, and (3) for hard examples, human explanations are significantly better than GPT-3 explanations both in terms of label-supportiveness and generalizability judgements.
Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks
For knowledge integration to yield peak performance, it is critical to select a knowledge graph (KG) that is well-aligned with the given task's objective.
VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives
In this paper, we show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason (RRR) metrics by optimizing for four key model objectives: (1) accurate predictions given limited but sufficient information (Sufficiency); (2) max-entropy predictions given no important information (Uncertainty); (3) invariance of predictions to changes in unimportant features (Invariance); and (4) alignment between model FI explanations and human FI explanations (Plausibility).
Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?
MPoSM can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction.
WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models
While vision-and-language models perform well on tasks such as visual question answering, they struggle when it comes to basic human commonsense reasoning skills.
Ranked #1 on
Common Sense Reasoning
on WinoGAViL
Continual Few-Shot Learning for Text Classification
In this work, we propose a continual few-shot learning (CFL) task, in which a system is challenged with a difficult phenomenon and asked to learn to correct mistakes with only a few (10 to 15) training examples.
NDH-Full: Learning and Evaluating Navigational Agents on Full-Length Dialogue
In this paper, we explore the Navigation from Dialogue History (NDH) task, which is based on the Cooperative Vision-and-Dialogue Navigation (CVDN) dataset, and present a state-of-the-art model which is built upon Vision-Language transformers.
HistAlign: Improving Context Dependency in Language Generation by Aligning with History
Cache-LMs, which augment LMs with a memory of recent history, can increase context dependency and have shown remarkable performance in diverse language generation tasks.
Low-Cost Algorithmic Recourse for Users With Uncertain Cost Functions
Current approaches try to optimize for the cost incurred by users when adopting a recourse, but they assume that all users share the same cost function.
Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality
Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models.
Data Factors for Better Compositional Generalization
However, in contrast to this poor performance, state-of-the-art models trained on larger and more general datasets show better generalization ability.
Multi-Reward Reinforced Summarization with Saliency and Entailment
Abstractive text summarization is the task of compressing and rewriting a long document into a short summary while maintaining saliency, directed logical entailment, and non-redundancy.
Ranked #41 on
Abstractive Text Summarization
on CNN / Daily Mail
Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation
An accurate abstractive summary of a document should contain all its salient information and should be logically entailed by the input document.
Ranked #33 on
Text Summarization
on GigaWord
Object Ordering with Bidirectional Matchings for Visual Reasoning
Visual reasoning with compositional natural language instructions, e. g., based on the newly-released Cornell Natural Language Visual Reasoning (NLVR) dataset, is a challenging task, where the model needs to have the ability to create an accurate mapping between the diverse phrases and the several objects placed in complex arrangements in the image.
Robust Machine Comprehension Models via Adversarial Training
It is shown that many published models for the Stanford Question Answering Dataset (Rajpurkar et al., 2016) lack robustness, suffering an over 50% decrease in F1 score during adversarial evaluation based on the AddSent (Jia and Liang, 2017) algorithm.
Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models
More importantly, we next interpret what these neural models have learned about the linguistic characteristics of AD patients, via analysis based on activation clustering and first-derivative saliency techniques.
Source-Target Inference Models for Spatial Instruction Understanding
Models that can execute natural language instructions for situated robotic tasks such as assembly and navigation have several useful applications in homes, offices, and remote scenarios.
Hierarchically-Attentive RNN for Album Summarization and Storytelling
For this task, we make use of the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode the album photos, select representative (summary) photos, and compose the story.
Ranked #15 on
Visual Storytelling
on VIST
(BLEU-3 metric)
Multi-Task Video Captioning with Video and Entailment Generation
Video captioning, the task of describing the content of a video, has seen some promising improvements in recent years with sequence-to-sequence models, but accurately learning the temporal and logical dynamics involved in the task still remains a challenge, especially given the lack of sufficient annotated data.
Reinforced Video Captioning with Entailment Rewards
Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they optimize word-level cross-entropy loss during training.
Video Highlight Prediction Using Audience Chat Reactions
Sports channel video portals offer an exciting domain for research on multimodal, multilingual analysis.
The Role of Context Types and Dimensionality in Learning Word Embeddings
We provide the first extensive evaluation of how using different types of context to learn skip-gram word embeddings affects performance on a wide range of intrinsic and extrinsic NLP tasks.
Contextual RNN-GANs for Abstract Reasoning Diagram Generation
Understanding, predicting, and generating object motions and transformations is a core problem in artificial intelligence.
Coherent Dialogue with Attention-based Language Models
We model coherent conversation continuation via RNN-based dialogue models equipped with a dynamic attention mechanism.
Sort Story: Sorting Jumbled Images and Captions into Stories
Temporal common sense has applications in AI tasks such as QA, multi-document summarization, and human-AI communication.
Navigational Instruction Generation as Inverse Reinforcement Learning with Neural Machine Translation
We first decide which information to share with the user according to their preferences, using a policy trained from human demonstrations via inverse reinforcement learning.
Interpreting Neural Networks to Improve Politeness Comprehension
We present an interpretable neural network approach to predicting and understanding politeness in natural language requests.
Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions
We introduce the novel problem of determining the relevance of questions to images in VQA.
Who did What: A Large-Scale Person-Centered Cloze Dataset
We have constructed a new "Who-did-What" dataset of over 200, 000 fill-in-the-gap (cloze) multiple choice reading comprehension problems constructed from the LDC English Gigaword newswire corpus.
Charagram: Embedding Words and Sentences via Character n-grams
We present Charagram embeddings, a simple approach for learning character-based compositional models to embed textual sequences.
Learning Articulated Motion Models from Visual and Lingual Signals
In this paper, we present a multimodal learning framework that incorporates both visual and lingual information to estimate the structure and parameters that define kinematic models of articulated objects.
Mapping Unseen Words to Task-Trained Embedding Spaces
We consider the supervised training setting in which we learn task-specific word embeddings.
We Are Humor Beings: Understanding and Predicting Visual Humor
We collect two datasets of abstract scenes that facilitate the study of humor at both the scene-level and the object-level.
Towards Universal Paraphrastic Sentence Embeddings
We again find that the word averaging models perform well for sentence similarity and entailment, outperforming LSTMs.
Accurate Vision-based Vehicle Localization using Satellite Imagery
We propose a method for accurately localizing ground vehicles with the aid of satellite imagery.
From Paraphrase Database to Compositional Paraphrase Model and Back
The Paraphrase Database (PPDB; Ganitkevitch et al., 2013) is an extensive semantic resource, consisting of a list of phrase pairs with (heuristic) confidence estimates.
Web-scale Surface and Syntactic n-gram Features for Dependency Parsing
We develop novel first- and second-order features for dependency parsing based on the Google Syntactic Ngrams corpus, a collection of subtree counts of parsed sentences from scanned books.
Dynamic Multi-Level Multi-Task Learning for Sentence Simplification
In this work, we first present a strong pointer-copy mechanism based sequence-to-sequence sentence simplification model, and then improve its entailment and paraphrasing capabilities via multi-task learning with related auxiliary tasks of entailment and paraphrase generation.
Ranked #2 on
Text Simplification
on Newsela
