Search Results for author:
Found 322 papers, 209 papers with code
GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution
Learning from multimodal data has become a popular research topic in recent years.
GENE: Global Event Network Embedding
Current methods for event representation ignore related events in a corpus-level global context.
The Effect of Pretraining on Extractive Summarization for Scientific Documents
Large pretrained models have seen enormous success in extractive summarization tasks.
Multimodal Intent Discovery from Livestream Videos
We construct and present a new multimodal dataset consisting of software instructional livestreams and containing manual annotations for both detailed and abstract procedural intent that enable training and evaluation of joint video and text understanding models.
Continual Few-Shot Learning for Text Classification
In this work, we propose a continual few-shot learning (CFL) task, in which a system is challenged with a difficult phenomenon and asked to learn to correct mistakes with only a few (10 to 15) training examples.
An Overview of Uncertainty Calibration for Text Classification and the Role of Distillation
Many recalibration methods have been proposed in the literature for quantifying predictive uncertainty and calibrating model outputs, with varying degrees of complexity.
Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization
Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image sequence (or visual story).
NDH-Full: Learning and Evaluating Navigational Agents on Full-Length Dialogue
In this paper, we explore the Navigation from Dialogue History (NDH) task, which is based on the Cooperative Vision-and-Dialogue Navigation (CVDN) dataset, and present a state-of-the-art model which is built upon Vision-Language transformers.
Inducing Transformer’s Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks
Motivated by the failure of a Transformer model on the SCAN compositionality challenge (Lake and Baroni, 2018), which requires parsing a command into actions, we propose two auxiliary sequence prediction tasks as additional training supervision.
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning
Recent advances in commonsense reasoning have been fueled by the availability of large-scale human annotated datasets.
On Curriculum Learning for Commonsense Reasoning
Hence, we examine the effect of a human-like easy-to-difficult curriculum during finetuning of language models for commonsense reasoning tasks.
ERNIE-NLI: Analyzing the Impact of Domain-Specific External Knowledge on Enhanced Representations for NLI
We examine the effect of domain-specific external knowledge variations on deep large scale language model performance.
Interactive Query-Assisted Summarization via Deep Reinforcement Learning
Interactive summarization is a task that facilitates user-guided exploration of information within a document set.
Enhancing Knowledge Selection for Grounded Dialogues via Document Semantic Graphs
In this work, we propose to automatically convert the background knowledge documents into document semantic graphs and then perform knowledge selection over such graphs.
RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios
We introduce RESIN-11, a new schema-guided event extraction&prediction framework that can be applied to a large variety of newsworthy scenarios.
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation
To address this gap, we first introduce a multimodal unlearning benchmark, UnLOK-VQA (Unlearning Outside Knowledge VQA), as well as an attack-and-defense framework to evaluate methods for deleting specific multimodal knowledge from MLLMs.
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
Currently, existing surveys on LLM safety primarily focus on specific stages of the LLM lifecycle, e. g., deployment phase or fine-tuning phase, lacking a comprehensive understanding of the entire "lifechain" of LLMs.
Retrieval-Augmented Generation with Conflicting Evidence
Large language model (LLM) agents are increasingly employing retrieval-augmented generation (RAG) to improve the factuality of their responses.
Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization
Video-MSG consists of three steps, where in the first two steps, Video-MSG creates Video Sketch, a fine-grained spatio-temporal plan for the final video, specifying background, foreground, and object trajectories, in the form of draft video frames.
CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization
Evaluating creative text such as human-written stories using language models has always been a challenging task -- owing to the subjectivity of multi-annotator ratings.
MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration
We investigate how iterative collaboration among multiple instances and types of large language models (LLMs) enhances subtasks in the refinement process, such as error detection, critiquing unfaithful sentences, and making corrections based on critiques.
VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation
To support diverse tasks and complex instructions, we employ a curriculum learning strategy: first aligning the MLLM and video diffusion model with large-scale instructional image editing data, followed by end-to-end fine-tuning on high-quality multitask video data.
Symbolic Mixture-of-Experts: Adaptive Skill-based Routing for Heterogeneous Reasoning
We propose a skill-based recruiting strategy that dynamically selects the most relevant set of expert LLMs for diverse reasoning tasks based on their strengths.
RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
Building on this finding, we propose RSQ (Rotate, Scale, then Quantize), which (1) applies rotations (orthogonal transformation) to the model to mitigate outliers (those with exceptionally large magnitude), (2) scales the token feature based on its importance, and (3) quantizes the model using the GPTQ framework with the second-order statistics computed by scaled tokens.
MutaGReP: Execution-Free Repository-Grounded Plan Search for Code-Use
When a human requests an LLM to complete a coding task using functionality from a large code repository, how do we provide context from the repo to the LLM?
UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning
We find that UPCORE improves both standard metrics and AUC, benefitting from positive transfer between the coreset and pruned points while reducing negative transfer from the forget set to points outside of it.
Learning to Generate Unit Tests for Automated Debugging
However, we uncover a trade-off between generating unit test inputs that reveal errors when given a faulty code and correctly predicting the unit test output without access to the gold solution.
TimeRefine: Temporal Grounding with Time Refining Video LLM
Recent work has focused on enabling Video LLMs to perform video temporal grounding via next-token prediction of temporal timestamps.
Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel
In this paper, we introduce a Self-Refining Data Flywheel (SRDF) that generates high-quality and large-scale navigational instruction-trajectory pairs by iteratively refining the data pool through the collaboration between two models, the instruction generator and the navigator, without any human-in-the-loop annotation.
QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarization
The Pyramid human evaluation protocol, which assesses content selection by breaking the reference summary into sub-units and verifying their presence in the system summary, has been widely adopted.
SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts
The academic field of learning instruction-guided visual navigation can be generally categorized into high-level category-specific search and low-level language-guided navigation, depending on the granularity of language instruction, in which the former emphasizes the exploration process, while the latter concentrates on following detailed textual commands.
Reverse Thinking Makes LLMs Stronger Reasoners
Reverse thinking plays a crucial role in human reasoning.
DreamRunner: Fine-Grained Storytelling Video Generation with Retrieval-Augmented Motion Adaptation
To address these challenges, we propose DreamRunner, a novel story-to-video generation method: First, we structure the input script using a large language model (LLM) to facilitate both coarse-grained scene planning as well as fine-grained object-level layout and motion planning.
VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement
In (2) refinement planning, we identify accurately generated objects and then create localized prompts to refine other areas in the video.
Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level
In this paper, we introduce Motion-Grounded Video Reasoning, a new motion understanding task that requires generating visual answers (video segmentation masks) according to the input question, and hence needs implicit spatiotemporal reasoning and grounding.
M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
However, there are difficulties in applying these methods in real-world scenarios: (a) questions often require information across different pages or documents, where MLMs cannot handle many long documents; (b) documents often have important information in visual elements such as figures, but text extraction tools ignore them.
Self-Consistency Preference Optimization
Self-alignment, whereby models learn to improve themselves without human annotation, is a rapidly growing research area.
Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM
In FactualityPrompts, an open-ended text generation benchmark, sampling using APD significantly boosts factuality in comparison to the CD sampling and its variants, and achieves state-of-the-art results for Pythia 6. 9B and OPT 6. 7B.
On Positional Bias of Faithfulness for Long-form Summarization
Large Language Models (LLMs) often exhibit positional bias in long-context settings, under-attending to information in the middle of inputs.
Unbounded: A Generative Infinite Game of Character Life Simulation
We introduce the concept of a generative infinite game, a video game that transcends the traditional boundaries of finite, hard-coded systems by using generative models.
Teaching Models to Balance Resisting and Accepting Persuasion
PBT consistently improves resistance to misinformation and resilience to being challenged while also resulting in the best overall performance on holistic data containing both positive and negative persuasion.
SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation
To address these, we propose SAFREE, a novel, training-free approach for safe T2I and T2V, that does not alter the model's weights.
Adapt-$\infty$: Scalable Lifelong Multimodal Instruction Tuning via Dynamic Data Selection
This data selector samples a subset of the most important samples from each skill cluster for training.
LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints
To address the performance gap between open-source and proprietary models, we propose the Decompose, Critique and Refine (DeCRIM) self-correction pipeline, which enhances LLMs' ability to follow constraints.
Glider: Global and Local Instruction-Driven Expert Router
Our experiments using T5-based models for T0 and FLAN tasks demonstrate that GLIDER achieves substantially improved held-in performance while maintaining strong generalization on held-out tasks.
Localizing Factual Inconsistencies in Attributable Text Generation
There has been an increasing interest in detecting hallucinations in model-generated texts, both manually and automatically, at varying levels of granularity.
DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback
Students are iteratively trained and evaluated on generated data, and their feedback (in the form of errors or weak skills) is reported to the agent after each iteration.
What Matters for Model Merging at Scale?
This leaves many unanswered questions about the effect of scaling model size and how it interplays with other key factors -- like the base model quality and number of expert models -- , to affect the merged model's performance.
VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning
In this work, we show that a strong off-the-shelf frozen pretrained visual encoder, along with a well designed prediction model, can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning without the need for pretraining the prediction model, nor requiring additional supervision from language or ASR.
LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits
Our results on commonsense and math reasoning tasks demonstrate that LASeR can boost iterative LLM optimization by optimizing for multiple RMs, improving the absolute average accuracy of Llama-3-8B over three datasets by 2. 67% over training with ensemble RM scores while also showing superior training efficiency (e. g., a 2x speedup).
MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
Moreover, to ensure effective refinement, we employ a multi-agent loop with three agents: Solver, Reviewer (which generates targeted feedback based on step-wise RM scores), and the Refiner (which incorporates feedback).
AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge
Knowledge conflict arises from discrepancies between information in the context of a large language model (LLM) and the knowledge stored in its parameters.
Knowledge-Aware Reasoning over Multimodal Semi-structured Tables
With the evolution of AI models capable of multimodal reasoning, it is pertinent to assess their efficacy in handling such structured data.
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task.
System-1.x: Learning to Balance Fast and Slow Planning with Language Models
To this end, we propose the System-1. x Planner, a controllable planning framework with LLMs that is capable of generating hybrid plans and balancing between the two planning modes based on the difficulty of the problem at hand.
Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models
Vision-and-Language Navigation (VLN) has gained increasing attention over recent years and many approaches have emerged to advance their development.
Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?
With this goal in mind, this paper critiques the standard formulation of the model editing problem and proposes a formal testbed for model editing research.
See It from My Perspective: Diagnosing the Western Cultural Bias of Large Vision-Language Models in Image Understanding
Moreover, while prompting in the language of a target culture can lead to reductions in bias, it is not a substitute for building AI more representative of the world's languages.
REAL Sampling: Boosting Factuality and Diversity of Open-Ended Generation via Asymptotic Entropy
If a LLM's entropy is higher than the asymptotic entropy (i. e., the LLM is more uncertain than it should be), the THF model predicts a high hallucination hazard, which leads to a lower p threshold in REAL sampling.
Are language models rational? The case of coherence norms and belief revision
We consider both logical coherence norms as well as coherence norms tied to the strength of belief.
The Power of Summary-Source Alignments
Multi-document summarization (MDS) is a challenging task, often decomposed to subtasks of salience and redundancy detection, followed by text generation.
LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models
To calibrate both implicit and explicit confidence markers, we introduce a pragmatic, listener-aware finetuning method (LACIE) that models the listener, considering not only whether an answer is right, but whether it will be accepted by a listener.
VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos
Specifically, we incorporate multigranularity information into a tree-based representation, allowing VideoTree to extract query-relevant details from long videos in a coarse-to-fine manner.
RACCooN: A Versatile Instructional Video Editing Framework with Auto-Generated Narratives
(3) RACCooN also plans to imagine new objects in a given video, so users simply prompt the model to receive a detailed video editing plan for complex video editing.
FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation
Controllable text-to-image (T2I) diffusion models generate images conditioned on both text prompts and semantic inputs of other modalities like edge maps.
Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model
ControlNets are widely used for adding spatial control to text-to-image diffusion models with different conditions, such as depth maps, scribbles/sketches, and human poses.
Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts
In light of this, we reintroduce this dense design into the generalist models, to facilitate the development of generalist models with high segmentation quality.
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Despite these efforts, such models encounter challenges such as limited multilingual capabilities, risks of catastrophic forgetting during continual pretraining, and the high costs of training models from scratch, alongside the need to align with AI safety standards and regulatory frameworks.
EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
Then, we enable the LLM to continuously adapt the generated environments to progressively improve the skills that the agent is weak at, by providing feedback to the LLM in the form of the agent's performance.
DAM: Dynamic Adapter Merging for Continual Video QA Learning
Our DAM model outperforms prior state-of-the-art continual learning approaches by 9. 1% while exhibiting 1. 9% less forgetting on 6 VidQA datasets spanning various domains.
SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging.
Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training
Highlighting particularly relevant regions of an image can improve the performance of vision-language models (VLMs) on various vision-language (VL) tasks by guiding the model to attend more closely to these regions of interest.
NewsQs: Multi-Source Question Generation for the Inquiring Mind
We present NewsQs (news-cues), a dataset that provides question-answer pairs for multiple news documents.
Evaluating Very Long-Term Conversational Memory of LLM Agents
Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 300 turns and 9K tokens on avg., over up to 35 sessions.
Soft Self-Consistency Improves Language Model Agents
Current "sample and select" methods such as self-consistency (SC) rely on majority voting to score answers.
GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
We further characterize the game-theoretic properties of LLMs, such as equilibrium and Pareto Efficiency in repeated games.
Rethinking Machine Unlearning for Large Language Models
We explore machine unlearning (MU) in the domain of large language models (LLMs), referred to as LLM unlearning.
Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings
Transformers generalize to novel compositions of structures and entities after being trained on a complex dataset, but easily overfit on datasets of insufficient complexity.
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion
Despite impressive advancements in recent multimodal reasoning approaches, they are still limited in flexibility and efficiency, as these models typically process only a few fixed modality inputs and require updates to numerous parameters.
Ranked #1 on
Question Answering
on SQA3D
VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigation
Outdoor Vision-and-Language Navigation (VLN) requires an agent to navigate through realistic 3D outdoor environments based on natural language instructions.
MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models
Experiments on seven widely used commonsense and math reasoning benchmarks show that MAGDi improves the reasoning capabilities of smaller models, outperforming several methods that distill from a single teacher and multiple teachers.
ReGAL: Refactoring Programs to Discover Generalizable Abstractions
While large language models (LLMs) are increasingly being used for program synthesis, they lack the global view needed to develop useful abstractions; they generally predict programs one at a time, often repeating the same functionality.
Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences
However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated.
The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
In this paper, we present the surprising conclusion that current pretrained language models often generalize relatively well from easy to hard data, even performing as well as oracle models finetuned on hard data.
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts
In light of this we reintroduce this dense design into the generalist models to facilitate the development of generalist models with high segmentation quality.
CoDi-2: In-Context Interleaved and Interactive Any-to-Any Generation
We present CoDi-2 a Multimodal Large Language Model (MLLM) for learning in-context interleaved multimodal representations.
A Simple LLM Framework for Long-Range Video Question-Answering
Furthermore, we show that a specialized prompt that asks the LLM first to summarize the noisy short-term visual captions and then answer a given input question leads to a significant LVQA performance boost.
Ranked #4 on
Zero-Shot Video Question Answer
on NExT-GQA
Merging by Matching Models in Task Parameter Subspaces
Model merging aims to cheaply combine individual task-specific models into a single multitask model.
CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation
We present CoDi-2, a versatile and interactive Multimodal Large Language Model (MLLM) that can follow complex multimodal interleaved instructions, conduct in-context learning (ICL), reason, chat, edit, etc., in an any-to-any input-output modality paradigm.
Debiasing Multimodal Models via Causal Information Minimization
In this paper, we study bias arising from confounders in a causal graph for multimodal data and examine a novel approach that leverages causally-motivated information minimization to learn the confounder representations.
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
Despite the efficiency of PEFT methods, the size of expert models can make it onerous to retrieve expert models per query over high-latency networks like the Internet or serve multiple experts on a single GPU.
Multimodal Representation Learning by Alternating Unimodal Adaptation
This optimization process is controlled by a gradient modification mechanism to prevent the shared head from losing previously acquired information.
ADaPT: As-Needed Decomposition and Planning with Language Models
Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment.
Data Factors for Better Compositional Generalization
However, in contrast to this poor performance, state-of-the-art models trained on larger and more general datasets show better generalization ability.
Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation
With extensive experimentation and human evaluation on a range of model configurations (LLM, VQA, and T2I), we empirically demonstrate that DSG addresses the challenges noted above.
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
Large Language Models (LLMs) are frequently used for multi-faceted language generation and evaluation tasks that involve satisfying intricate user constraints or taking into account multiple aspects and criteria.
DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning
In the second stage, we use a diagram generator, DiagramGLIGEN, and a text label rendering module to generate diagrams (with clear text labels) following the diagram plans.
Generating Summaries with Controllable Readability Levels
Readability refers to how easily a reader can understand a written text.
D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning
There are two dominant approaches: (1) geometry-based data selection for maximizing data diversity in the coreset, and (2) functions that assign difficulty scores to samples based on training dynamics.
Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
An increasing number of vision-language tasks can be handled with little to no training, i. e., in a zero and few-shot manner, by marrying large language models (LLMs) to vision encoders, resulting in large vision-language models (LVLMs).
ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models
We first determine the sparsity ratios of different layers or blocks by leveraging the global importance score, which is efficiently computed based on the zeroth-order approximation of the global model gradients.
Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy
Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks, however, they have issues like (a) High Memory Usage, due to duplication of the network layers into multiple copies as experts; and (b) Redundancy in Experts, as common learning-based routing policies suffer from representational collapse.
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
Large vision-language models (LVLMs) have shown remarkable abilities in understanding visual information with human languages.
Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks
Experimentally, we show that even state-of-the-art model editing methods such as ROME struggle to truly delete factual information from models like GPT-J, as our whitebox and blackbox attacks can recover "deleted" information from an edited model 38% of the time.
VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning
Our experiments demonstrate that our proposed VideoDirectorGPT framework substantially improves layout and movement control in both single- and multi-scene video generation and can generate multi-scene videos with consistency, while achieving competitive performance with SOTAs in open-domain single-scene T2V generation.
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
In each round, ReConcile initiates discussion between agents via a 'discussion prompt' that consists of (a) grouped answers and explanations generated by each agent in the previous round, (b) their confidence scores, and (c) demonstrations of answer-rectifying human explanations, used for convincing other agents.
Unified Coarse-to-Fine Alignment for Video-Text Retrieval
Specifically, our model captures the cross-modal similarity information at different granularity levels.
Ranked #12 on
Video Retrieval
on MSR-VTT
Scaling Data Generation in Vision-and-Language Navigation
Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents.
Exploring Continual Learning for Code Generation Models
Large-scale code generation models such as Codex and CodeT5 have achieved impressive performance.
On Conditional and Compositional Language Model Differentiable Prompting
Prompts have been shown to be an effective method to adapt a frozen Pretrained Language Model (PLM) to perform well on downstream tasks.
Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization
We first show that teacher LLMs can indeed intervene on student reasoning to improve their performance.
TIES-Merging: Resolving Interference When Merging Models
To address this, we propose our method, TRIM, ELECT SIGN & MERGE (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign.
Non-Sequential Graph Script Induction via Multimedia Grounding
To automate the induction of such graph scripts for given tasks, we propose to take advantage of loosely aligned videos of people performing the tasks.
MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies
Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Q relative to the data distribution P -- that is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood estimation (MLE).
Visual Programming for Text-to-Image Generation and Evaluation
First, we introduce VPGen, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation.
Any-to-Any Generation via Composable Diffusion
We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities.
Ranked #8 on
Audio Generation
on AudioCaps
(FAD metric)
Paxion: Patching Action Knowledge in Video-Language Foundation Models
Action knowledge involves the understanding of textual, visual, and temporal aspects of actions.
Ranked #42 on
Video Question Answering
on NExT-QA
(using extra training data)
Self-Chained Image-Language Model for Video Localization and Question Answering
SeViLA framework consists of two modules: Localizer and Answerer, where both are parameter-efficiently fine-tuned from BLIP-2.
Ranked #2 on
Video Question Answering
on NExT-QA (Efficient)
HistAlign: Improving Context Dependency in Language Generation by Aligning with History
Cache-LMs, which augment LMs with a memory of recent history, can increase context dependency and have shown remarkable performance in diverse language generation tasks.
ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness
Multi-step reasoning ability is fundamental to many natural language tasks, yet it is unclear what constitutes a good reasoning chain and how to evaluate them.
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
In this paper, we propose LayoutBench, a diagnostic benchmark for layout-guided image generation that examines four categories of spatial control skills: number, position, size, and shape.
Improving Vision-and-Language Navigation by Generating Future-View Image Semantics
We then fine-tune the agent on the VLN task with an auxiliary loss that minimizes the difference between the view semantics generated by the agent and the ground truth view semantics of the next step.
Hierarchical Video-Moment Retrieval and Step-Captioning
Our hierarchical benchmark consists of video retrieval, moment retrieval, and two novel moment segmentation and step captioning tasks.
Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models
As general purpose vision models get increasingly effective at a wide set of tasks, it is imperative that they be consistent across the tasks they support.
Faithfulness-Aware Decoding Strategies for Abstractive Summarization
We present a systematic study of the effect of generation techniques such as beam search and nucleus sampling on faithfulness in abstractive summarization.
Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
This finding raises questions about how past work relies on Causal Tracing to select which model layers to edit.
MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation
We propose MURMUR, a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning.
Vision Transformers are Parameter-Efficient Audio-Visual Learners
To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT.
Ranked #4 on
Audio-visual Question Answering
on MUSIC-AVQA
VindLU: A Recipe for Effective Video-and-Language Pretraining
Furthermore, our model also obtains state-of-the-art video question-answering results on ActivityNet-QA, MSRVTT-QA, MSRVTT-MC and TVQA.
Ranked #2 on
Video Retrieval
on Condensed Movies
(using extra training data)
Unifying Vision, Text, and Layout for Universal Document Processing
UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation.
Ranked #5 on
Visual Question Answering (VQA)
on InfographicVQA
(using extra training data)
Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality
Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models.
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
We present Perceiver-VL, a vision-and-language framework that efficiently handles high-dimensional multimodal inputs such as long videos and text.
Evaluating the Factual Consistency of Large Language Models Through News Summarization
To generate summaries that are factually inconsistent, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent.
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations
We observe that (1) GPT-3 explanations are as grammatical as human explanations regardless of the hardness of the test samples, (2) for easy examples, GPT-3 generates highly supportive explanations but human explanations are more generalizable, and (3) for hard examples, human explanations are significantly better than GPT-3 explanations both in terms of label-supportiveness and generalizability judgements.
Evaluating and Improving Factuality in Multimodal Abstractive Summarization
Current metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judgment, but they do not account for the vision modality and thus are not adequate for vision-and-language summarization.
Exclusive Supermask Subnetwork Training for Continual Learning
Although there is no forgetting, the performance of SupSup is sub-optimal because fixed weights restrict its representational power.
TVLT: Textless Vision-Language Transformer
In this work, we present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and-language representation learning with minimal modality-specific design, and do not use text-specific modules such as tokenization or automatic speech recognition (ASR).
Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees
We demonstrate that SP-Search effectively represents the generative process behind human summaries using modules that are typically faithful to their intended behavior.
StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation
Hence, we first propose the task of story continuation, where the generated visual story is conditioned on a source image, allowing for better generalization to narratives with new characters.
Ranked #3 on
Story Continuation
on FlintstonesSV
Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization
Though extractive summarization is less prone to the common unfaithfulness issues of abstractive summaries, does that mean extractive is equal to faithful?
WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models
While vision-and-language models perform well on tasks such as visual question answering, they struggle when it comes to basic human commonsense reasoning skills.
Ranked #1 on
Common Sense Reasoning
on WinoGAViL
CoSIm: Commonsense Reasoning for Counterfactual Scene Imagination
Next, a counterfactual imagined scene change (in textual form) is applied, and the model has to predict the new response to the initial question based on this scene change.
SETSum: Summarization and Visualization of Student Evaluations of Teaching
Ten university professors from diverse departments serve as evaluators of the system and all agree that SETSum helps them interpret SET results more efficiently; and 6 out of 10 instructors prefer our system over the standard static PDF report (while the remaining 4 would like to have both).
CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations
Empirically, on the Room-Across-Room dataset, we show that our multilingual agent gets large improvements in all metrics over the strong baseline model when generalizing to unseen environments with the cross-lingual language representation and the environment-agnostic visual representation.
Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?
MPoSM can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction.
VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives
In this paper, we show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason (RRR) metrics by optimizing for four key model objectives: (1) accurate predictions given limited but sufficient information (Sufficiency); (2) max-entropy predictions given no important information (Uncertainty); (3) invariance of predictions to changes in unimportant features (Invariance); and (4) alignment between model FI explanations and human FI explanations (Plausibility).
Enhanced Knowledge Selection for Grounded Dialogues via Document Semantic Graphs
Providing conversation models with background knowledge has been shown to make open-domain dialogues more informative and engaging.
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
LST saves 69% of the memory costs to fine-tune the whole network, while other methods only save 26% of that in similar parameter usages (hence, 2. 7x more memory savings).
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Revealing Single Frame Bias for Video-and-Language Learning
Training an effective video-and-language model intuitively requires multiple frames as model inputs.
Ranked #5 on
Video Retrieval
on SSv2-template retrieval
(using extra training data)
Fine-grained Image Captioning with CLIP Reward
Toward more descriptive and distinctive caption generation, we propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function.
Ranked #26 on
Image Captioning
on COCO Captions
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction.
On the Limits of Evaluating Embodied Agent Model Generalization Using Validation Sets
Natural language guided embodied task completion is a challenging problem since it requires understanding natural language instructions, aligning them with egocentric visual observations, and choosing appropriate actions to execute in the environment to produce desired changes.
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization
We present FactPEGASUS, an abstractive summarization model that addresses the problem of factuality during pre-training and fine-tuning: (1) We augment the sentence selection strategy of PEGASUS's (Zhang et al., 2020) pre-training objective to create pseudo-summaries that are both important and factual; (2) We introduce three complementary components for fine-tuning.
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made.
Ranked #1 on
Few-Shot Text Classification
on RAFT
Efficient Few-Shot Fine-Tuning for Opinion Summarization
In the same vein, we pre-train the adapters in a query-based manner on customer reviews and then fine-tune them on annotated datasets.
How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?
Since a skewed data distribution is considered to be harmful, a sampling strategy is usually used to balance languages in the corpus.
How can NLP Help Revitalize Endangered Languages? A Case Study and Roadmap for the Cherokee Language
We hope that our work serves not only to inform the NLP community about Cherokee, but also to provide inspiration for future work on endangered languages in general.
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Despite recent improvements in abstractive summarization, most current approaches generate summaries that are not factually consistent with the source document, severely restricting their trust and usage in real-world applications.
Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning
In this work, we study pre-trained language models that generate explanation graphs in an end-to-end manner and analyze their ability to learn the structural constraints and semantics of such graphs.
ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound
We introduce an audiovisual method for long-range text-to-video retrieval.
EnvEdit: Environment Editing for Vision-and-Language Navigation
Training on these edit-augmented environments prevents the agent from overfitting to existing environments and helps generalize better to new, unseen environments.
Ranked #2 on
Vision and Language Navigation
on RxR
(using extra training data)
GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models
Providing natural language instructions in prompts is a useful new paradigm for improving task performance of large language models in a zero-shot setting.
LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval
In this work, we propose LoopITR, which combines them in the same network for joint learning.
CAISE: Conversational Agent for Image Search and Editing
To our knowledge, this is the first dataset that provides conversational image search and editing annotations, where the agent holds a grounded conversation with users and helps them to search and edit images according to their requests.
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models
In this work, we investigate the visual reasoning capabilities and social biases of different text-to-image models, covering both multimodal transformer language models and diffusion models.
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding
Specifically, the task involves multi-hop questions that require reasoning over image-caption pairs to identify the grounded visual object being referred to and then predicting a span from the news body text to answer the question.
Proposition-Level Clustering for Multi-Document Summarization
Text clustering methods were traditionally incorporated into multi-document summarization (MDS) as a means for coping with considerable information repetition.
Learning and Analyzing Generation Order for Undirected Sequence Models
On examples with a maximum source and target length of 30 from De-En, WMT'16 English-Romanian, and WMT'21 English-Chinese translation tasks, our learned order outperforms all heuristic generation orders on four out of six tasks.
Analyzing the Limits of Self-Supervision in Handling Bias in Language
We define three broad classes of task descriptions for these tasks: statement, question, and completion, with numerous lexical variants within each class.
VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Our results demonstrate that training the adapter with the weight-sharing technique (4. 18% of total parameters for image-text tasks and 3. 39% for video-text tasks) can match the performance of fine-tuning the entire model.
MLP Architectures for Vision-and-Language Modeling: An Empirical Study
Based on this, we ask an even bolder question: can we have an all-MLP architecture for VL modeling, where both VL fusion and the vision encoder are replaced with MLPs?
Detecting Moments and Highlights in Videos via Natural Language Queries
Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w. r. t.
Ranked #6 on
Video Grounding
on QVHighlights
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs
In this paper, we discuss approaches to detecting when models have beliefs about the world, and we improve on methods for updating model beliefs to be more truthful, with a focus on methods based on learned optimizers or hypernetworks.
Low-Cost Algorithmic Recourse for Users With Uncertain Cost Functions
Current approaches try to optimize for the cost incurred by users when adopting a recourse, but they assume that all users share the same cost function.
Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization
Prior work in this domain has shown that there is ample room for improvement in the generated image sequence in terms of visual quality, consistency and relevance.
Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks
Motivated by the failure of a Transformer model on the SCAN compositionality challenge (Lake and Baroni, 2018), which requires parsing a command into actions, we propose two auxiliary sequence prediction tasks that track the progress of function and argument semantics, as additional training supervision.
iFacetSum: Coreference-based Interactive Faceted Summarization for Multi-Document Exploration
We introduce iFacetSum, a web application for exploring topical document sets.
Finding a Balanced Degree of Automation for Summary Evaluation
In this work, we propose flexible semiautomatic to automatic summary evaluation metrics, following the Pyramid human evaluation method.
Continuous Language Generative Flow
Recent years have witnessed various types of generative models for natural language generation (NLG), especially RNNs or transformer based sequence-to-sequence models, as well as variational autoencoder (VAE) and generative adversarial network (GAN) based models.
InfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection
To defend against machine-generated fake news, an effective mechanism is urgently needed.
MTVR: Multilingual Moment Retrieval in Videos
We introduce mTVR, a large-scale multilingual video moment retrieval dataset, containing 218K English and Chinese queries from 21. 8K TV show video clips.
EmailSum: Abstractive Email Thread Summarization
Furthermore, we find that widely used automatic evaluation metrics (ROUGE, BERTScore) are weakly correlated with human judgments on this email thread summarization task.
Ranked #1 on
Email Thread Summarization
on EmailSum (short)
ChrEnTranslate: Cherokee-English Machine Translation Demo with Quality Estimation and Corrective Feedback
The quantitative evaluation demonstrates that our backbone translation models achieve state-of-the-art translation performance and our quality estimation well correlates with both BLEU and human judgment.
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w. r. t.
Ranked #18 on
Highlight Detection
on QVHighlights
How Much Can CLIP Benefit Vision-and-Language Tasks?
Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world.
Ranked #4 on
Vision and Language Navigation
on RxR
(using extra training data)
VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
We train a multi-modal teacher model on a video-text dataset, and then transfer its knowledge to a student language model with a text dataset.
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e. g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations.
Ranked #11 on
Action Recognition
on Diving-48
Disentangling Online Chats with DAG-Structured LSTMs
In this paper, we apply DAG-LSTMs to the conversation disentanglement task.
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP
NLP has achieved great progress in the past decade through the use of neural models and large labeled datasets.
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task.
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning
In order to jointly learn from all proof graphs and exploit the correlations between multiple proofs for a question, we pose this task as a set generation problem over structured output spaces where each proof is represented as a directed graph.
Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization
On several syntactic and semantic probing tasks, we demonstrate the emergent structural information in the role vectors and improved syntactic interpretability in the TPR layer outputs.
The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
In this paper, we study several under-explored dimensions of FI explanations, providing conceptual and empirical improvements for this form of explanation.
Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters
We also show improvements in a transfer-only setup on the DUC-2004 dataset.
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization
Second, to alleviate the temporal misalignment issue, our method incorporates an entropy minimization-based constrained attention loss, to encourage the model to automatically focus on the correct caption from a pool of candidate ASR captions.
