Search Results for author:
Found 16 papers, 14 papers with code
Multimodal Intent Discovery from Livestream Videos
We construct and present a new multimodal dataset consisting of software instructional livestreams and containing manual annotations for both detailed and abstract procedural intent that enable training and evaluation of joint video and text understanding models.
Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization
Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image sequence (or visual story).
On Curriculum Learning for Commonsense Reasoning
Hence, we examine the effect of a human-like easy-to-difficult curriculum during finetuning of language models for commonsense reasoning tasks.
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning
Recent advances in commonsense reasoning have been fueled by the availability of large-scale human annotated datasets.
REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
Building on these insights, we introduce two benchmark tasks: (1) persona simulation where a model continues a conversation on behalf of a specific user given prior dialogue context; and (2) memory probing where a model answers targeted questions requiring long-term memory of past interactions.
Adapt-$\infty$: Scalable Lifelong Multimodal Instruction Tuning via Dynamic Data Selection
This data selector samples a subset of the most important samples from each skill cluster for training.
Evaluating Very Long-Term Conversational Memory of LLM Agents
Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 300 turns and 9K tokens on avg., over up to 35 sessions.
Debiasing Multimodal Models via Causal Information Minimization
In this paper, we study bias arising from confounders in a causal graph for multimodal data and examine a novel approach that leverages causally-motivated information minimization to learn the confounder representations.
D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning
There are two dominant approaches: (1) geometry-based data selection for maximizing data diversity in the coreset, and (2) functions that assign difficulty scores to samples based on training dynamics.
Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models
As general purpose vision models get increasingly effective at a wide set of tasks, it is imperative that they be consistent across the tasks they support.
StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation
Hence, we first propose the task of story continuation, where the generated visual story is conditioned on a source image, allowing for better generalization to narratives with new characters.
Ranked #3 on
Story Continuation
on FlintstonesSV
Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization
Prior work in this domain has shown that there is ample room for improvement in the generated image sequence in terms of visual quality, consistency and relevance.
Improving Generation and Evaluation of Visual Stories via Semantic Consistency
Therefore, we also provide an exploration of evaluation metrics for the model, focused on aspects of the generated frames such as the presence/quality of generated characters, the relevance to captions, and the diversity of the generated images.
Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension
In this work, we present several effective adversaries and automated data augmentation policy search methods with the goal of making reading comprehension models more robust to adversarial evaluation, but also improving generalization to the source domain as well as new domains and languages.
Clinical Event Detection with Hybrid Neural Architecture
Event detection from clinical notes has been traditionally solved with rule based and statistical natural language processing (NLP) approaches that require extensive domain knowledge and feature engineering.
