Search Results for author:
Found 45 papers, 24 papers with code
The Sound of Water: Inferring Physical Properties from Pouring Liquids
We study the connection between audio-visual observations and the underlying physics of a mundane yet intriguing everyday activity: pouring liquids.
Ranked #1 on
Physical Attribute Prediction
on Sound of Water 50
(using extra training data)
IdentifyMe: A Challenging Long-Context Mention Resolution Benchmark for LLMs
The benchmark also consists of a curated mixture of different mention types and corresponding entities, allowing for a fine-grained analysis of model performance.
Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation
Specifically, we assess the ability of MLLMs to capture specific points of visual differences using self-retrieval, i. e., by retrieving the target image using its generated caption against the other image in the pair serving as the distractor.
No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning
We evaluate and compare several state-of-the-art open-source MLLMs on TrueMatch, and find that our SR approach outperforms them all by a significant margin (e. g. +4. 8% - 7. 1% over Cambrian) while having 1-2 orders of magnitude fewer parameters.
Major Entity Identification: A Generalizable Alternative to Coreference Resolution
Rather than relying on this additional annotation, we propose an alternative referential task, Major Entity Identification (MEI), where we: (a) assume the target entities to be specified in the input, and (b) limit the task to only the frequent entities.
VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment
To this end, we introduce VELOCITI, a benchmark to study Video-LLMs by disentangling and assessing the comprehension of agents, actions, and their associations across multiple events.
MICap: A Unified Model for Identity-aware Movie Descriptions
While previous work has largely ignored identity and generated captions with someone (anonymized names), recent work formulates id-aware captioning as a fill-in-the-blanks (FITB) task, where, given a caption with blanks, the goal is to predict person id labels.
"Previously on ..." From Recaps to Story Summarization
We introduce multimodal story summarization by leveraging TV episode recaps - short video sequences interweaving key story moments from previous episodes to bring viewers up to speed.
NurtureNet: A Multi-task Video-based Approach for Newborn Anthropometry
Our goal is to equip health workers and public health systems with a solution for contactless newborn anthropometry in the community.
FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos
We use the SRL and verb information to create rule-based detailed captions, making sure they capture most of the visual concepts.
Previously on ... From Recaps to Story Summarization
We introduce multimodal story summarization by leveraging TV episode recaps - short video sequences interweaving key story moments from previous episodes to bring viewers up to speed.
Seeing Eye to AI: Comparing Human Gaze and Model Attention in Video Memorability
Understanding what makes a video memorable has important applications in advertising or education technology.
Generalized Cross-domain Multi-label Few-shot Learning for Chest X-rays
Real-world application of chest X-ray abnormality classification requires dealing with several challenges: (i) limited training data; (ii) training and evaluation sets that are derived from different domains; and (iii) classes that appear during training may have partial overlap with classes of interest during evaluation.
How you feelin'? Learning Emotions and Mental States in Movie Scenes
Towards this goal, we formulate emotion understanding as predicting a diverse and multi-label set of emotions at the level of a movie scene and for each character.
GrapeQA: GRaph Augmentation and Pruning to Enhance Question-Answering
Commonsense question-answering (QA) methods combine the power of pre-trained Language Models (LM) with the reasoning provided by Knowledge Graphs (KG).
Ranked #8 on
Question Answering
on OpenBookQA
Test of Time: Instilling Video-Language Models with a Sense of Time
Our work serves as a first step towards probing and instilling a sense of time in existing video-language models without the need for data and compute-intense training from scratch.
Ranked #3 on
Video-Text Retrieval
on Test-of-Time
(using extra training data)
Sonus Texere! Automated Dense Soundtrack Construction for Books using Movie Adaptations
Reading, much like music listening, is an immersive experience that transports readers while taking them on an emotional journey.
Can we Adopt Self-supervised Pretraining for Chest X-Rays?
Chest radiograph (or Chest X-Ray, CXR) is a popular medical imaging modality that is used by radiologists across the world to diagnose heart or lung conditions.
Language Conditioned Spatial Relation Reasoning for 3D Object Grounding
In this work we propose a language-conditioned transformer model for grounding 3D objects and their spatial relations.
Unsupervised Audio-Visual Lecture Segmentation
We formulate lecture segmentation as an unsupervised task that leverages visual, textual, and OCR cues from the lecture, while clip representations are fine-tuned on a pretext self-supervised task of matching the narration with the temporally aligned visual content.
Grounded Video Situation Recognition
Recently, Video Situation Recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, and actions and various verb-role pairs attached to descriptive entities.
Instruction-driven history-aware policies for robotic manipulations
In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions.
Ranked #6 on
Robot Manipulation Generalization
on GEMBench
Learning from Unlabeled 3D Environments for Vision-and-Language Navigation
Our resulting HM3D-AutoVLN dataset is an order of magnitude larger than existing VLN datasets in terms of navigation environments and instructions.
Ranked #1 on
Visual Navigation
on SOON Test
Learning Object Manipulation Skills from Video via Approximate Differentiable Physics
We evaluate our approach on a 3D reconstruction task that consists of 54 video demonstrations sourced from 9 actions such as pull something from right to left or put something in front of something.
Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation
To balance the complexity of large action space reasoning and fine-grained language grounding, we dynamically combine a fine-scale encoding over local observations and a coarse-scale encoding on a global map via graph transformers.
Ranked #5 on
Visual Navigation
on SOON Test
Feature Generation for Long-tail Classification
Oversampling instances of the tail classes attempts to solve this imbalance.
Ranked #1 on
Long-tail Learning
on mini-ImageNet-LT
Airbert: In-domain Pretraining for Vision-and-Language Navigation
Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging.
Ranked #3 on
Vision and Language Navigation
on VLN Challenge
Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
We evaluate our method on simple single- and two-object actions from the Something-Something dataset.
Clustering based Contrastive Learning for Improving Face Representations
We demonstrate our method on the challenging task of learning representations for video face clustering.
Deep Multimodal Feature Encoding for Video Ordering
True understanding of videos comes from a joint analysis of all its modalities: the video frames, the audio track, and any accompanying text such as closed captions.
Learning Interactions and Relationships between Movie Characters
Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels.
The Shmoop Corpus: A Dataset of Stories with Loosely Aligned Summaries
Understanding stories is a challenging reading comprehension problem for machines as it requires reading a large volume of text and following long-range dependencies.
Video Face Clustering with Unknown Number of Clusters
Understanding videos such as TV series and movies requires analyzing who the characters are and what they are doing.
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations.
Ranked #4 on
Temporal Action Localization
on CrossTask
Action Localization
Long Video Retrieval (Background Removed)
+3
Self-Supervised Learning of Face Representations for Video Face Clustering
In this paper, we address video face clustering using unsupervised methods.
Visual Reasoning by Progressive Module Networks
Thus, a module for a new task learns to query existing modules and composes their outputs in order to produce its own output.
Now You Shake Me: Towards Automatic 4D Cinema
We are interested in enabling automatic 4D cinema by parsing physical and special effects from untrimmed movies.
MovieGraphs: Towards Understanding Human-Centric Situations from Videos
Towards this goal, we introduce a novel dataset called MovieGraphs which provides detailed, graph-based annotations of social situations depicted in movie clips.
Situation Recognition with Graph Neural Networks
We address the problem of recognizing situations in images.
Ranked #10 on
Situation Recognition
on imSitu
Relaxed Earth Mover's Distances for Chain- and Tree-connected Spaces and their use as a Loss Function in Deep Learning
The Earth Mover's Distance (EMD) computes the optimal cost of transforming one distribution into another, given a known transport metric between them.
Recovering the Missing Link: Predicting Class-Attribute Associations for Unsupervised Zero-Shot Learning
In this work, we aim to carry out attribute-based zero-shot classification in an unsupervised manner.
MovieQA: Understanding Stories in Movies through Question-Answering
We introduce the MovieQA dataset which aims to evaluate automatic story comprehension from both video and text.
Book2Movie: Aligning Video Scenes With Book Chapters
Such an alignment facilitates finding differences between the adaptation and the original source, and also acts as a basis for deriving rich descriptions from the novel for the video clips.
StoryGraphs: Visualizing Character Interactions as a Timeline
We present a novel way to automatically summarize and represent the storyline of a TV episode by visualizing character interactions as a chart.
Semi-supervised Learning with Constraints for Person Identification in Multimedia Data
We address the problem of person identification in TV series.
