Search Results for author:
Found 13 papers, 8 papers with code
SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers
Leveraging the breadth of expertise and ability of multimodal large language models (MLLMs) to understand figures, we employ automatic and manual curation to create the dataset.
Jack of All Tasks Master of Many: Designing General-Purpose Coarse-to-Fine Vision-Language Model
In this work we introduce VistaLLM a powerful visual system that addresses coarse- and fine grained VL tasks over single and multiple input images using a unified framework.
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
In this work, we introduce VistaLLM, a powerful visual system that addresses coarse- and fine-grained VL tasks over single and multiple input images using a unified framework.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
UniVTG: Towards Unified Video-Language Temporal Grounding
Most methods in this direction develop taskspecific models that are trained with type-specific labels, such as moment retrieval (time interval) and highlight detection (worthiness curve), which limits their abilities to generalize to various VTG tasks and labels.
Ranked #4 on
Natural Language Moment Retrieval
on TACoS
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone
Video-language pre-training (VLP) has become increasingly important due to its ability to generalize to various vision and language tasks.
VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment
Extensive experiments on a wide range of vision- and vision-language downstream tasks demonstrate the effectiveness of VoLTA on fine-grained applications without compromising the coarse-grained downstream performance, often outperforming methods using significantly more caption and box annotations.
Where in the World is this Image? Transformer-based Geo-localization in the Wild
Predicting the geographic location (geo-localization) from a single ground-level RGB image taken anywhere in the world is a very challenging problem.
Ranked #3 on
Photo geolocation estimation
on Im2GPS3k
Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection
Multimodal learning is an emerging yet challenging research area.
Detecting Harmful Memes and Their Targets
In this work, we propose two novel problem formulations: detecting harmful memes and the social entities that these harmful memes target.
MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets
We focus on two tasks: (i)detecting harmful memes, and (ii)identifying the social entities they target.
See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
However, existing methods use short videos as the visual modality and short summary as the ground-truth, therefore, perform poorly on lengthy videos and long ground-truth summary.
Exercise? I thought you said 'Extra Fries': Leveraging Sentence Demarcations and Multi-hop Attention for Meme Affect Analysis
We evaluate MHA-MEME on the 'Memotion Analysis' dataset for all three sub-tasks - sentiment classification, affect classification, and affect class quantification.
