Search Results for author:
Found 29 papers, 6 papers with code
A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrieval
To provide a more thorough evaluation of the capabilities of long video retrieval systems, we propose a pipeline that leverages state-of-the-art large language models to carefully generate a diverse set of synthetic captions for long videos.
DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback
The critique NLF identifies the strengths and weaknesses of the responses and is used to align the LVLMs with human preferences.
Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning
Paraphrasing of offensive content is a better alternative to content removal and helps improve civility in a communication environment.
SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments
We present SayNav, a new approach that leverages human knowledge from Large Language Models (LLMs) for efficient generalization to complex navigation tasks in unknown large-scale environments.
Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models
Based on this pipeline and the existing coarse-grained annotated dataset, we build the CURE benchmark to measure both the zero-shot reasoning performance and consistency of VLMs.
TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models
We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversion using Joint Optimization).
Multilingual Content Moderation: A Case Study on Reddit
Content moderation is the process of flagging content based on pre-defined platform rules.
Dual-Key Multimodal Backdoors for Visual Question Answering
This is challenging for the attacker as the detector can distort or ignore the visual trigger entirely, which leads to models where backdoors are over-reliant on the language trigger.
Challenges in Procedural Multimodal Machine Comprehension:A Novel Way To Benchmark
We also observe a drop in performance across all the models when testing on RecipeQA and proposed Meta-RecipeQA (e. g. 83. 6% versus 67. 1% for HTRN), which shows that the proposed dataset is relatively less biased.
Towards Solving Multimodal Comprehension
We then evaluate M3C using a textual cloze style question-answering task and highlight an inherent bias in the question answer generation method from [35] that enables a naive baseline to cheat by learning from only answer choices.
MISA: Online Defense of Trojaned Models using Misattributions
Recent studies have shown that neural networks are vulnerable to Trojan attacks, where a network is trained to respond to specially crafted trigger patterns in the inputs in specific and potentially malicious ways.
Detecting Trojaned DNNs Using Counterfactual Attributions
We target the problem of detecting Trojans or backdoors in DNNs.
Zero-Shot Learning with Knowledge Enhanced Visual Semantic Embeddings
We improve zero-shot learning (ZSL) by incorporating common-sense knowledge in DNNs.
RGB2LIDAR: Towards Solving Large-Scale Cross-Modal Visual Localization
To enable large-scale evaluation, we introduce a new dataset containing over 550K pairs (covering 143 km^2 area) of RGB and aerial LIDAR depth images.
Deep Adaptive Semantic Logic (DASL): Compiling Declarative Knowledge into Deep Neural Networks
We introduce Deep Adaptive Semantic Logic (DASL), a novel framework for automating the generation of deep neural networks that incorporates user-provided formal knowledge to improve learning from data.
Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation
For instance, if a model answers "red" to "What color is the balloon?
FoodX-251: A Dataset for Fine-grained Food Classification
Food classification is a challenging problem due to the large number of categories, high visual similarity between different foods, as well as the lack of datasets for training state-of-the-art deep models.
Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks
We also show that the user embeddings learned within our joint multimodal embedding model are better at predicting user interests compared to those learned with unimodal content on Instagram data.
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts
Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image.
Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment
We propose a novel end-to-end model that uses caption-to-image retrieval as a `downstream' task to guide the process of phrase localization.
Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization
Furthermore, we present an extensive study demonstrating the contribution of each component of our model, showing $8$--$15\%$ and $4\%$ improvement from adding semantic information and our proposed attention module.
Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
We show that our model outperforms other baselines on the benchmark Ad dataset and also show qualitative results to highlight the advantages of using multihop co-attention.
Zero-Shot Object Detection
We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training.
Combining Weakly and Webly Supervised Learning for Classifying Food Images
Food classification from images is a fine-grained classification problem.
AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks for Human Action Recognition in Videos
We propose a novel method for temporally pooling frames in a video for the task of human action recognition.
Discriminatively Trained Latent Ordinal Model for Video Classification
We study the problem of video classification for facial analysis and human action recognition.
LOMo: Latent Ordinal Model for Facial Analysis in Videos
We study the problem of facial analysis in videos.
Deep Active Object Recognition by Joint Label and Action Prediction
The results of experiments suggest that the proposed model equipped with Dirichlet state encoding is superior in performance, and selects images that lead to better training and higher accuracy of label prediction at test time.
Pseudo vs. True Defect Classification in Printed Circuits Boards using Wavelet Features
In recent years, Printed Circuit Boards (PCB) have become the backbone of a large number of consumer electronic devices leading to a surge in their production.
