Search Results for author:
Found 42 papers, 6 papers with code
FoodX-251: A Dataset for Fine-grained Food Classification
Food classification is a challenging problem due to the large number of categories, high visual similarity between different foods, as well as the lack of datasets for training state-of-the-art deep models.
Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models
Based on this pipeline and the existing coarse-grained annotated dataset, we build the CURE benchmark to measure both the zero-shot reasoning performance and consistency of VLMs.
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts
Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image.
Modular Adaptation for Cross-Domain Few-Shot Learning
Adapting pre-trained representations has become the go-to recipe for learning new downstream tasks with limited examples.
Multilingual Content Moderation: A Case Study on Reddit
Content moderation is the process of flagging content based on pre-defined platform rules.
Probing Conceptual Understanding of Large Visual-Language Models
In recent years large visual-language (V+L) models have achieved great success in various downstream tasks.
Zero-Shot Object Detection
We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training.
Combining Weakly and Webly Supervised Learning for Classifying Food Images
Food classification from images is a fine-grained classification problem.
Human Social Interaction Modeling Using Temporal Deep Networks
We present a novel approach to computational modeling of social interactions based on modeling of essential social interaction predicates (ESIPs) such as joint attention and entrainment.
Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
We show that our model outperforms other baselines on the benchmark Ad dataset and also show qualitative results to highlight the advantages of using multihop co-attention.
Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation
We show results on CAD120 (which provides pre-computed node features and edge weights for fair performance comparison across algorithms) as well as a more complex real-world activity dataset, Charades.
Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment
We propose a novel end-to-end model that uses caption-to-image retrieval as a `downstream' task to guide the process of phrase localization.
Can You Explain That? Lucid Explanations Help Human-AI Collaborative Image Retrieval
Our experiments, therefore, demonstrate that ExAG is an effective means to evaluate the efficacy of AI-generated explanations on a human-AI collaborative task.
Data-Efficient Mutual Information Neural Estimator
We demonstrate the effectiveness of our estimators on synthetic benchmarks and a real world fMRI data, with application of inter-subject correlation analysis.
Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks
We also show that the user embeddings learned within our joint multimodal embedding model are better at predicting user interests compared to those learned with unimodal content on Instagram data.
Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation
For instance, if a model answers "red" to "What color is the balloon?
Progressive Growing of Neural ODEs
Neural Ordinary Differential Equations (NODEs) have proven to be a powerful modeling tool for approximating (interpolation) and forecasting (extrapolation) irregularly sampled time series data.
Deep Adaptive Semantic Logic (DASL): Compiling Declarative Knowledge into Deep Neural Networks
We introduce Deep Adaptive Semantic Logic (DASL), a novel framework for automating the generation of deep neural networks that incorporates user-provided formal knowledge to improve learning from data.
Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition, and Selective Transfer
We propose a wake-sleep cycle of alternating task learning and knowledge consolidation for learning in our framework, and instantiate it for lifelong supervised learning and lifelong RL.
Hybrid Consistency Training with Prototype Adaptation for Few-Shot Learning
Few-Shot Learning (FSL) aims to improve a model's generalization capability in low data regimes.
Zero-Shot Learning with Knowledge Enhanced Visual Semantic Embeddings
We improve zero-shot learning (ZSL) by incorporating common-sense knowledge in DNNs.
Detecting Trojaned DNNs Using Counterfactual Attributions
We target the problem of detecting Trojans or backdoors in DNNs.
Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models
Hence, we propose Error Maps that clarify the error by highlighting image regions where the model is prone to err.
Confidence Calibration for Domain Generalization under Covariate Shift
Existing calibration algorithms address the problem of covariate shift via unsupervised domain adaptation.
Towards Solving Multimodal Comprehension
We then evaluate M3C using a textual cloze style question-answering task and highlight an inherent bias in the question answer generation method from [35] that enables a naive baseline to cheat by learning from only answer choices.
Comprehension Based Question Answering using Bloom's Taxonomy
Current pre-trained language models have lots of knowledge, but a more limited ability to use that knowledge.
Challenges in Procedural Multimodal Machine Comprehension:A Novel Way To Benchmark
We also observe a drop in performance across all the models when testing on RecipeQA and proposed Meta-RecipeQA (e. g. 83. 6% versus 67. 1% for HTRN), which shows that the proposed dataset is relatively less biased.
Comprehension Based Question Answering using Bloom’s Taxonomy
Current pre-trained language models have lots of knowledge, but a more limited ability to use that knowledge.
A Data-Efficient Mutual Information Neural Estimator for Statistical Dependency Testing
Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications.
Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition and Selective Transfer
We propose a wake-sleep cycle of alternating task learning and knowledge consolidation for learning in our framework, and instantiate it for lifelong supervised learning and lifelong RL.
Detecting out-of-context objects using contextual cues
GCRN consists of two separate graphs to predict object labels based on the contextual cues in the image: 1) a representation graph to learn object features based on the neighboring objects and 2) a context graph to explicitly capture contextual cues from the neighboring objects.
Ranked #1 on
Anomaly Detection
on COCO-OOC
Towards Understanding Confusion and Affective States Under Communication Failures in Voice-Based Human-Machine Interaction
We present a series of two studies conducted to understand user's affective states during voice-based human-machine interactions.
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2
We present a version of GR for LRL that satisfies two desiderata: (a) Introspective density modelling of the latent representations of policies learned using deep RL, and (b) Model-free end-to-end learning.
Unpacking Large Language Models with Conceptual Consistency
The success of Large Language Models (LLMs) indicates they are increasingly able to answer queries like these accurately, but that ability does not necessarily imply a general understanding of concepts relevant to the anchor query.
System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games
In this paper, we introduce the Lifelong Reinforcement Learning Components Framework (L2RLCF), which standardizes L2RL systems and assimilates different continual learning components (each addressing different aspects of the lifelong learning problem) into a unified system.
Class Prototypes Based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos
We learn a class prototype for each class and a loss function is employed to minimize the distances between a class prototype and the samples from the class.
TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models
We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversion using Joint Optimization).
Confidence Calibration for Systems with Cascaded Predictive Modules
Existing conformal prediction algorithms estimate prediction intervals at target confidence levels to characterize the performance of a regression model on new test samples.
Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning
Paraphrasing of offensive content is a better alternative to content removal and helps improve civility in a communication environment.
DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback
The critique NLF identifies the strengths and weaknesses of the responses and is used to align the LVLMs with human preferences.
A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrieval
To provide a more thorough evaluation of the capabilities of long video retrieval systems, we propose a pipeline that leverages state-of-the-art large language models to carefully generate a diverse set of synthetic captions for long videos.
BloomVQA: Assessing Hierarchical Multi-modal Comprehension
We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks.
