Search Results for author:
Found 44 papers, 8 papers with code
Comprehension Based Question Answering using Bloom’s Taxonomy
Current pre-trained language models have lots of knowledge, but a more limited ability to use that knowledge.
Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification
Large Visual Language Models (LVLMs) struggle with hallucinations in visual instruction following task(s), limiting their trustworthiness and real-world applicability.
Empowering Interdisciplinary Insights with Dynamic Graph Embedding Trajectories
Our work not only addresses the ongoing challenges in visualizing and analyzing DTDG models but also establishes a foundational framework for future investigations into dynamic graph representation and analysis across various disciplines.
BloomVQA: Assessing Hierarchical Multi-modal Comprehension
We propose a novel VQA dataset, BloomVQA, to facilitate comprehensive evaluation of large vision-language models on comprehension tasks.
A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrieval
To provide a more thorough evaluation of the capabilities of long video retrieval systems, we propose a pipeline that leverages state-of-the-art large language models to carefully generate a diverse set of synthetic captions for long videos.
DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback
The critique NLF identifies the strengths and weaknesses of the responses and is used to align the LVLMs with human preferences.
Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning
Paraphrasing of offensive content is a better alternative to content removal and helps improve civility in a communication environment.
Confidence Calibration for Systems with Cascaded Predictive Modules
Existing conformal prediction algorithms estimate prediction intervals at target confidence levels to characterize the performance of a regression model on new test samples.
Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models
Based on this pipeline and the existing coarse-grained annotated dataset, we build the CURE benchmark to measure both the zero-shot reasoning performance and consistency of VLMs.
TIJO: Trigger Inversion with Joint Optimization for Defending Multimodal Backdoored Models
We present a Multimodal Backdoor Defense technique TIJO (Trigger Inversion using Joint Optimization).
Probing Conceptual Understanding of Large Visual-Language Models
In this work we focus on conceptual understanding of these large V+L models.
Multilingual Content Moderation: A Case Study on Reddit
Content moderation is the process of flagging content based on pre-defined platform rules.
Class Prototypes Based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos
We learn a class prototype for each class and a loss function is employed to minimize the distances between a class prototype and the samples from the class.
System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games
In this paper, we introduce the Lifelong Reinforcement Learning Components Framework (L2RLCF), which standardizes L2RL systems and assimilates different continual learning components (each addressing different aspects of the lifelong learning problem) into a unified system.
Unpacking Large Language Models with Conceptual Consistency
The success of Large Language Models (LLMs) indicates they are increasingly able to answer queries like these accurately, but that ability does not necessarily imply a general understanding of concepts relevant to the anchor query.
Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2
We present a version of GR for LRL that satisfies two desiderata: (a) Introspective density modelling of the latent representations of policies learned using deep RL, and (b) Model-free end-to-end learning.
Towards Understanding Confusion and Affective States Under Communication Failures in Voice-Based Human-Machine Interaction
We present a series of two studies conducted to understand user's affective states during voice-based human-machine interactions.
Detecting out-of-context objects using contextual cues
GCRN consists of two separate graphs to predict object labels based on the contextual cues in the image: 1) a representation graph to learn object features based on the neighboring objects and 2) a context graph to explicitly capture contextual cues from the neighboring objects.
Ranked #1 on
Anomaly Detection
on COCO-OOC
Challenges in Procedural Multimodal Machine Comprehension:A Novel Way To Benchmark
We also observe a drop in performance across all the models when testing on RecipeQA and proposed Meta-RecipeQA (e. g. 83. 6% versus 67. 1% for HTRN), which shows that the proposed dataset is relatively less biased.
Comprehension Based Question Answering using Bloom's Taxonomy
Current pre-trained language models have lots of knowledge, but a more limited ability to use that knowledge.
Towards Solving Multimodal Comprehension
We then evaluate M3C using a textual cloze style question-answering task and highlight an inherent bias in the question answer generation method from [35] that enables a naive baseline to cheat by learning from only answer choices.
Modular Adaptation for Cross-Domain Few-Shot Learning
Adapting pre-trained representations has become the go-to recipe for learning new downstream tasks with limited examples.
Confidence Calibration for Domain Generalization under Covariate Shift
Existing calibration algorithms address the problem of covariate shift via unsupervised domain adaptation.
Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models
Hence, we propose Error Maps that clarify the error by highlighting image regions where the model is prone to err.
Detecting Trojaned DNNs Using Counterfactual Attributions
We target the problem of detecting Trojans or backdoors in DNNs.
Zero-Shot Learning with Knowledge Enhanced Visual Semantic Embeddings
We improve zero-shot learning (ZSL) by incorporating common-sense knowledge in DNNs.
Hybrid Consistency Training with Prototype Adaptation for Few-Shot Learning
Few-Shot Learning (FSL) aims to improve a model's generalization capability in low data regimes.
Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition, and Selective Transfer
We propose a wake-sleep cycle of alternating task learning and knowledge consolidation for learning in our framework, and instantiate it for lifelong supervised learning and lifelong RL.
Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition and Selective Transfer
We propose a wake-sleep cycle of alternating task learning and knowledge consolidation for learning in our framework, and instantiate it for lifelong supervised learning and lifelong RL.
Deep Adaptive Semantic Logic (DASL): Compiling Declarative Knowledge into Deep Neural Networks
We introduce Deep Adaptive Semantic Logic (DASL), a novel framework for automating the generation of deep neural networks that incorporates user-provided formal knowledge to improve learning from data.
Progressive Growing of Neural ODEs
Neural Ordinary Differential Equations (NODEs) have proven to be a powerful modeling tool for approximating (interpolation) and forecasting (extrapolation) irregularly sampled time series data.
A Data-Efficient Mutual Information Neural Estimator for Statistical Dependency Testing
Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications.
Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generation
For instance, if a model answers "red" to "What color is the balloon?
FoodX-251: A Dataset for Fine-grained Food Classification
Food classification is a challenging problem due to the large number of categories, high visual similarity between different foods, as well as the lack of datasets for training state-of-the-art deep models.
Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks
We also show that the user embeddings learned within our joint multimodal embedding model are better at predicting user interests compared to those learned with unimodal content on Instagram data.
Data-Efficient Mutual Information Neural Estimator
We demonstrate the effectiveness of our estimators on synthetic benchmarks and a real world fMRI data, with application of inter-subject correlation analysis.
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts
Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image.
Can You Explain That? Lucid Explanations Help Human-AI Collaborative Image Retrieval
Our experiments, therefore, demonstrate that ExAG is an effective means to evaluate the efficacy of AI-generated explanations on a human-AI collaborative task.
Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignment
We propose a novel end-to-end model that uses caption-to-image retrieval as a `downstream' task to guide the process of phrase localization.
Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation
We show results on CAD120 (which provides pre-computed node features and edge weights for fair performance comparison across algorithms) as well as a more complex real-world activity dataset, Charades.
Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
We show that our model outperforms other baselines on the benchmark Ad dataset and also show qualitative results to highlight the advantages of using multihop co-attention.
Zero-Shot Object Detection
We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training.
Combining Weakly and Webly Supervised Learning for Classifying Food Images
Food classification from images is a fine-grained classification problem.
Human Social Interaction Modeling Using Temporal Deep Networks
We present a novel approach to computational modeling of social interactions based on modeling of essential social interaction predicates (ESIPs) such as joint attention and entrainment.
