Search Results for author:
Found 112 papers, 48 papers with code
To Find Waldo You Need Contextual Cues: Debiasing Who’s Waldo
We find that the original Who’s Waldo dataset compiled for this task contains a large number of biased samples that are solvable simply by heuristic methods; for instance, in many cases the first name in the sentence corresponds to the largest bounding box, or the sequence of names in the sentence corresponds to an exact left-to-right order in the image.
Steering Rectified Flow Models in the Vector Field for Controlled Image Generation
In this work, we first develop a theoretical and empirical understanding of the vector field dynamics of RFMs in efficiently guiding the denoising trajectory.
Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model
Despite advancements in text-to-image models, generating images that precisely align with textual descriptions remains challenging due to misalignment in training data.
TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP.
Help Me Identify: Is an LLM+VQA System All We Need to Identify Visual Concepts?
An ability to learn about new objects from a small amount of visual data and produce convincing linguistic justification about the presence/absence of certain concepts (that collectively compose the object) in novel scenarios is an important characteristic of human cognition.
ActionCOMET: A Zero-shot Approach to Learn Image-specific Commonsense Concepts about Actions
We present baseline results of ActionCOMET over the collected dataset and compare them with the performance of the best existing VQA approaches.
VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Tasks
We show that this benchmark is quite challenging for existing large-scale vision-language models and encourage development of systems that possess robust visuo-linguistic reasoning capabilities.
TROPE: TRaining-Free Object-Part Enhancement for Seamlessly Improving Fine-Grained Zero-Shot Image Captioning
Zero-shot inference, where pre-trained models perform tasks without specific training data, is an exciting emergent ability of large models like CLIP.
Latent Space Energy-based Neural ODEs
This family of models generates each data point in the time series by a neural emission model, which is a non-linear transformation of a latent state vector.
Roundabout Dilemma Zone Data Mining and Forecasting with Trajectory Prediction and Graph Neural Networks
This research represents an advancement in roundabout DZ data mining and forecasting, contributing to the assurance of intersection safety in the era of autonomous vehicles.
Recent Event Camera Innovations: A Survey
Event-based vision, inspired by the human visual system, offers transformative capabilities such as low latency, high dynamic range, and reduced power consumption.
SynTraC: A Synthetic Dataset for Traffic Signal Control from Traffic Monitoring Cameras
This paper introduces SynTraC, the first public image-based traffic signal control dataset, aimed at bridging the gap between simulated environments and real-world traffic management challenges.
Deep Geometric Moments Promote Shape Consistency in Text-to-3D Generation
To address the data scarcity associated with 3D assets, 2D-lifting techniques such as Score Distillation Sampling (SDS) have become a widely adopted practice in text-to-3D generation pipelines.
REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models
Text-to-Image (T2I) and multimodal large language models (MLLMs) have been adopted in solutions for several computer vision and multimodal learning tasks.
R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Model
In the evolving landscape of text-to-image (T2I) diffusion models, the remarkable capability to generate high-quality images from textual descriptions faces challenges with the potential misuse of reproducing sensitive content.
Grounding Stylistic Domain Generalization with Quantitative Domain Shift Measures and Synthetic Scene Images
We first introduce two new quantitative measures ICV and IDD to describe domain shifts in terms of consistency of classes within one domain and similarity between two stylistic domains.
`Eyes of a Hawk and Ears of a Fox': Part Prototype Network for Generalized Zero-Shot Learning
Current approaches in Generalized Zero-Shot Learning (GZSL) are built upon base models which consider only a single class attribute vector representation over the entire image.
On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation
Recent advances in monocular depth estimation have been made by incorporating natural language as additional guidance.
SEVD: Synthetic Event-based Vision Dataset for Ego and Fixed Traffic Perception
In response to this gap, we present SEVD, a first-of-its-kind multi-view ego, and fixed perception synthetic event-based dataset using multiple dynamic vision sensors within the CARLA simulator.
Getting it Right: Improving Spatial Consistency in Text-to-Image Models
One of the key shortcomings in current text-to-image (T2I) models is their inability to consistently generate images which faithfully follow the spatial relationships specified in the text prompt.
eTraM: Event-based Traffic Monitoring Dataset
Event cameras, with their high temporal and dynamic range and minimal memory usage, have found applications in various fields.
Learning Decomposable and Debiased Representations via Attribute-Centric Information Bottlenecks
Biased attributes, spuriously correlated with target labels in a dataset, can problematically lead to neural networks that learn improper shortcuts for classifications and limit their capabilities for out-of-distribution (OOD) generalization.
Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts
Benchmarks of the multilingual capabilities of text-to-image (T2I) models compare generated images prompted in a test language to an expected image distribution over a concept set.
$λ$-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space
While LDMs offer distinct advantages, P-T2I methods' reliance on the latent space of these diffusion models significantly escalates resource demands, leading to inconsistent results and necessitating numerous iterations for a single desired image.
Segment Anything Model Can Not Segment Anything: Assessing AI Foundation Model's Generalizability in Permafrost Mapping
To evaluate the performance of large AI vision models, especially Meta's Segment Anything Model (SAM), we implemented different instance segmentation pipelines that minimize the changes to SAM to leverage its power as a foundation model.
Open-TI: Open Traffic Intelligence with Augmented Language Model
Transportation has greatly benefited the cities' development in the modern civilization process.
ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations
The T2I prior model alone adds a billion parameters compared to the Latent Diffusion Models, which increases the computational and high-quality data requirements.
SKoPe3D: A Synthetic Dataset for Vehicle Keypoint Perception in 3D from Traffic Monitoring Cameras
Intelligent transportation systems (ITS) have revolutionized modern road infrastructure, providing essential functionalities such as traffic monitoring, road safety assessment, congestion reduction, and law enforcement.
Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Grounding
Object proposal generation serves as a standard pre-processing step in Vision-Language (VL) tasks (image captioning, visual question answering, etc.).
Adversarial Bayesian Augmentation for Single-Source Domain Generalization
Generalizing to unseen image domains is a challenging problem primarily due to the lack of diverse training data, inaccessible target data, and the large domain shift that may exist in many real-world settings.
Ranked #5 on
Photo to Rest Generalization
on PACS
ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models
To quantify the ability of T2I models in learning and synthesizing novel visual concepts (a. k. a.
WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models
This paper introduces a novel approach to model fingerprinting that assigns responsibility for the generated images, thereby serving as a potential countermeasure to model misuse.
End-to-end Knowledge Retrieval with Multi-modal Queries
We investigate knowledge retrieval with multi-modal queries, i. e. queries containing information split across image and text inputs, a challenging task that differs from previous work on cross-modal retrieval.
CAROM Air -- Vehicle Localization and Traffic Scene Reconstruction from Aerial Videos
Road traffic scene reconstruction from videos has been desirable by road safety regulators, city planners, researchers, and autonomous driving technology developers.
Attributing Image Generative Models using Latent Fingerprints
This paper investigates the use of latent semantic dimensions as fingerprints, from where we can analyze the effects of design variables, including the choice of fingerprinting dimensions, strength, and capacity, on the accuracy-quality tradeoff.
Mole Recruitment: Poisoning of Image Classifiers via Selective Batch Sampling
In this work, we present a data poisoning attack that confounds machine learning models without any manipulation of the image or label.
Benchmarking Spatial Relationships in Text-to-Image Generation
We investigate the ability of T2I models to generate correct spatial relationships among objects and present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image.
Learning Action-Effect Dynamics from Pairs of Scene-graphs
'Actions' play a vital role in how humans interact with the world.
Learning Action-Effect Dynamics for Hypothetical Vision-Language Reasoning Task
'Actions' play a vital role in how humans interact with the world.
CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering
Videos often capture objects, their visible properties, their motion, and the interactions between different objects.
Ranked #1 on
Counterfactual Planning
on CRIPP-VQA
Reasoning about Actions over Visual and Linguistic Modalities: A Survey
'Actions' play a vital role in how humans interact with the world and enable them to achieve desired goals.
Formalizing and Evaluating Requirements of Perception Systems for Automated Vehicles using Spatio-Temporal Perception Logic
Automated vehicles (AV) heavily depend on robust perception systems.
Improving Diversity with Adversarially Learned Transformations for Domain Generalization
To be successful in single source domain generalization, maximizing diversity of synthesized domains has emerged as one of the most effective strategies.
Tragedy Plus Time: Capturing Unintended Human Activities from Weakly-labeled Videos
In videos that contain actions performed unintentionally, agents do not achieve their desired goals.
SSR-GNNs: Stroke-based Sketch Representation with Graph Neural Networks
This paper follows cognitive studies to investigate a graph representation for sketches, where the information of strokes, i. e., parts of a sketch, are encoded on vertices and information of inter-stroke on edges.
To Find Waldo You Need Contextual Cues: Debiasing Who's Waldo
We find that the original Who's Waldo dataset compiled for this task contains a large number of biased samples that are solvable simply by heuristic methods; for instance, in many cases the first name in the sentence corresponds to the largest bounding box, or the sequence of names in the sentence corresponds to an exact left-to-right order in the image.
Injecting Semantic Concepts into End-to-End Image Captioning
In this paper, we are concerned with a better-performing detector-free image captioning model, and propose a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features.
Semantically Distributed Robust Optimization for Vision-and-Language Inference
Analysis of vision-and-language models has revealed their brittleness under linguistic phenomena such as paraphrasing, negation, textual entailment, and word substitutions with synonyms or antonyms.
Targeted Attack on Deep RL-based Autonomous Driving with Learned Visual Patterns
This paper investigates the feasibility of targeted attacks through visually learned patterns placed on physical objects in the environment, a threat model that combines the practicality and effectiveness of the existing ones.
Weakly Supervised Relative Spatial Reasoning for Visual Question Answering
In this work, we evaluate the faithfulness of V\&L models to such geometric understanding, by formulating the prediction of pair-wise relative locations of objects as a classification as well as a regression task.
SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis
The open-ended nature of visual captioning makes it a challenging area for evaluation.
CLEVR\_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images
Most existing research on visual question answering (VQA) is limited to information explicitly present in an image or a video.
CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images
Most existing research on visual question answering (VQA) is limited to information explicitly present in an image or a video.
Compressing Visual-linguistic Model via Knowledge Distillation
In this paper, we study knowledge distillation (KD) to effectively compress a transformer-based large VL model into a small VL model.
Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph
We present a novel two-layer hierarchical reinforcement learning approach equipped with a Goals Relational Graph (GRG) for tackling the partially observable goal-driven task, such as goal-driven visual navigation.
Hierarchical Reinforcement Learning
Reinforcement Learning (RL)
+1
SEED: Self-supervised Distillation For Visual Representation
This paper is concerned with self-supervised learning for small models.
WeaQA: Weak Supervision via Captions for Visual Question Answering
Methodologies for training visual question answering (VQA) models assume the availability of datasets with human-annotated \textit{Image-Question-Answer} (I-Q-A) triplets.
Attribute-Guided Adversarial Training for Robustness to Natural Perturbations
While this deviation may not be exactly known, its broad characterization is specified a priori, in terms of attributes.
Decentralized Attribution of Generative Models
Growing applications of generative models have led to new threats such as malicious personation and digital copyright infringement.
Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
Despite the significant success at enabling robots with autonomous behaviors makes deep reinforcement learning a promising approach for robotic object search task, the deep reinforcement learning approach severely suffers from the nature sparse reward setting of the task.
MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
In this paper, we present MUTANT, a training paradigm that exposes the model to perceptually similar, yet semantically distinct mutations of the input, to improve OOD generalization, such as the VQA-CP challenge.
Low to High Dimensional Modality Hallucination using Aggregated Fields of View
While hallucinating data from a modality with richer information, e. g., RGB to depth, has been researched extensively, we investigate the more challenging low-to-high modality hallucination with interesting use cases in robotics and autonomous systems.
Weak Supervision and Referring Attention for Temporal-Textual Association Learning
The referring attention is our designed mechanism acting as a scoring function for grounding the given queries over frames temporally.
Resisting Crowd Occlusion and Hard Negatives for Pedestrian Detection in the Wild
Pedestrian detection has been heavily studied in the last decade due to its wide application.
ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language
Person search by natural language aims at retrieving a specific person in a large-scale image pool that matches the given textual descriptions.
Ranked #20 on
Text based Person Retrieval
on CUHK-PEDES
Learning hierarchical behavior and motion planning for autonomous driving
To improve the tactical decision-making for learning-based driving solution, we introduce hierarchical behavior and motion planning (HBMP) to explicitly model the behavior in learning-based solution.
Visuo-Linguistic Question Answering (VLQA) Challenge
Understanding images and text together is an important aspect of cognition and building advanced Artificial Intelligence (AI) systems.
memeBot: Towards Automatic Image Meme Generation
For a given input sentence, an image meme is generated by combining a meme template image and a text caption where the meme template image is selected from a set of popular candidates using a selection module, and the meme caption is generated by an encoder-decoder model.
Enabling Incremental Knowledge Transfer for Object Detection at the Edge
However, we need a limited knowledge of the observed environment at inference time which can be learned using a shallow neural network (SHNN).
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning
In videos that involve active agents such as humans, the agent's actions can bring about myriad changes in the scene.
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN)
Visual Indoor Navigation (VIN) task has drawn increasing attention from the data-driven machine learning communities especially with the recently reported success from learning-based methods.
VQA-LOL: Visual Question Answering under the Lens of Logic
We propose our {Lens of Logic (LOL)} model which uses question-attention and logic-attention to understand logical connectives in the question, and a novel Fr\'echet-Compatibility Loss, which ensures that the answers of the component questions and the composed question are consistent with the inferred logical operation.
Good, Better, Best: Textual Distractors Generation for Multiple-Choice Visual Question Answering via Reinforcement Learning
Multiple-choice VQA has drawn increasing attention from researchers and end-users recently.
ROS-HPL: Robotic Object Search with Hierarchical Policy Learning and Intrinsic-Extrinsic Modeling
Despite significant progress in Robotic Object Search (ROS) over the recent years with deep reinforcement learning based approaches, the sparsity issue in reward setting as well as the lack of interpretability of the previous ROS approaches leave much to be desired.
A Novel Design of Adaptive and Hierarchical Convolutional Neural Networks using Partial Reconfiguration on FPGA
On the other hand, for a large chunk of recognition challenges, a system can classify images correctly using simple models or so-called shallow networks.
Integrating Knowledge and Reasoning in Image Understanding
Deep learning based data-driven approaches have been successfully applied in various image understanding applications ranging from object recognition, semantic segmentation to visual question answering.
Blocksworld Revisited: Learning and Reasoning to Generate Event-Sequences from Image Pairs
The process of identifying changes or transformations in a scene along with the ability of reasoning about their causes and effects, is a key aspect of intelligence.
Fluorescence Image Histology Pattern Transformation using Image Style Transfer
To improve the diagnostic quality of CLE, we used a micrograph of an H&E slide from a glioma tumor biopsy and image style transfer, a neural network method for integrating the content and style of two images.
Active Adversarial Evader Tracking with a Probabilistic Pursuer under the Pursuit-Evasion Game Framework
To validate the efficiency of the framework, we conduct several experiments in simulation by using Gazebo and evaluate the success rate of tracking an evader in various environments with different pursuer to evader speed ratios.
Modularized Textual Grounding for Counterfactual Resilience
Computer Vision applications often require a textual grounding module with precision, interpretability, and resilience to counterfactual inputs/queries.
TKD: Temporal Knowledge Distillation for Active Perception
Deep neural networks based methods have been proved to achieve outstanding performance on object detection and classification tasks.
Image Decomposition and Classification through a Generative Model
We demonstrate in this paper that a generative model can be designed to perform classification tasks under challenging settings, including adversarial attacks and input distribution shifts.
Augmenting Model Robustness with Transformation-Invariant Attacks
The vulnerability of neural networks under adversarial attacks has raised serious concerns and motivated extensive research.
How Shall I Drive? Interaction Modeling and Motion Planning towards Empathetic and Socially-Graceful Driving
While intelligence of autonomous vehicles (AVs) has significantly advanced in recent years, accidents involving AVs suggest that these autonomous systems lack gracefulness in driving when interacting with human drivers.
Robotics Computer Science and Game Theory
Spatial Knowledge Distillation to aid Visual Reasoning
We propose a framework that combines recent advances in knowledge distillation (teacher-student framework), relational reasoning and probabilistic logical languages to incorporate such knowledge in existing neural networks for the task of Visual Question Answering.
GAPLE: Generalizable Approaching Policy LEarning for Robotic Object Searching in Indoor Environment
We study the problem of learning a generalizable action policy for an intelligent agent to actively approach an object of interest in an indoor environment solely from its visual inputs.
Active Object Perceiver: Recognition-guided Policy Learning for Object Searching on Mobile Robots
We study the problem of learning a navigation policy for a robot to actively search for an object of interest in an indoor environment solely from its visual inputs.
Interpretable Partitioned Embedding for Customized Fashion Outfit Composition
In the supervised attributes module, multiple attributes labels are adopted to ensure that different parts of the overall embedding correspond to different attributes.
Weakly Supervised Attention Learning for Textual Phrases Grounding
Grounding textual phrases in visual content is a meaningful yet challenging problem with various potential applications such as image-text inference or text-driven multimedia interaction.
Prospects for Theranostics in Neurosurgical Imaging: Empowering Confocal Laser Endomicroscopy Diagnostics via Deep Learning
We present an overview and discuss deep learning models for automatic detection of the diagnostic CLE images and discuss various training regimes and ensemble modeling effect on the power of deep learning predictive models.
Weakly-Supervised Learning-Based Feature Localization in Confocal Laser Endomicroscopy Glioma Images
To overcome this problem, we propose a Weakly-Supervised Learning (WSL)-based model for feature localization that trains on image-level annotations, and then localizes incidences of a class-of-interest in the test image.
Transductive Unbiased Embedding for Zero-Shot Learning
Most existing Zero-Shot Learning (ZSL) methods have the strong bias problem, in which instances of unseen (target) classes tend to be categorized as one of the seen (source) classes.
Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering
Here we adopt Visual Question Answering (VQA) as an example task, where a system is expected to answer a question in natural language about an image.
Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields
In this paper, we present a stroke controllable style transfer network that can achieve continuous and spatial stroke size control.
DeepSIC: Deep Semantic Image Compression
The same practice also enable the compressed code to carry the image semantic information during storage and transmission.
Improving utility of brain tumor confocal laser endomicroscopy: objective value assessment and diagnostic frame detection with convolutional neural networks
Examining all the hundreds or thousands of images from a single case to discriminate diagnostic images from nondiagnostic ones can be tedious.
TripletGAN: Training Generative Model with Triplet Loss
As an effective way of metric learning, triplet loss has been widely used in many deep learning tasks, including face recognition and person-ReID, leading to many states of the arts.
Convolutional Neural Networks: Ensemble Modeling, Fine-Tuning and Unsupervised Semantic Localization for Intraoperative CLE Images
While manual examination of thousands of nondiagnostic images during surgery would be impractical, this creates an opportunity for a model to select diagnostic images for the pathologists or surgeon's review.
On the Importance of Consistency in Training Deep Neural Networks
We conclude this paper with the construction of a novel contractive neural network.
Neural Style Transfer: A Review
We first propose a taxonomy of current algorithms in the field of NST.
Answering Image Riddles using Vision and Reasoning through Probabilistic Soft Logic
We compile a dataset of over 3k riddles where each riddle consists of 4 images and a groundtruth answer.
Fast Task-Specific Target Detection via Graph Based Constraints Representation and Checking
In this work, we present a fast target detection framework for real-world robotics applications.
Prediction of Manipulation Actions
In psychophysical experiments, we evaluated human observers' skills in predicting actions from video sequences of different length, depicting the hand movement in the preparation and execution of actions before and after contact with the object.
Reliable Attribute-Based Object Recognition Using High Predictive Value Classifiers
We consider the problem of object recognition in 3D using an ensemble of attribute-based classifiers.
Co-active Learning to Adapt Humanoid Movement for Manipulation
It is designed to adapt the original imitation trajectories, which are learned from demonstrations, to novel situations with various constraints.
LightNet: A Versatile, Standalone Matlab-based Environment for Deep Learning
LightNet is a lightweight, versatile and purely Matlab-based deep learning framework.
What Can I Do Around Here? Deep Functional Scene Understanding for Cognitive Robots
For robots that have the capability to interact with the physical environment through their end effectors, understanding the surrounding scenes is not merely a task of image classification or object recognition.
Neural Self Talk: Image Understanding via Continuous Questioning and Answering
In this paper we consider the problem of continuously discovering image contents by actively asking image based questions and subsequently answering the questions being asked.
Learning the Semantics of Manipulation Action
In this paper we present a formal computational framework for modeling manipulation actions.
From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge
Specifically, commonsense reasoning is applied on (a) detections obtained from existing perception methods on given images, (b) a "commonsense" knowledge base constructed using natural language processing of image annotations and (c) lexical ontological knowledge from resources such as WordNet.
Grasp Type Revisited: A Modern Perspective on a Classical Feature for Vision
The grasp type provides crucial information about human action.
Detection of Manipulation Action Consequences (MAC)
There is a small set of fundamental primitive action consequences that provides a systematic high-level classification of manipulation actions.
