Search Results for author:
Found 43 papers, 34 papers with code
Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels
Thereby we demonstrate that 2D vision language models such as CLIP can be used to complement 3D representation learning to improve classification performance without the need for expensive class annotations.
Language Guided Domain Generalized Medical Image Segmentation
Incorporating text features alongside visual features is a potential solution to enhance the model's understanding of the data, as it goes beyond pixel-level information to provide valuable context.
Composed Video Retrieval via Enriched Context and Discriminative Embeddings
Composed video retrieval (CoVR) is a challenging problem in computer vision which has recently highlighted the integration of modification text with visual queries for more sophisticated video search in large databases.
VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understanding
In contrast, this paper introduces a Video Understanding and Reasoning Framework (VURF) based on the reasoning power of LLMs.
Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning
Self-supervised representation learning has been highly promising for histopathology image analysis with numerous approaches leveraging their patient-slide-patch hierarchy to learn better representations.
Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery
Recent advances in unsupervised learning have demonstrated the ability of large vision models to achieve promising results on downstream tasks by pre-training on large amount of unlabelled data.
ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes
We produce various versions of standard vision datasets (ImageNet, COCO), incorporating either diverse and realistic backgrounds into the images or introducing color, texture, and adversarial changes in the background.
MedContext: Learning Contextual Cues for Efficient Volumetric Medical Segmentation
The proposed approach induces contextual knowledge in the network by learning to reconstruct the missing organ or parts of an organ in the output segmentation space.
Learning to Prompt with Text Only Supervision for Vision-Language Models
While effective, most of these works require labeled data which is not practical, and often struggle to generalize towards new datasets due to over-fitting on the source data.
Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding
Our contributions include a novel spatio-temporal video grounding model, surpassing state-of-the-art results in closed-set evaluations on multiple datasets and demonstrating superior performance in open-vocabulary scenarios.
GeoChat: Grounded Large Vision-Language Model for Remote Sensing
Furthermore, the lack of domain-specific multimodal instruction following data as well as strong backbone models for RS make it hard for the models to align their behavior with user queries.
Enhancing Novel Object Detection via Cooperative Foundational Models
We present a novel approach to transform existing closed-set detectors into open-set detectors.
Ranked #1 on
Novel Object Detection
on LVIS v1.0 val
Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization
The promising zero-shot generalization of vision-language models such as CLIP has led to their adoption using prompt learning for numerous downstream tasks.
Videoprompter: an ensemble of foundational models for zero-shot video understanding
Consistent improvements across multiple benchmarks and with various VLMs demonstrate the effectiveness of our proposed framework.
Ranked #2 on
Video-Text Retrieval
on Test-of-Time
LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts
Diffusion-based generative models have significantly advanced text-to-image generation but encounter challenges when processing lengthy and intricate text prompts describing complex scenes with multiple objects.
FLIP: Cross-domain Face Anti-spoofing with Language Guidance
Specifically, we show that aligning the image representation with an ensemble of class descriptions (based on natural language semantics) improves FAS generalizability in low-data regimes.
Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment
To address this challenge, we propose the Self Structural Semantic Alignment (S^3A) framework, which extracts the structural semantic information from unlabeled data while simultaneously self-learning.
Foundational Models Defining a New Era in Vision: A Survey and Outlook
Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world.
Frequency Domain Adversarial Training for Robust Volumetric Medical Segmentation
While recent advances in deep learning have improved the performance of volumetric medical image segmentation models, these models cannot be deployed for real-world applications immediately due to their vulnerability to adversarial attacks.
Video-FocalNets: Spatio-Temporal Focal Modulation for Video Action Recognition
Video transformer designs are based on self-attention that can model global context at a high computational cost.
Ranked #1 on
Action Recognition
on Diving-48
Self-regulating Prompts: Foundational Model Adaptation without Forgetting
To the best of our knowledge, this is the first regularization framework for prompt learning that avoids overfitting by jointly attending to pre-trained model features, the training trajectory during prompting, and the textual diversity.
Ranked #2 on
Prompt Engineering
on ImageNet V2
CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via Adversarial Latent Search
We propose a novel two-step approach for facial privacy protection that relies on finding adversarial latent codes in the low-dimensional manifold of a pretrained generative model.
Learnable Weight Initialization for Volumetric Medical Image Segmentation
Hybrid volumetric medical image segmentation models, combining the advantages of local convolution and global attention, have recently received considerable attention.
Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting
Through this prompting scheme, we can achieve state-of-the-art zero-shot performance on Kinetics-600, HMDB51 and UCF101 while remaining competitive in the supervised setting.
Boosting Adversarial Transferability using Dynamic Cues
Our temporal prompts are the result of a learnable transformation that allows optimizing for temporal gradients during an adversarial attack to fool the motion dynamics.
Guidance Through Surrogate: Towards a Generic Diagnostic Attack
Recently, different adversarial training defenses are proposed that not only maintain a high clean accuracy but also show significant robustness against popular and well studied adversarial attacks such as PGD.
PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery
The GNCD setting aims to categorize unlabeled training data coming from known and novel classes by leveraging the information of partially labeled known classes.
How to Train Vision Transformer on Small-scale Datasets?
However, in contrast to convolutional neural networks, Vision Transformer lacks inherent inductive biases.
Self-Distilled Vision Transformer for Domain Generalization
Similar to CNNs, ViTs also struggle in out-of-distribution scenarios and the main culprit is overfitting to source domains.
Adversarial Pixel Restoration as a Pretext Task for Transferable Perturbations
Our training approach is based on a min-max scheme which reduces overfitting via an adversarial objective and thus optimizes for a more generalizable surrogate model.
Self-supervised Video Transformer
To the best of our knowledge, the proposed approach is the first to alleviate the dependency on negative samples or dedicated memory banks in Self-supervised Video Transformer (SVT).
Ranked #55 on
Action Recognition
on UCF101
On Improving Adversarial Transferability of Vision Transformers
(ii) Token Refinement: We then propose to refine the tokens to further enhance the discriminative capacity at each block of ViT.
Intriguing Properties of Vision Transformers
We show and analyze the following intriguing properties of ViT: (a) Transformers are highly robust to severe occlusions, perturbations and domain shifts, e. g., retain as high as 60% top-1 accuracy on ImageNet even after randomly occluding 80% of the image content.
Rich Semantics Improve Few-shot Learning
Human learning benefits from multi-modal inputs that often appear as rich semantics (e. g., description of an object's attributes while learning about it).
Ranked #1 on
Few-Shot Image Classification
on Oxford 102 Flower
(using extra training data)
On Generating Transferable Targeted Perturbations
To this end, we propose a new objective function that not only aligns the global distributions of source and target images, but also matches the local neighbourhood structure between the two domains.
Orthogonal Projection Loss
The CE loss encourages features of a class to have a higher projection score on the true class-vector compared to the negative classes.
Transformers in Vision: A Survey
Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems.
Stylized Adversarial Defense
In contrast to existing adversarial training methods that only use class-boundary information (e. g., using a cross-entropy loss), we propose to exploit additional information from the feature space to craft stronger adversaries that are in turn used to learn a robust model.
A Self-supervised Approach for Adversarial Robustness
Adversarial examples can cause catastrophic mistakes in Deep Neural Network (DNNs) based vision systems e. g., for classification, segmentation and object detection.
Cross-Domain Transferability of Adversarial Perturbations
To this end, we propose a framework capable of launching highly transferable attacks that crafts adversarial patterns to mislead networks trained on wholly different domains.
Task-generalizable Adversarial Attack based on Perceptual Metric
Deep neural networks (DNNs) can be easily fooled by adding human imperceptible perturbations to the images.
Local Gradients Smoothing: Defense against localized adversarial attacks
Deep neural networks (DNNs) have shown vulnerability to adversarial attacks, i. e., carefully perturbed inputs designed to mislead the network at inference time.
Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Survey
With the availability of low-cost and compact 2. 5/3D visual sensing devices, computer vision community is experiencing a growing interest in visual scene understanding of indoor environments.
