Search Results for author:
Found 95 papers, 46 papers with code
Combining Label Propagation and Simple Models Out-performs Graph Neural Networks
Graph Neural Networks (GNNs) are the predominant technique for learning over graphs.
Node Classification on Non-Homophilic (Heterophilic) Graphs
Node Property Prediction
A Metric Learning Reality Check
Deep metric learning papers from the past four years have consistently claimed great advances in accuracy, often more than doubling the performance of decade-old methods.
PyTorch Metric Learning
Deep metric learning algorithms have a wide variety of applications, but implementing these algorithms can be tedious and time consuming.
PyTorch Adapt
PyTorch Adapt is a library for domain adaptation, a type of machine learning algorithm that re-purposes existing models to work in new domains.
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions
In this paper, we show that the key ingredients behind the vision Transformers, namely input-adaptive, long-range and high-order spatial interactions, can also be efficiently implemented with a convolution-based framework.
Ranked #20 on
Semantic Segmentation
on ADE20K
Visual Prompt Tuning
The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning.
Ranked #2 on
Prompt Engineering
on ImageNet-21k
Detecting Everything in the Open World: Towards Universal Object Detection
In this paper, we formally address universal object detection, which aims to detect every scene and predict every category.
Three New Validators and a Large-Scale Benchmark Ranking for Unsupervised Domain Adaptation
In a supervised setting, these validators evaluate checkpoints by computing accuracy on a validation set that has labels.
NeRV: Neural Representations for Videos
In contrast, with NeRV, we can use any neural network compression method as a proxy for video compression, and achieve comparable performance to traditional frame-based video compression approaches (H. 264, HEVC \etc).
Ranked #6 on
Video Reconstruction
on UVG
Edge Proposal Sets for Link Prediction
Here, we demonstrate how simply adding a set of edges, which we call a \emph{proposal set}, to the graph as a pre-processing step can improve the performance of several link prediction algorithms.
Ranked #1 on
Link Property Prediction
on ogbl-ddi
Unconstrained Facial Expression Transfer using Style-based Generator
Given two face images, our method can create plausible results that combine the appearance of one image and the expression of the other.
RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness
We show that the effectiveness of the well celebrated Mixup [Zhang et al., 2018] can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss.
On Feature Normalization and Data Augmentation
The moments (a. k. a., mean and standard deviation) of latent features are often removed as noise when training image recognition models, to increase stability and reduce training time.
Ranked #32 on
Domain Generalization
on ImageNet-A
Enhancing Adversarial Example Transferability with an Intermediate Level Attack
We show that we can select a layer of the source model to perturb without any knowledge of the target models while achieving high transferability.
What makes fake images detectable? Understanding properties that generalize
The quality of image generation and manipulation is reaching impressive levels, making it increasingly difficult for a human to distinguish between what is real and what is fake.
New Benchmarks for Learning on Non-Homophilous Graphs
Much data with graph structures satisfy the principle of homophily, meaning that connected nodes tend to be similar with respect to a specific attribute.
Ranked #4 on
Node Classification
on Yelp-Fraud
Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods
Many widely used datasets for graph machine learning tasks have generally been homophilous, where nodes with similar labels connect to each other.
Ranked #2 on
Node Classification
on wiki
Graph Learning
Node Classification on Non-Homophilic (Heterophilic) Graphs
Neural Manifold Ordinary Differential Equations
To better conform to data geometry, recent deep generative modelling techniques adapt Euclidean constructions to non-Euclidean spaces.
HNeRV: A Hybrid Neural Representation for Videos
Such embedding largely limits the regression capacity and internal generalization for video interpolation.
Ranked #3 on
Video Reconstruction
on UVG
Object Recognition as Next Token Prediction
We present an approach to pose object recognition as next token prediction.
MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
However, existing LLM-based large multimodal models (e. g., Video-LLaMA, VideoChat) can only take in a limited number of frames for short video understanding.
Ranked #1 on
Video Classification
on COIN
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection
The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images.
Graph Inductive Biases in Transformers without Message Passing
Graph inductive biases are crucial for Graph Transformers, and previous works incorporate them using message-passing modules and/or positional encodings.
Ranked #1 on
Node Classification
on PATTERN
Differentiating through the Fréchet Mean
Recent advances in deep representation learning on Riemannian manifolds extend classical deep learning operations to better capture the geometry of the manifold.
Rethinking Nearest Neighbors for Visual Classification
In this paper, we investigate $k$-Nearest-Neighbor (k-NN) classifiers, a classical model-free learning method from the pre-deep learning era, as an augmentation to modern neural network based approaches.
Intentonomy: a Dataset and Study towards Human Intent Understanding
Based on our findings, we conduct further study to quantify the effect of attending to object and context classes as well as textual information in the form of hashtags when training an intent classifier.
Better Set Representations For Relational Reasoning
Incorporating relational reasoning into neural networks has greatly expanded their capabilities and scope.
Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectors
We present a systematic study of adversarial attacks on state-of-the-art object detection frameworks.
Towards Adversarial Evaluations for Inexact Machine Unlearning
Machine Learning models face increased concerns regarding the storage of personal user data and adverse impacts of corrupted data like backdoors or systematic bias.
Spartan: Differentiable Sparsity via Regularized Transportation
We present Spartan, a method for training sparse neural network models with a predetermined level of sparsity.
Test-Time Distribution Normalization for Contrastively Learned Vision-language Models
Advances in the field of vision-language contrastive learning have made it possible for many downstream applications to be carried out efficiently and accurately by simply taking the dot product between image and text representations.
Computationally Budgeted Continual Learning: What Does Matter?
Our conclusions are consistent in a different number of stream time steps, e. g., 20 to 200, and under several computational budgets.
When in Doubt: Improving Classification Performance with Alternating Normalization
We introduce Classification with Alternating Normalization (CAN), a non-parametric post-processing step for classification.
Deep Co-Training with Task Decomposition for Semi-Supervised Domain Adaptation
To integrate the strengths of the two classifiers, we apply the well-established co-training framework, in which the two classifiers exchange their high confident predictions to iteratively "teach each other" so that both classifiers can excel in the target domain.
Semi-supervised Domain Adaptation
Unsupervised Domain Adaptation
Measuring Dataset Granularity
We also investigate the interplay between dataset granularity with a variety of factors and find that fine-grained datasets are more difficult to learn from, more difficult to transfer to, more difficult to perform few-shot learning with, and more vulnerable to adversarial attacks.
Exploring Visual Engagement Signals for Representation Learning
Visual engagement in social media platforms comprises interactions with photo posts including comments, shares, and likes.
Equivariant Manifold Flows
Tractably modelling distributions over manifolds has long been an important goal in the natural sciences.
TIPI: Test Time Adaptation With Transformation Invariance
Test Time Adaptation offers a means to combat this problem, as it allows the model to adapt during test time to the new data distribution, using only unlabeled test data batches.
Object-Centric Unsupervised Image Captioning
Our work in this paper overcomes this by harvesting objects corresponding to a given sentence from the training set, even if they don't belong to the same image.
Quantization Guided JPEG Artifact Correction
The JPEG image compression algorithm is the most popular method of image compression because of its ability for large compression ratios.
Sample-dependent Adaptive Temperature Scaling for Improved Calibration
The most common post-hoc approach to compensate for this is to perform temperature scaling, which adjusts the confidences of the predictions on any input by scaling the logits by a fixed value.
$BT^2$: Backward-compatible Training with Basis Transformation
In this way, the new representation can be directly compared with the old representation, in principle avoiding the need for any backfilling.
Rapid Adaptation in Online Continual Learning: Are We Evaluating It Right?
We revisit the common practice of evaluating adaptation of Online Continual Learning (OCL) algorithms through the metric of online accuracy, which measures the accuracy of the model on the immediate next few samples.
BT^2: Backward-compatible Training with Basis Transformation
In this way, the new representation can be directly compared with the old representation, in principle avoiding the need for any backfilling.
GAPX: Generalized Autoregressive Paraphrase-Identification X
Paraphrase Identification is a fundamental task in Natural Language Processing.
Language-free Compositional Action Generation via Decoupling Refinement
Composing simple elements into complex concepts is crucial yet challenging, especially for 3D action generation.
Intermediate Level Adversarial Attack for Enhanced Transferability
Neural networks are vulnerable to adversarial examples, malicious inputs crafted to fool trained models.
Adversarial Example Decomposition
Research has shown that widely used deep neural networks are vulnerable to carefully crafted adversarial perturbations.
Efficient Object Embedding for Spliced Image Retrieval
Detecting spliced images is one of the emerging challenges in computer vision.
Cross-X Learning for Fine-Grained Visual Categorization
Recognizing objects from subcategories with very subtle differences remains a challenging task due to the large intra-class and small inter-class variation.
Ranked #18 on
Fine-Grained Image Classification
on NABirds
(using extra training data)
Fine-Grained Image Classification
Fine-Grained Visual Categorization
Unsupervised Deep Metric Learning via Auxiliary Rotation Loss
In this work, we propose to generate pseudo-labels for deep metric learning directly from clustering assignment and we introduce unsupervised deep metric learning (UDML) regularized by a self-supervision (SS) task.
Fine-grained Synthesis of Unrestricted Adversarial Examples
We propose a novel approach for generating unrestricted adversarial examples by manipulating fine-grained aspects of image generation.
Deep Multi-Modal Sets
Most deep learning-based methods rely on a late fusion technique whereby multiple feature types are encoded and concatenated and then a multi layer perceptron (MLP) combines the fused embedding to make predictions.
One-Shot Domain Adaptation For Face Generation
To generate images of the same distribution, we introduce a style-mixing technique that transfers the low-level statistics from the target to faces randomly generated with the model.
Detecting Deep-Fake Videos from Appearance and Behavior
Synthetically-generated audios and videos -- so-called deep fakes -- continue to capture the imagination of the computer-graphics and computer-vision communities.
Curriculum Manager for Source Selection in Multi-Source Domain Adaptation
In this paper, we proposed an adversarial agent that learns a dynamic curriculum for source samples, called Curriculum Manager for Source Selection (CMSS).
Multi-Source Unsupervised Domain Adaptation
Unsupervised Domain Adaptation
Analyzing and Mitigating JPEG Compression Defects in Deep Learning
We show that there is a significant penalty on common performance metrics for high compression.
GTA: Global Temporal Attention for Video Action Understanding
To this end, we introduce Global Temporal Attention (GTA), which performs global temporal attention on top of spatial attention in a decoupled manner.
Deep Video Inpainting Detection
This paper studies video inpainting detection, which localizes an inpainted region in a video both spatially and temporally.
THAT: Two Head Adversarial Training for Improving Robustness at Scale
Many variants of adversarial training have been proposed, with most research focusing on problems with relatively few classes.
Multimodal Fusion Refiner Networks
In this work, we develop a Refiner Fusion Network (ReFNet) that enables fusion modules to combine strong unimodal representation with strong multimodal representations.
Cross-Modal Retrieval Augmentation for Multi-Modal Classification
Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing.
Joint Audio-Visual Deepfake Detection
Deepfakes ("deep learning" + "fake") are synthetically-generated videos from AI algorithms.
Ranked #6 on
DeepFake Detection
on FakeAVCeleb
Mix-MaxEnt: Creating High Entropy Barriers To Improve Accuracy and Uncertainty Estimates of Deterministic Neural Networks
We propose an extremely simple approach to regularize a single deterministic neural network to obtain improved accuracy and reliable uncertainty estimates.
Testing-Time Adaptation through Online Normalization Estimation
We present a simple and effective way to estimate the batch-norm statistics during test time, to fast adapt a source model to target test samples.
Refining Multimodal Representations using a modality-centric self-supervised module
Tasks that rely on multi-modal information typically include a fusion module that combines information from different modalities.
Fast Adaptive Anomaly Detection
Anything outside of a given normal population is by definition an anomaly.
Differential Motion Evolution for Fine-Grained Motion Deformation in Unsupervised Image Animation
Key findings are twofold: (1) by capturing the motion transfer with an ordinary differential equation (ODE), it helps to regularize the motion field, and (2) by utilizing the source image itself, we are able to inpaint occluded/missing regions arising from large motion changes.
MixNorm: Test-Time Adaptation Through Online Normalization Estimation
We present a simple and effective way to estimate the batch-norm statistics during test time, to fast adapt a source model to target test samples.
Learning to Ground Multi-Agent Communication with Autoencoders
Communication requires having a common language, a lingua franca, between agents.
A Frequency Perspective of Adversarial Robustness
Our analysis shows that adversarial examples are neither in high-frequency nor in low-frequency components, but are simply dataset dependent.
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
To this end, we introduce AdaViT, an adaptive computation framework that learns to derive usage policies on which patches, self-attention heads and transformer blocks to use throughout the backbone on a per-input basis, aiming to improve inference efficiency of vision transformers with a minimal drop of accuracy for image recognition.
Unsupervised Domain Adaptation: A Reality Check
Interest in unsupervised domain adaptation (UDA) has surged in recent years, resulting in a plethora of new algorithms.
ObjectFormer for Image Manipulation Detection and Localization
Recent advances in image editing techniques have posed serious challenges to the trustworthiness of multimedia data, which drives the research of image tampering detection.
VRAG: Region Attention Graphs for Content-Based Video Retrieval
In this paper, we introduce Video Region Attention Graph Networks (VRAG) that improves the state-of-the-art of video-level methods.
Ranked #13 on
Video Retrieval
on FIVR-200K
Raising the Bar on the Evaluation of Out-of-Distribution Detection
In this paper, we define 2 categories of OoD data using the subtly different concepts of perceptual/visual and semantic similarity to in-distribution (iD) data.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
+2
Diversified Dynamic Routing for Vision Tasks
Deep learning models for vision tasks are trained on large datasets under the assumption that there exists a universal representation that can be used to make predictions for all samples.
Totems: Physical Objects for Verifying Visual Integrity
We introduce a new approach to image forensics: placing physical refractive objects, which we call totems, into a scene so as to protect any photograph taken of that scene.
CNeRV: Content-adaptive Neural Representation for Visual Data
We match the performance of NeRV, a state-of-the-art implicit neural representation, on the reconstruction task for frames seen during training while far surpassing for frames that are skipped during training (unseen images).
Unifying Tracking and Image-Video Object Detection
We propose TrIVD (Tracking and Image-Video Detection), the first framework that unifies image OD, video OD, and MOT within one end-to-end model.
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning
We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder.
Online Backfilling with No Regret for Large-Scale Image Retrieval
Backfilling is the process of re-extracting all gallery embeddings from upgraded models in image retrieval systems.
Open-vocabulary Panoptic Segmentation with Embedding Modulation
Open-vocabulary image segmentation is attracting increasing attention due to its critical applications in the real world.
ScribbleSeg: Scribble-based Interactive Image Segmentation
We hope this could serve as a more powerful and general solution for interactive segmentation.
Towards Scalable Neural Representation for Diverse Videos
Implicit neural representations (INR) have gained increasing attention in representing 3D scenes and images, and have been recently applied to encode videos (e. g., NeRV, E-NeRV).
LASER: A Neuro-Symbolic Framework for Learning Spatial-Temporal Scene Graphs with Weak Supervision
We propose LASER, a neuro-symbolic approach to learn semantic video representations that capture rich spatial and temporal properties in video data by leveraging high-level logic specifications.
Stable Estimation of Survival Causal Effects
Our experiments on synthetic and semi-synthetic data demonstrate that our method has competitive bias and smaller variance than debiased machine learning approaches.
From Categories to Classifier: Name-Only Continual Learning by Exploring the Web
Continual Learning (CL) often relies on the availability of extensive annotated datasets, an assumption that is unrealistically time-consuming and costly in practice.
CLAMP: Contrastive LAnguage Model Prompt-tuning
Our results show that LLMs can, indeed, achieve good image classification performance when adapted this way.
Label Delay in Continual Learning
We introduce a new continual learning framework with explicit modeling of the label delay between data and label streams over time steps.
On the Robustness of Large Multimodal Models Against Image Adversarial Attacks
Recent advances in instruction tuning have led to the development of State-of-the-Art Large Multimodal Models (LMMs).
Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements
In this paper, we present a novel robust framework for low-level vision tasks, including denoising, object removal, frame interpolation, and super-resolution, that does not require any external training data corpus.
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
In this work, we introduce VistaLLM, a powerful visual system that addresses coarse- and fine-grained VL tasks over single and multiple input images using a unified framework.
Universal Pyramid Adversarial Training for Improved ViT Performance
Recently, Pyramid Adversarial training (Herrmann et al., 2022) has been shown to be very effective for improving clean accuracy and distribution-shift robustness of vision transformers.
Mitigating Dialogue Hallucination for Large Multi-modal Models via Adversarial Instruction Tuning
To precisely measure this, we first present an evaluation benchmark by extending popular multi-modal benchmark datasets with prepended hallucinatory dialogues generated by our novel Adversarial Question Generator, which can automatically generate image-related yet adversarial dialogues by adopting adversarial attacks on LMMs.
