Search Results for author:
Found 88 papers, 57 papers with code
TaxaDiffusion: Progressively Trained Diffusion Model for Fine-Grained Species Generation
We propose TaxaDiffusion, a taxonomy-informed training framework for diffusion models to generate fine-grained animal images with high morphological and identity accuracy.
BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning
We find such emergent behaviors in biological vision models via large-scale contrastive vision-language training.
Revisiting semi-supervised learning in the era of foundation models
We make a surprising observation: parameter-efficient fine-tuning (PEFT) using only labeled data often matches SSL performance, even without leveraging unlabeled data.
Federated Inverse Probability Treatment Weighting for Individual Treatment Effect Estimation
To address this, we propose FED-IPTW, a novel algorithm to extend IPTW into a federated setting that enforces both global (over all the data) and local (within each hospital) decorrelation between covariates and treatments.
Building Machine Learning Challenges for Anomaly Detection in Science
We present the different datasets along with a scheme to make machine learning challenges around the three datasets findable, accessible, interoperable, and reusable (FAIR).
A Closer Look at TabPFN v2: Strength, Limitation, and Extension
Tabular datasets are inherently heterogeneous, posing significant challenges for developing pre-trained foundation models.
Mixed Signals: A Diverse Point Cloud Dataset for Heterogeneous LiDAR V2X Collaboration
Vehicle-to-everything (V2X) collaborative perception has emerged as a promising solution to address the limitations of single-vehicle perception systems.
Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene
Self-driving cars relying solely on ego-centric perception face limitations in sensing, often failing to detect occluded, faraway objects.
Finer-CAM: Spotting the Difference Reveals Finer Details for Visual Explanation
Despite its simplicity and computational efficiency, CAM often struggles to identify discriminative regions that distinguish visually similar fine-grained classes.
Prompt-CAM: A Simpler Interpretable Transformer for Fine-Grained Analysis
We present a simple usage of pre-trained Vision Transformers (ViTs) for fine-grained analysis, aiming to identify and localize the traits that distinguish visually similar categories, such as different bird species or dog breeds.
Explainable Artificial Intelligence (XAI)
Explainable Models
+3
Static Segmentation by Tracking: A Frustratingly Label-Efficient Approach to Fine-Grained Segmentation
Powered by Segment Anything Model 2 (SAM~2) initially developed for video segmentation, we show that SST can achieve high-quality trait and part segmentation with merely one labeled image per species -- a breakthrough for analyzing specimen images.
Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis
We present a simple approach to make pre-trained Vision Transformers (ViTs) interpretable for fine-grained analysis, aiming to identify and localize the traits that distinguish visually similar categories, such as bird species.
Learning 3D Perception from Others' Predictions
We investigate a new scenario to construct 3D object detectors: learning from the predictions of a nearby unit that is equipped with an accurate detector.
KnobGen: Controlling the Sophistication of Artwork in Sketch-Based Diffusion Models
Recent advances in diffusion models have significantly improved text-to-image (T2I) generation, but they often struggle to balance fine-grained precision with high-level control.
Fine-Tuning is Fine, if Calibrated
For example, fine-tuning a pre-trained classifier capable of recognizing a large number of classes to master a subset of classes at hand is shown to drastically degrade the model's accuracy in the other classes it had previously learned.
Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition
Second, despite similar accuracy, we find that PEFT methods make different mistakes and high-confidence predictions, likely due to their different inductive biases.
FedNE: Surrogate-Assisted Federated Neighbor Embedding for Dimensionality Reduction
Federated learning (FL) has rapidly evolved as a promising paradigm that enables collaborative model training across distributed participants without exchanging their local data.
Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning
Second, we employ a two-branch framework empowered by knowledge distillation, enabling the model to take both the filtered and original images as input, largely reducing the burden of downstream tasks.
VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images
Images are increasingly becoming the currency for documenting biodiversity on the planet, providing novel opportunities for accelerating scientific discoveries in the field of organismal biology, especially with the advent of large vision-language models (VLMs).
MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs
In this paper, we introduce MLLM-CompBench, a benchmark designed to evaluate the comparative reasoning capability of multimodal large language models (MLLMs).
Jigsaw Game: Federated Clustering
Federated learning has recently garnered significant attention, especially within the domain of supervised learning.
Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images
Fishes are integral to both ecological systems and economic sectors, and studying fish traits is crucial for understanding biodiversity patterns and macro-evolution trends.
DiffuBox: Refining 3D Object Detection with Point Diffusion
Ensuring robust 3D object detection and localization is crucial for many applications in robotics and autonomous driving.
Better Monocular 3D Detectors with LiDAR from the Past
Accurate 3D object detection is crucial to autonomous driving.
Dual-View Visual Contextualization for Web Navigation
Automatic web navigation aims to build a web agent that can follow language instructions to execute complex and diverse tasks on real-world websites.
Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs
In this work, we exploit the structured context linked to camera trap images to boost out-of-distribution generalization for species classification tasks in camera traps.
Ranked #1 on
Image Classification
on iWildCam2020-WILDS
(using extra training data)
BioCLIP: A Vision Foundation Model for the Tree of Life
We then develop BioCLIP, a foundation model for the tree of life, leveraging the unique properties of biology captured by TreeOfLife-10M, namely the abundance and variety of images of plants, animals, and fungi, together with the availability of rich structured biological knowledge.
A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis
We present a novel usage of Transformers to make image classification interpretable.
Pre-Training LiDAR-Based 3D Object Detectors Through Colorization
Accurate 3D object detection and understanding for self-driving cars heavily relies on LiDAR point clouds, necessitating large amounts of labeled data to train.
FLEE-GNN: A Federated Learning System for Edge-Enhanced Graph Neural Network in Analyzing Geospatial Resilience of Multicommodity Food Flows
This paper proposes FLEE-GNN, a novel Federated Learning System for Edge-Enhanced Graph Neural Network, designed to overcome these challenges and enhance the analysis of geospatial resilience of multicommodity food flow network, which is one type of spatial networks.
Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features
Additionally, we leverage the statistics for a novel self-training process to stabilize the training.
Discovering Novel Biological Traits From Images Using Phylogeny-Guided Neural Networks
Discovering evolutionary traits that are heritable across species on the tree of life (also referred to as a phylogenetic tree) is of great interest to biologists to understand how organisms diversify and evolve.
Towards Open-World Segmentation of Parts
Second, we introduce a novel approach to improve part segmentation on unseen objects, inspired by an interesting finding -- for unseen objects, the pixel-wise features extracted by the model often reveal high-quality part segments.
Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation
Weakly supervised semantic segmentation (WSSS) aims to bypass the need for laborious pixel-level annotation by using only image-level annotation.
Federated Learning of Shareable Bases for Personalization-Friendly Image Classification
FedBasis learns a set of few shareable ``basis'' models, which can be linearly combined to form personalized models for clients.
Unified Out-Of-Distribution Detection: A Model-Specific Perspective
We show that this framework unifies the detection of OOD examples caused by semantic shift and covariate shift, and closely addresses the concern of applying a machine learning model to uncontrolled environments.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Unsupervised Adaptation from Repeated Traversals for Autonomous Driving
For a self-driving car to operate reliably, its perceptual system must generalize to the end-user's environment -- ideally without additional annotation efforts.
Learning Fractals by Gradient Descent
Fractals are geometric shapes that can display complex and self-similar patterns found in nature (e. g., clouds and plants).
Making Batch Normalization Great in Federated Deep Learning
We reinvestigate factors that are believed to cause this problem, including the mismatch of BN statistics across clients and the deviation of gradients during local training.
Train-Once-for-All Personalization
We study the problem of how to train a "personalization-friendly" model such that given only the task descriptions, the model can be adapted to different end-users' needs, e. g., for accurately classifying different subsets of objects.
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
In this work, we propose a novel method, LLM-Planner, that harnesses the power of large language models to do few-shot planning for embodied agents.
Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning
Through introducing a handful of learnable ``query'' tokens to each layer, VQT leverages the inner workings of Transformers to ``summarize'' rich intermediate features of each layer, which can then be used to train the prediction heads of downstream tasks.
Image-to-Image Translation for Autonomous Driving from Coarsely-Aligned Image Pairs
In this paper, we investigate the idea of turning sensor inputs (i. e., images) captured in an adverse condition into a benign one (i. e., sunny), upon which the downstream tasks (e. g., semantic segmentation) can attain high accuracy.
PreSTU: Pre-Training for Scene-Text Understanding
The ability to recognize and reason about text embedded in visual inputs is often lacking in vision-and-language (V&L) models, perhaps because V&L pre-training methods have often failed to include such an ability in their training objective.
Ithaca365: Dataset and Driving Perception under Repeated and Challenging Weather Conditions
Advances in perception for self-driving cars have accelerated in recent years due to the availability of large-scale datasets, typically collected at specific locations and under nice weather conditions.
Gradual Domain Adaptation without Indexed Intermediate Domains
This coarse domain sequence then undergoes a fine indexing step via a novel cycle-consistency loss, which encourages the next intermediate domain to preserve sufficient discriminative knowledge of the current intermediate domain.
On the Importance and Applicability of Pre-Training for Federated Learning
To make our findings applicable to situations where pre-trained models are not directly available, we explore pre-training with synthetic data or even with clients' data in a decentralized manner, and found that they can already improve FL notably.
Learning to Detect Mobile Objects from LiDAR Scans Without Labels
Current 3D object detectors for autonomous driving are almost entirely trained on human-annotated data.
Hindsight is 20/20: Leveraging Past Traversals to Aid 3D Perception
Self-driving cars must detect vehicles, pedestrians, and other traffic participants accurately to operate safely.
Learning with Free Object Segments for Long-Tailed Instance Segmentation
One fundamental challenge in building an instance segmentation model for a large number of classes in complex scenes is the lack of training examples, especially for rare objects.
One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones
We study the problem of developing autonomous agents that can follow human instructions to infer and perform a sequence of actions to complete the underlying task.
Two-Stage Mesh Deep Learning for Automated Tooth Segmentation and Landmark Localization on 3D Intraoral Scans
Our TS-MDL first adopts an end-to-end \emph{i}MeshSegNet method (i. e., a variant of the existing MeshSegNet with both improved accuracy and efficiency) to label each tooth on the downsampled scan.
Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering
We found that many of the "unknowns" to the learned VQA model are indeed "known" in the dataset implicitly.
On Model Calibration for Long-Tailed Object Detection and Instance Segmentation
We propose NorCal, Normalized Calibration for long-tailed object detection and instance segmentation, a simple and straightforward recipe that reweighs the predicted scores of each class by its training sample size.
On Bridging Generic and Personalized Federated Learning for Image Classification
On the one hand, we introduce a family of losses that are robust to non-identical class distributions, enabling clients to train a generic predictor with a consistent objective across them.
Few-Shot Learning with a Strong Teacher
Many existing works take the meta-learning approach, constructing a few-shot learner that can learn from few-shot examples to generate a classifier.
How to Train Your MAML to Excel in Few-Shot Classification
We find that these permutations lead to a huge variance of accuracy, making MAML unstable in few-shot classification.
Revisiting Document Representations for Large-Scale Zero-Shot Learning
Zero-shot learning aims to recognize unseen objects using their semantic representations.
Procrustean Training for Imbalanced Deep Learning
To correct these wrong predictions, the neural network then must focus on pushing features of minor class data across the decision boundaries between major and minor classes, leading to much larger gradients for features of minor classes.
Exploiting Playbacks in Unsupervised Domain Adaptation for 3D Object Detection
Self-driving cars must detect other vehicles and pedestrians in 3D to plan safe routes and avoid collisions.
MosaicOS: A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection
Many objects do not appear frequently enough in complex scenes (e. g., certain handbags in living rooms) for training an accurate object detector, but are often found frequently by themselves (e. g., in product images).
FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
Federated learning aims to collaboratively train a strong global model by accessing users' locally trained models but not their own data.
Deep Co-Training with Task Decomposition for Semi-Supervised Domain Adaptation
To integrate the strengths of the two classifiers, we apply the well-established co-training framework, in which the two classifiers exchange their high confident predictions to iteratively "teach each other" so that both classifiers can excel in the target domain.
Semi-supervised Domain Adaptation
Unsupervised Domain Adaptation
Wasserstein Distances for Stereo Disparity Estimation
Existing approaches to depth or disparity estimation output a distribution over a set of pre-defined discrete values.
Ranked #2 on
Stereo Depth Estimation
on KITTI2015
(three pixel error metric)
3D Object Detection From Stereo Images
Autonomous Driving
+5
Train in Germany, Test in The USA: Making 3D Object Detectors Generalize
In the domain of autonomous driving, deep learning has substantially improved the 3D object detection accuracy for LiDAR and stereo camera data alike.
End-to-End Pseudo-LiDAR for Image-Based 3D Object Detection
Reliable and accurate 3D object detection is a necessity for safe autonomous driving.
Interactive Natural Language-based Person Search
In this work, we consider the problem of searching people in an unconstrained environment, with natural language descriptions.
Revisiting Meta-Learning as Supervised Learning
Recent years have witnessed an abundance of new publications and approaches on meta-learning.
An Empirical Study of Person Re-Identification with Attributes
Person re-identification aims to identify a person from an image collection, given one image of that person as the query.
Visual Question Answering on 360° Images
We then study two different VQA models on VQA 360, including one conventional model that takes an equirectangular image (with intrinsic distortion) as input and one dedicated model that first projects a 360 image onto cubemaps and subsequently aggregates the information from multiple spatial resolutions.
Identifying and Compensating for Feature Deviation in Imbalanced Deep Learning
Classifiers trained with class-imbalanced data are known to perform poorly on test data of the "minor" classes, of which we have insufficient training data.
SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning
Few-shot learners aim to recognize new object classes based on a small number of labeled training examples.
LDLS: 3-D Object Segmentation Through Label Diffusion From 2-D Images
We obtain 2-D segmentation predictions by applying Mask-RCNN to the RGB image, and then link this image to a 3-D lidar point cloud by building a graph of connections among 3-D points and 2-D pixels.
A New Defense Against Adversarial Images: Turning a Weakness into a Strength
Natural images are virtually surrounded by low-density misclassified regions that can be efficiently discovered by gradient-guided search --- enabling the generation of adversarial images.
An Empirical Study on Leveraging Scene Graphs for Visual Question Answering
Specifically, we investigate the use of scene graphs derived from images for Visual QA: an image is abstractly represented by a graph with nodes corresponding to object entities and edges to object relationships.
Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving
In this paper we provide substantial advances to the pseudo-LiDAR framework through improvements in stereo depth estimation.
3D Object Detection From Stereo Images
Autonomous Driving
+2
Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving
However, in this paper we argue that it is not the quality of the data but its representation that accounts for the majority of the difference.
3D Object Detection From Stereo Images
Autonomous Driving
+2
Classifier and Exemplar Synthesis for Zero-Shot Learning
Zero-shot learning (ZSL) enables solving a task without the need to see its examples.
Learning Answer Embeddings for Visual Question Answering
These properties make the approach particularly appealing for transfer learning for open-ended Visual QA, where the source dataset on which the model is learned has limited overlapping with the target dataset in the space of answers.
Cross-Dataset Adaptation for Visual Question Answering
Analogous to domain adaptation for visual recognition, this setting is appealing when the target dataset does not have a sufficient amount of labeled data to learn an "in-domain" model.
Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
We apply the procedures to re-construct decoy answers for two popular Visual QA datasets as well as to create a new Visual QA dataset from the Visual Genome project, resulting in the largest dataset for this task.
Video Summarization with Long Short-term Memory
We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots.
Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning
Leveraging class semantic descriptions and examples of known objects, zero-shot learning makes it possible to train a recognition model for an object class whose examples are not available.
An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild
Zero-shot learning (ZSL) methods have been studied in the unrealistic setting where test data are assumed to come from unseen classes only.
Summary Transfer: Exemplar-based Subset Selection for Video Summarization
Video summarization has unprecedented importance to help us digest, browse, and search today's ever-growing video collections.
Synthesized Classifiers for Zero-Shot Learning
Given semantic descriptions of object classes, zero-shot learning aims to accurately recognize objects of the unseen classes, from which no examples are available at the training stage, by associating them to the seen classes, from which labeled examples are provided.
Ranked #1 on
Few-Shot Image Classification
on AWA - 0-Shot
Diverse Sequential Subset Selection for Supervised Video Summarization
Video summarization is a challenging problem with great application potential.
Large-Margin Determinantal Point Processes
Extensive empirical studies validate our contributions, including applications on challenging document and video summarization, where flexibility in modeling the kernel matrix and balancing different errors is indispensable.
