Search Results for author:
Found 72 papers, 59 papers with code
FD-GAN: Pose-guided Feature Distilling GAN for Robust Person Re-identification
Our proposed FD-GAN achieves state-of-the-art performance on three person reID datasets, which demonstrates that the effectiveness and robust feature distilling capability of the proposed FD-GAN.
Ranked #3 on
Person Re-Identification
on CUHK03
Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification
In order to mitigate the effects of noisy pseudo labels, we propose to softly refine the pseudo labels in the target domain by proposing an unsupervised framework, Mutual Mean-Teaching (MMT), to learn better features from the target domain via off-line refined hard pseudo labels and on-line refined soft pseudo labels in an alternative training manner.
Structured Domain Adaptation with Online Relation Regularization for Unsupervised Person Re-ID
To tackle the challenges, we propose an end-to-end structured domain adaptation framework with an online relation-consistency regularization term.
Ranked #4 on
Unsupervised Domain Adaptation
on Market to MSMT
Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID
To solve these problems, we propose a novel self-paced contrastive learning framework with hybrid memory.
Ranked #3 on
Unsupervised Domain Adaptation
on Market to MSMT
Self-supervising Fine-grained Region Similarities for Large-scale Image Localization
The task of large-scale retrieval-based image localization is to estimate the geographical location of a query image by recognizing its nearest reference images from a city-scale dataset.
Improved Mutual Mean-Teaching for Unsupervised Domain Adaptive Re-ID
SDA, a domain-translation-based framework, focuses on carefully translating the source-domain images to the target domain.
Online Pseudo Label Generation by Hierarchical Cluster Dynamics for Adaptive Person Re-Identification
To tackle the problem, we propose an online pseudo label generation by hierarchical cluster dynamics for adaptive ReID.
Progressive Correspondence Pruning by Consensus Learning
Correspondence selection aims to correctly select the consistent matches (inliers) from an initial set of putative correspondences.
DivCo: Diverse Conditional Image Synthesis via Contrastive Generative Adversarial Network
Conditional generative adversarial networks (cGANs) target at synthesizing diverse images given the input conditions and latent codes, but unfortunately, they usually suffer from the issue of mode collapse.
Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification
In this way, our BAKE framework achieves online knowledge ensembling across multiple samples with only a single network.
Refining Pseudo Labels with Clustering Consensus over Generations for Unsupervised Object Re-identification
Unsupervised object re-identification targets at learning discriminative representations for object retrieval without any annotations.
Mutual CRF-GNN for Few-Shot Learning
In this MCGN, the labels and features of support data are used by the CRF for inferring GNN affinities in a principled and probabilistic way.
Hot-Refresh Model Upgrades with Regression-Free Compatible Training in Image Retrieval
In contrast, hot-refresh model upgrades deploy the new model immediately and then gradually improve the retrieval accuracy by backfilling the gallery on-the-fly.
Object-aware Video-language Pre-training for Retrieval
In this work, we present Object-aware Transformers, an object-centric approach that extends video-language transformer to incorporate object representations.
Ranked #20 on
Zero-Shot Video Retrieval
on DiDeMo
Video-Text Pre-training with Learned Regions
Video-Text pre-training aims at learning transferable representations from large-scale video-text pairs via aligning the semantics between visual and textual information.
Dynamic Token Normalization Improves Vision Transformers
It is difficult for Transformers to capture inductive bias such as the positional context in an image with LN.
Bridging Video-text Retrieval with Multiple Choice Questions
As an additional benefit, our method achieves competitive results with much shorter pre-training videos on single-modality downstream tasks, e. g., action recognition with linear evaluation.
Ranked #8 on
Zero-Shot Video Retrieval
on MSVD
Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval
In contrast, hot-refresh model upgrades deploy the new model immediately and then gradually improve the retrieval accuracy by backfilling the gallery on-the-fly.
Uncertainty Modeling for Out-of-Distribution Generalization
In this paper, we improve the network generalization ability by modeling the uncertainty of domain shifts with synthesized feature statistics during training.
Towards Universal Backward-Compatible Representation Learning
The task of backward-compatible representation learning is therefore introduced to support backfill-free model upgrades, where the new query features are interoperable with the old gallery features.
All in One: Exploring Unified Video-Language Pre-training
In this work, we for the first time introduce an end-to-end video-language model, namely \textit{all-in-one Transformer}, that embeds raw video and textual signals into joint representations using a unified backbone architecture.
Ranked #6 on
TGIF-Transition
on TGIF-QA
(using extra training data)
Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval
Recent dominant methods for video-language pre-training (VLP) learn transferable representations from the raw pixels in an end-to-end manner to achieve advanced performance on downstream video-language retrieval.
mc-BEiT: Multi-choice Discretization for Image BERT Pre-training
Image BERT pre-training with masked image modeling (MIM) becomes a popular practice to cope with self-supervised representation learning.
Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection
We present an approach to efficiently and effectively adapt a masked image modeling (MIM) pre-trained vanilla Vision Transformer (ViT) for object detection, which is based on our two novel observations: (i) A MIM pre-trained vanilla ViT encoder can work surprisingly well in the challenging object-level recognition scenario even with randomly sampled partial observations, e. g., only 25% $\sim$ 50% of the input embeddings.
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval
Dominant pre-training work for video-text retrieval mainly adopt the "dual-encoder" architectures to enable efficient retrieval, where two separate encoders are used to contrast global video and text representations, but ignore detailed local semantics.
Ranked #7 on
Zero-Shot Video Retrieval
on MSVD
Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval
The task of privacy-preserving model upgrades in image retrieval desires to reap the benefits of rapidly evolving new models without accessing the raw gallery images.
Masked Image Modeling with Denoising Contrast
Since the development of self-supervised visual representation learning from contrastive learning to masked image modeling (MIM), there is no significant difference in essence, that is, how to design proper pretext tasks for vision dictionary look-up.
Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space
It is challenging because the ground-truth model ranking for each task can only be generated by fine-tuning the pre-trained models on the target dataset, which is brute-force and computationally expensive.
Ranked #2 on
Transferability
on classification benchmark
Equivariant Filter Design for Discrete-time systems
The kinematics of many nonlinear control systems, especially in the robotics field, admit a transitive Lie-group symmetry, which is useful in high performance observer design.
Learning Transferable Spatiotemporal Representations from Natural Script Knowledge
Pre-training on large-scale video data has become a common recipe for learning transferable spatiotemporal representations in recent years.
Darwinian Model Upgrades: Model Evolving with Selective Compatibility
The traditional model upgrading paradigm for retrieval requires recomputing all gallery embeddings before deploying the new model (dubbed as "backfilling"), which is quite expensive and time-consuming considering billions of instances in industrial applications.
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis
Vector-Quantized (VQ-based) generative models usually consist of two basic components, i. e., VQ tokenizers and generative transformers.
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator.
Modeling Uncertain Feature Representation for Domain Generalization
In this paper, we improve the network generalization ability by modeling domain shifts with uncertainty (DSU), i. e., characterizing the feature statistics as uncertain distributions during training.
RILS: Masked Visual Reconstruction in Language Semantic Space
Both masked image modeling (MIM) and natural language supervision have facilitated the progress of transferable visual pre-training.
Binary Embedding-based Retrieval at Tencent
To tackle the challenge, we propose a binary embedding-based retrieval (BEBR) engine equipped with a recurrent binarization algorithm that enables customized bits per dimension.
BoxSnake: Polygonal Instance Segmentation with Box Supervision
Box-supervised instance segmentation has gained much attention as it requires only simple box annotations instead of costly mask or polygon annotations.
Accelerating Vision-Language Pretraining with Free Language Modeling
FLM successfully frees the prediction rate from the tie-up with the corruption rate while allowing the corruption spans to be customized for each token to be predicted.
TagGPT: Large Language Models are Zero-shot Multimodal Taggers
Given a new entity that needs tagging for distribution, TagGPT introduces two alternative options for zero-shot tagging, i. e., a generative method with late semantic matching with the tag set, and another selective method with early matching in prompts.
Attack is Good Augmentation: Towards Skeleton-Contrastive Representation Learning
In particular, we propose a novel Attack-Augmentation Mixing-Contrastive learning (A$^2$MC) to contrast hard positive features and hard negative features for learning more robust skeleton representations.
$π$-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation
Foundation models have achieved great advances in multi-task learning with a unified interface of unimodal and multimodal tasks.
What Makes for Good Visual Tokenizers for Large Language Models?
In our benchmark, which is curated to evaluate MLLMs visual semantic understanding and fine-grained perception capabilities, we discussed different visual tokenizers pre-trained with dominant methods (i. e., DeiT, CLIP, MAE, DINO), and observe that: i) Fully/weakly supervised models capture more semantics than self-supervised models, but the gap is narrowed by scaling up the pre-training dataset.
TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale
We argue that tuning a text encoder end-to-end, as done in previous work, is suboptimal since it may overfit in terms of styles, thereby losing its original generalization ability to capture the semantics of various language registers.
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community.
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
This paper aims to efficiently enable Large Language Models (LLMs) to use multimodal tools.
Sticker820K: Empowering Interactive Retrieval with Stickers
Therefore, we propose StickerCLIP as a benchmark model on the Sticker820K dataset.
TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter
In situations involving system upgrades that require updating the upstream foundation model, it becomes essential to re-train all downstream modules to adapt to the new foundation model, which is inflexible and inefficient.
PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television Dramas
Art forms such as movies and television (TV) dramas are reflections of the real world, which have attracted much attention from the multimodal learning community recently.
DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
This paper introduces DreamDiffusion, a novel method for generating high-quality images directly from brain electroencephalogram (EEG) signals, without the need to translate thoughts into text.
Planting a SEED of Vision in Large Language Model
Research on image tokenizers has previously reached an impasse, as frameworks employing quantized visual tokens have lost prominence due to subpar performance and convergence in multimodal comprehension (compared to BLIP-2, etc.)
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
Based on powerful Large Language Models (LLMs), recent generative Multimodal Large Language Models (MLLMs) have gained prominence as a pivotal research area, exhibiting remarkable capability for both comprehension and generation.
ViT-Lens: Initiating Omni-Modal Exploration through 3D Insights
A well-trained lens with a ViT backbone has the potential to serve as one of these foundation models, supervising the learning of subsequent modalities.
Ranked #2 on
Zero-Shot Transfer 3D Point Cloud Classification
on ModelNet40
(using extra training data)
Exploring Model Transferability through the Lens of Potential Energy
The experimental results on 10 downstream tasks and 12 self-supervised models demonstrate that our approach can seamlessly integrate into existing ranking techniques and enhance their performances, revealing its effectiveness for the model selection task and its potential for understanding the mechanism in transfer learning.
A Note on the Extended Kalman Filter on a Manifold
With this structure, we show that a naive coordinate implementation of the EKF fails to account for geometry of the manifold in the update step and in the reset step.
One For All: Video Conversation is Feasible Without Video Instruction Tuning
Without bells and whistles, BT-Adapter achieves (1) state-of-the-art zero-shot results on various video tasks using thousands of fewer GPU hours.
Ranked #5 on
Zero-Shot Video Retrieval
on LSMDC
Video-based Generative Performance Benchmarking (Consistency)
Video-based Generative Performance Benchmarking (Contextual Understanding)
+6
Making LLaMA SEE and Draw with SEED Tokenizer
We identify two crucial design principles: (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a 1D causal dependency, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs.
Meta-Adapter: An Online Few-shot Learner for Vision-Language Model
Without bells and whistles, our approach outperforms the state-of-the-art online few-shot learning method by an average of 3. 6\% on eight image classification datasets with higher inference speed.
Vision-Language Instruction Tuning: A Review and Analysis
Instruction tuning is a crucial supervised training phase in Large Language Models (LLMs), aiming to enhance the LLM's ability to generalize instruction execution and adapt to user preferences.
ViT-Lens: Towards Omni-modal Representations
In this paper, we present ViT-Lens-2 that facilitates efficient omni-modal representation learning by perceiving novel modalities with a pretrained ViT and aligning them to a pre-defined space.
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition
1) We propose four architectural guidelines for designing large-kernel ConvNets, the core of which is to exploit the essential characteristics of large kernels that distinguish them from small kernels - they can see wide without going deep.
Ranked #1 on
Object Detection
on COCO 2017
(mAP metric)
SEED-Bench-2: Benchmarking Multimodal Large Language Models
Multimodal large language models (MLLMs), building upon the foundation of powerful large language models (LLMs), have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs (acting like a combination of GPT-4V and DALL-E 3).
SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models
Both quantitative and qualitative results on this evaluation dataset indicate that our SmartEdit surpasses previous methods, paving the way for the practical application of complex instruction-based image editing.
EgoPlan-Bench: Benchmarking Egocentric Embodied Planning with Multimodal Large Language Models
Given diverse environmental inputs, including real-time task progress, visual observations, and open-form language instructions, a proficient task planner is expected to predict feasible actions, which is a feat inherently achievable by Multimodal Large Language Models (MLLMs).
VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation
In combination with the existing text tokenizer and detokenizer, this framework allows for the encoding of interleaved image-text data into a multimodal sequence, which can subsequently be fed into the transformer model.
Cached Transformers: Improving Transformers with Differentiable Memory Cache
This work introduces a new Transformer model called Cached Transformer, which uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens.
LLaMA Pro: Progressive LLaMA with Block Expansion
Humans generally acquire new skills without compromising the old; however, the opposite holds for Large Language Models (LLMs), e. g., from LLaMA to CodeLLaMA.
Towards A Better Metric for Text-to-Video Generation
Experiments on the TVGE dataset demonstrate the superiority of the proposed T2VScore on offering a better metric for text-to-video generation.
Supervised Fine-tuning in turn Improves Visual Foundation Models
Image-text training like CLIP has dominated the pretraining of vision foundation models in recent years.
Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities
We propose to improve transformers of a specific modality with irrelevant data from other modalities, e. g., improve an ImageNet model with audio or point cloud datasets.
YOLO-World: Real-Time Open-Vocabulary Object Detection
The You Only Look Once (YOLO) series of detectors have established themselves as efficient and practical tools.
A Geometric Perspective on Fusing Gaussian Distributions on Lie Groups
Stochastic inference on Lie groups plays a key role in state estimation problems, such as inertial navigation, visual inertial odometry, pose estimation in virtual reality, etc.
ST-LLM: Large Language Models Are Effective Temporal Learners
In this paper, we investigate a straightforward yet unexplored question: Can we feed all spatial-temporal tokens into the LLM, thus delegating the task of video sequence modeling to the LLMs?
