Search Results for author:
Found 32 papers, 20 papers with code
M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation
In this paper, we show that this scale-wise autoregressive framework can be effectively decoupled into \textit{intra-scale modeling}, which captures local spatial dependencies within each scale, and \textit{inter-scale modeling}, which models cross-scale relationships progressively from coarse-to-fine scales.
Causal Image Modeling for Efficient Visual Understanding
In this work, we present a comprehensive analysis of causal image modeling and introduce the Adventurer series models where we treat images as sequences of patch tokens and employ uni-directional language models to learn visual representations.
What If We Recaption Billions of Web Images with LLaMA-3?
For discriminative models like CLIP, we observe enhanced zero-shot performance in cross-modal retrieval tasks.
Ranked #119 on
Visual Question Answering
on MM-Vet
Autoregressive Pretraining with Mamba in Vision
The vision community has started to build with the recently developed state space model, Mamba, as the new backbone for a range of tasks.
Medical Vision Generalist: Unifying Medical Imaging Tasks in Context
This study presents Medical Vision Generalist (MVG), the first foundation model capable of handling various medical imaging tasks -- such as cross-modal synthesis, image segmentation, denoising, and inpainting -- within a unified image-to-image generation framework.
ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning
This paper presents a new self-supervised video representation learning framework, ARVideo, which autoregressively predicts the next video token in a tailored sequence order.
Mamba-R: Vision Mamba ALSO Needs Registers
Similar to Vision Transformers, this paper identifies artifacts also present within the feature maps of Vision Mamba.
Beyond Finite Data: Towards Data-free Out-of-distribution Generalization via Extrapolation
Starting with the class of interest, we query the LLMs to extract relevant knowledge for these novel domains.
Filter & Align: Leveraging Human Knowledge to Curate Image-Text Data
However, the vast scale of these datasets inevitably introduces significant variability in data quality, which can adversely affect the model performance.
Rejuvenating image-GPT as Strong Visual Representation Learners
This paper enhances image-GPT (iGPT), one of the pioneering works that introduce autoregressive pretraining to predict the next pixels for visual representation learning.
NPF-200: A Multi-Modal Eye Fixation Dataset and Method for Non-Photorealistic Videos
Non-photorealistic videos are in demand with the wave of the metaverse, but lack of sufficient research studies.
SG-Former: Self-guided Transformer with Evolving Token Reallocation
At the heart of our approach is to utilize a significance map, which is estimated through hybrid-scale self-attention and evolves itself during training, to reallocate tokens based on the significance of each region.
DeepMIM: Deep Supervision for Masked Image Modeling
Deep supervision, which involves extra supervisions to the intermediate features of a neural network, was widely used in image classification in the early deep learning era since it significantly reduces the training difficulty and eases the optimization like avoiding gradient vanish over the vanilla training.
TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models
Our TinyMIM model of tiny size achieves 79. 6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget.
Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusion
Data lies at the core of modern deep learning.
DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation
Prior approaches, despite the promising results, have relied on either estimating dense attention to compute per-point matching, which is limited to only coarse scales due to the quadratic memory cost, or fixing the number of correspondences to achieve linear complexity, which lacks flexibility.
A Simple Data Mixing Prior for Improving Self-Supervised Learning
More notably, our SDMP is the first method that successfully leverages data mixing to improve (rather than hurt) the performance of Vision Transformers in the self-supervised setting.
The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation
Crossmodal knowledge distillation (KD) extends traditional knowledge distillation to the area of multimodal learning and demonstrates great success in various applications.
Glance to Count: Learning to Rank with Anchors for Weakly-supervised Crowd Counting
To enable training under this new setting, we convert the crowd count regression problem to a ranking potential prediction problem.
Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization
Multimodal fusion emerges as an appealing technique to improve model performances on many tasks.
Self-supervision through Random Segments with Autoregressive Coding (RandSAC)
We illustrate that randomized serialization of the segments significantly improves the performance and results in distribution over spatially-long (across-segments) and -short (within-segment) predictions which are effective for feature learning.
Shunted Self-Attention via Multi-Scale Token Aggregation
This novel merging scheme enables the self-attention to learn relationships between objects with different sizes and simultaneously reduces the token numbers and the computational cost.
Fine-grained Domain Adaptive Crowd Counting via Point-derived Segmentation
To this end, we propose to untangle \emph{domain-invariant} crowd and \emph{domain-specific} background from crowd images and design a fine-grained domain adaption method for crowd counting.
Reducing Spatial Labeling Redundancy for Semi-supervised Crowd Counting
In this way, we can transfer the original spatial labeling redundancy caused by individual similarities to effective supervision signals on the unlabeled regions.
Unifying Global-Local Representations in Salient Object Detection with Transformer
In this paper, we introduce a new attention-based encoder, vision transformer, into salient object detection to ensure the globalization of the representations from shallow to deep layers.
Co-advise: Cross Inductive Bias Distillation
Transformers recently are adapted from the community of natural language processing as a promising substitute of convolution-based neural networks for visual learning tasks.
Reciprocal Transformations for Unsupervised Video Object Segmentation
Additionally, to exclude the information of the moving background objects from motion features, our transformation module enables to reciprocally transform the appearance features to enhance the motion features, so as to focus on the moving objects with salient appearance while removing the co-moving outliers.
Ranked #12 on
Unsupervised Video Object Segmentation
on DAVIS 2016 val
Delving Deep Into Many-to-Many Attention for Few-Shot Video Object Segmentation
The key is to model the relationship between the query videos and the support images for propagating the object information.
Learning From the Master: Distilling Cross-Modal Advanced Knowledge for Lip Reading
To these ends, we introduce a trainable "master" network which ingests both audio signals and silent lip videos instead of a pretrained teacher.
On Feature Decorrelation in Self-Supervised Learning
In self-supervised representation learning, a common idea behind most of the state-of-the-art approaches is to enforce the robustness of the representations to predefined augmentations.
Multimodal Knowledge Expansion
The popularity of multimodal sensors and the accessibility of the Internet have brought us a massive amount of unlabeled multimodal data.
Ranked #69 on
Semantic Segmentation
on NYU Depth v2
TENet: Triple Excitation Network for Video Salient Object Detection
In this paper, we propose a simple yet effective approach, named Triple Excitation Network, to reinforce the training of video salient object detection (VSOD) from three aspects, spatial, temporal, and online excitations.
