Search Results for author:
Found 41 papers, 28 papers with code
FlowTok: Flowing Seamlessly Across Text and Image Tokens
While conventional approaches treat the text modality as a conditioning signal that gradually guides the denoising process from Gaussian noise to the target image modality, we explore a much simpler paradigm-directly evolving between text and image modalities through flow matching.
Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation
In this paper, we propose xAR, a generalized AR framework that extends the notion of a token to an entity X, which can represent an individual patch token, a cell (a $k\times k$ grouping of neighboring patches), a subsample (a non-local grouping of distant patches), a scale (coarse-to-fine resolution), or even a whole image.
Ranked #1 on
Image Generation
on ImageNet 256x256
Dictionary-based Framework for Interpretable and Consistent Object Parsing
Designed around Contrastive Components and Logical Constraints, CoCal rethinks existing cluster-based mask transformer architectures used in segmentation; Specifically, CoCal utilizes a set of dictionary components, with each component being explicitly linked to a specific semantic class.
COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation
This dataset sets a new benchmark for evaluating models on joint panoptic segmentation and grounded captioning tasks, addressing the need for high-quality, detailed image-text annotations in multi-modal learning.
Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens
Building on this, we introduce a family of text-to-image Masked Generative Models (MaskGen), trained exclusively on open data while achieving comparable performance to models trained on private data.
FlowAR: Scale-wise Autoregressive Image Generation Meets Flow Matching
Recently, in image generation, VAR proposes scale-wise autoregressive modeling, which extends the next token prediction to the next scale prediction, preserving the 2D structure of images.
Ranked #21 on
Image Generation
on ImageNet 256x256
Randomized Autoregressive Visual Generation
This paper presents Randomized AutoRegressive modeling (RAR) for visual generation, which sets a new state-of-the-art performance on the image generation task while maintaining full compatibility with language modeling frameworks.
Ranked #10 on
Image Generation
on ImageNet 256x256
MaskBit: Embedding-free Image Generation via Bit Tokens
Masked transformer models for class-conditional image generation have become a compelling alternative to diffusion models.
Ranked #7 on
Image Reconstruction
on ImageNet
Alleviating Distortion in Image Generation via Multi-Resolution Diffusion Models and Time-Dependent Layer Normalization
This paper presents innovative enhancements to diffusion models by integrating a novel multi-resolution network and time-dependent layer normalization.
Ranked #20 on
Image Generation
on ImageNet 256x256
An Image is Worth 32 Tokens for Reconstruction and Generation
At ImageNet 512 x 512 benchmark, TiTok not only outperforms state-of-the-art diffusion model DiT-XL/2 (gFID 2. 74 vs. 3. 04), but also reduces the image tokens by 64x, leading to 410x faster generation process.
Ranked #8 on
Image Reconstruction
on ImageNet
GaussianTalker: Speaker-specific Talking Head Synthesis via 3D Gaussian Splatting
In this paper, we propose GaussianTalker, a novel method for audio-driven talking head synthesis based on 3D Gaussian Splatting.
COCONut: Modernizing COCO Segmentation
By enhancing the annotation quality and expanding the dataset to encompass 383K images with more than 5. 18M panoptic masks, we introduce COCONut, the COCO Next Universal segmenTation dataset.
ViTamin: Designing Scalable Vision Models in the Vision-Language Era
To this end, we introduce ViTamin, a new vision models tailored for VLMs.
A Simple Video Segmenter by Tracking Objects Along Axial Trajectories
In this work, we present Axial-VS, a general and simple framework that enhances video segmenters by tracking objects along axial trajectories.
Ranked #2 on
Video Panoptic Segmentation
on VIPSeg
Towards Open-Ended Visual Recognition with Large Language Model
Localizing and recognizing objects in the open-ended physical world poses a long-standing challenge within the domain of machine perception.
3D TransUNet: Advancing Medical Image Segmentation through Vision Transformers
In this paper, we extend the 2D TransUNet architecture to a 3D network by building upon the state-of-the-art nnU-Net architecture, and fully exploring Transformers' potential in both the encoder and decoder design.
Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
The proposed FC-CLIP, benefits from the following observations: the frozen CLIP backbone maintains the ability of open-vocabulary classification and can also serve as a strong mask generator, and the convolutional CLIP generalizes well to a larger input resolution than the one used during contrastive image-text pretraining.
Ranked #1 on
Open Vocabulary Semantic Segmentation
on Cityscapes
Open Vocabulary Panoptic Segmentation
Open Vocabulary Semantic Segmentation
+2
ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentation
This paper presents a new mechanism to facilitate the training of mask transformers for efficient panoptic segmentation, democratizing its deployment.
Compositor: Bottom-up Clustering and Compositing for Robust Part and Object Segmentation
Compositor achieves state-of-the-art performance on PartImageNet and Pascal-Part by outperforming previous methods by around 0. 9% and 1. 3% on PartImageNet, 0. 4% and 1. 7% on Pascal-Part in terms of part and object mIoU and demonstrates better robustness against occlusion by around 4. 4% and 7. 1% on part and object respectively.
Video-kMaX: A Simple Unified Approach for Online and Near-Online Video Panoptic Segmentation
The meta architecture of the proposed Video-kMaX consists of two components: within clip segmenter (for clip-level segmentation) and cross-clip associater (for association beyond clips).
A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision
A key finding is that a small decoder learned on top of a frozen pretrained encoder works surprisingly well.
CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans
Human readers or radiologists routinely perform full-body multi-organ multi-disease detection and diagnosis in clinical practice, while most medical AI systems are built to focus on single organs with a narrow list of a few diseases.
MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Models
The tiny-MOAT family is also benchmarked on downstream tasks, serving as a baseline for the community.
Ranked #1 on
Object Detection
on MS COCO
kMaX-DeepLab: k-means Mask Transformer
However, we observe that most existing transformer-based vision models simply borrow the idea from NLP, neglecting the crucial difference between languages and images, particularly the extremely large sequence length of spatially flattened pixel features.
Ranked #2 on
Panoptic Segmentation
on COCO test-dev
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation
We propose Clustering Mask Transformer (CMT-DeepLab), a transformer-based framework for panoptic segmentation designed around clustering.
Ranked #6 on
Panoptic Segmentation
on COCO test-dev
TubeFormer-DeepLab: Video Mask Transformer
We present TubeFormer-DeepLab, the first attempt to tackle multiple core video segmentation tasks in a unified manner.
PartImageNet: A Large, High-Quality Dataset of Parts
To help address this problem, we propose PartImageNet, a large, high-quality dataset with part segmentation annotations.
DeepLab2: A TensorFlow Library for Deep Labeling
DeepLab2 is a TensorFlow library for deep labeling, aiming to provide a state-of-the-art and easy-to-use TensorFlow codebase for general dense pixel prediction problems in computer vision.
Glance-and-Gaze Vision Transformer
It is motivated by the Glance and Gaze behavior of human beings when recognizing objects in natural scenes, with the ability to efficiently model both long-range dependencies and local context.
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Medical image segmentation is an essential prerequisite for developing healthcare systems, especially for disease diagnosis and treatment planning.
Ranked #6 on
Medical Image Segmentation
on ACDC
Mask Guided Matting via Progressive Refinement Network
We propose Mask Guided (MG) Matting, a robust matting framework that takes a general coarse mask as guidance.
Can Temporal Information Help with Contrastive Self-Supervised Learning?
To this end, we present Temporal-aware Contrastive self-supervised learningTaCo, as a general paradigm to enhance video CSL.
Shape-Texture Debiased Neural Network Training
To prevent models from exclusively attending on a single cue in representation learning, we augment training data with images with conflicting shape and texture information (eg, an image of chimpanzee shape but with lemon texture) and, most importantly, provide the corresponding supervisions from shape and texture simultaneously.
Ranked #656 on
Image Classification
on ImageNet
Neural Architecture Search for Lightweight Non-Local Networks
However, it has been rarely explored to embed the NL blocks in mobile neural networks, mainly due to the following challenges: 1) NL blocks generally have heavy computation cost which makes it difficult to be applied in applications where computational resources are limited, and 2) it is an open problem to discover an optimal configuration to embed NL blocks into mobile neural networks.
Ranked #60 on
Neural Architecture Search
on ImageNet
CAKES: Channel-wise Automatic KErnel Shrinking for Efficient 3D Networks
3D Convolution Neural Networks (CNNs) have been widely applied to 3D scene understanding, such as video analysis and volumetric image recognition.
Detecting Pancreatic Ductal Adenocarcinoma in Multi-phase CT Scans via Alignment Ensemble
Pancreatic ductal adenocarcinoma (PDAC) is one of the most lethal cancers among the population.
When Radiology Report Generation Meets Knowledge Graph
In addition, we proposed a new evaluation metric for radiology image reporting with the assistance of the same composed graph.
C2FNAS: Coarse-to-Fine Neural Architecture Search for 3D Medical Image Segmentation
3D convolution neural networks (CNN) have been proved very successful in parsing organs or tumours in 3D medical images, but it remains sophisticated and time-consuming to choose or design proper 3D networks given different task contexts.
Thickened 2D Networks for Efficient 3D Medical Image Segmentation
With this design, we achieve a higher performance while maintaining a lower inference latency on a few abdominal organs from CT scans, in particular when the organ has a peculiar 3D shape and thus strongly requires contextual information, demonstrating our method's effectiveness and ability in capturing 3D information.
Recurrent Saliency Transformation Network: Incorporating Multi-Stage Visual Cues for Small Organ Segmentation
The key innovation is a saliency transformation module, which repeatedly converts the segmentation probability map from the previous iteration as spatial weights and applies these weights to the current iteration.
Ranked #1 on
Pancreas Segmentation
on TCIA Pancreas-CT Dataset
