Search Results for author:
Found 37 papers, 24 papers with code
Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning
The AR property makes Selftok fundamentally distinct from traditional spatial tokens in the following two key ways: - Selftok offers an elegant and minimalist approach to unify diffusion and AR for vision-language models (VLMs): By representing images with Selftok tokens, we can train a VLM using a purely discrete autoregressive architecture -- like that in LLMs -- without requiring additional modules or training objectives.
Automated Movie Generation via Multi-Agent CoT Planning
Existing long-form video generation frameworks lack automated planning, requiring manual input for storylines, scenes, cinematography, and character interactions, resulting in high costs and inefficiencies.
ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting
Despite the significant advancements in text-to-image (T2I) generative models, users often face a trial-and-error challenge in practical scenarios.
MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation
Recent advancements in video generation models, like Stable Video Diffusion, show promising results, but primarily focus on short, single-scene videos.
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
In terms of efficiency, ZipCache also showcases a $37. 3\%$ reduction in prefill-phase latency, a $56. 9\%$ reduction in decoding-phase latency, and a $19. 8\%$ reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of $4096$.
DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data
Instance segmentation is data-hungry, and as model capacity increases, data scale becomes crucial for improving the accuracy.
Ranked #7 on
Instance Segmentation
on LVIS v1.0 val
VimTS: A Unified Video and Image Text Spotter for Enhancing the Cross-domain Generalization
Typically, we propose a Prompt Queries Generation Module and a Tasks-aware Adapter to effectively convert the original single-task model into a multi-task model suitable for both image and video scenarios with minimal additional parameters.
DragAnything: Motion Control for Anything using Entity Representation
We introduce DragAnything, which utilizes a entity representation to achieve motion control for any object in controllable video generation.
Towards Accurate Post-training Quantization for Reparameterized Models
Our framework outperforms previous methods by approximately 1\% for 8-bit PTQ and 2\% for 6-bit PTQ, showcasing its superior performance.
ControlCap: Controllable Region-level Captioning
The multimodal model is constrained to generate captions within a few sub-spaces containing the control words, which increases the opportunity of hitting less frequent captions, alleviating the caption degeneration issue.
Ranked #1 on
Dense Captioning
on Visual Genome
Continual Learning for Image Segmentation with Dynamic Query
Image segmentation based on continual learning exhibits a critical drop of performance, mainly due to catastrophic forgetting and background shift, as they are required to incorporate new classes continually.
Paragraph-to-Image Generation with Information-Enriched Diffusion Model
In this paper, we introduce an information-enriched diffusion model for paragraph-to-image generation task, termed ParaDiffusion, which delves into the transference of the extensive semantic comprehension capabilities of large language models to the task of image generation.
MotionDirector: Motion Customization of Text-to-Video Diffusion Models
Given a set of video clips of the same motion concept, the task of Motion Customization is to adapt existing text-to-video diffusion models to generate videos with this motion.
Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM
Fine-grained quantization has smaller quantization loss, consequently achieving superior performance.
EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models
In this paper, we introduce a data-free and parameter-efficient fine-tuning framework for low-bit diffusion models, dubbed EfficientDM, to achieve QAT-level performance with PTQ-like efficiency.
DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models
To showcase the power of the proposed approach, we generate datasets with rich dense pixel-wise labels for a wide range of downstream tasks, including semantic segmentation, instance segmentation, and depth estimation.
Generative Prompt Model for Weakly Supervised Object Localization
During training, GenPromp converts image category labels to learnable prompt embeddings which are fed to a generative model to conditionally recover the input image with noise and learn representative embeddings.
Ranked #1 on
Weakly-Supervised Object Localization
on CUB-200-2011
(Top-1 Localization Accuracy metric, using extra
training data)
Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community.
PTQD: Accurate Post-Training Quantization for Diffusion Models
To address these challenges, we propose a unified formulation for the quantization noise and diffusion perturbed noise in the quantized denoising process.
A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension
Most existing cross-modal language-to-video retrieval (VR) research focuses on single-modal input from video, i. e., visual representation, while the text is omnipresent in human environments and frequently critical to understand video.
FlowText: Synthesizing Realistic Scene Text Video with Optical Flow Estimation
This paper introduces a novel video text synthesis technique called FlowText, which utilizes optical flow estimation to synthesize a large amount of text video data at a low cost for training robust video text spotters.
ICDAR 2023 Video Text Reading Competition for Dense and Small Text
In this competition report, we establish a video text reading benchmark, DSText, which focuses on dense and small text reading challenges in the video with various scenarios.
DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
In contrast, synthetic data can be freely available using a generative model (e. g., DALL-E, Stable Diffusion).
POSGen: Personalized Opening Sentence Generation for Online Insurance Sales
Such personalized opening sentence generation is challenging because (i) there are limited historical samples for conversation topic recommendation in online insurance sales and (ii) existing text generation schemes often fail to support customized topic ordering based on user preferences.
BiViT: Extremely Compressed Binary Vision Transformers
To solve this, we propose Softmax-aware Binarization, which dynamically adapts to the data distribution and reduces the error caused by binarization.
BiViT: Extremely Compressed Binary Vision Transformer
To solve this, we propose Softmax-aware Binarization, which dynamically adapts to the data distribution and reduces the error caused by binarization.
Explore Faster Localization Learning For Scene Text Detection
Generally pre-training and long-time training computation are necessary for obtaining a good-performance text detector based on deep networks.
Binarizing by Classification: Is soft function really necessary?
Extensive experiments demonstrate that the proposed method yields surprising performance both in image classification and human pose estimation tasks.
Ranked #1 on
Binarization
on ImageNet
(Top 1 Accuracy metric)
Data-Free Quantization with Accurate Activation Clipping and Adaptive Batch Normalization
In this paper, we present a simple yet effective data-free quantization method with accurate activation clipping and adaptive batch normalization.
End-to-End Video Text Spotting with Transformer
Recent video text spotting methods usually require the three-staged pipeline, i. e., detecting text in individual images, recognizing localized text, tracking text streams with post-processing to generate final results.
Contrastive Learning of Semantic and Visual Representations for Text Tracking
Semantic representation is of great benefit to the video text tracking(VTT) task that requires simultaneously classifying, detecting, and tracking texts in the video.
A Bilingual, OpenWorld Video Text Dataset and End-to-end Video Text Spotter with Transformer
Most existing video text spotting benchmarks focus on evaluating a single language and scenario with limited data.
EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling
Extra rich non-paired single-modal text data is used for boosting the generalization of text branch.
Polygon-free: Unconstrained Scene Text Detection with Box Annotations
For example, without using polygon annotations, PSENet achieves an 80. 5% F-score on TotalText [3] (vs. 80. 9% of fully supervised counterpart), 31. 1% better than training directly with upright bounding box annotations, and saves 80%+ labeling costs.
Synthetic-to-Real Unsupervised Domain Adaptation for Scene Text Detection in the Wild
To address the severe domain distribution mismatch, we propose a synthetic-to-real domain adaptation method for scene text detection, which transfers knowledge from synthetic data (source domain) to real data (target domain).
Multi-task Sentence Encoding Model for Semantic Retrieval in Question Answering Systems
Sentence matching is an essential task in the QA systems and is usually reformulated as a Paraphrase Identification (PI) problem.
Ranked #13 on
Paraphrase Identification
on Quora Question Pairs
(Accuracy metric)
TextCohesion: Detecting Text for Arbitrary Shapes
In this paper, we propose a pixel-wise method named TextCohesion for scene text detection, which splits a text instance into five key components: a Text Skeleton and four Directional Pixel Regions.
Ranked #1 on
Curved Text Detection
on SCUT-CTW1500
