Search Results for author:
Found 88 papers, 47 papers with code
PromptFix: You Prompt and We Fix the Photo
To address these limitations, we propose PromptFix, a comprehensive framework that enables diffusion models to follow human instructions to perform a wide variety of image-processing tasks.
Zero-Reference Low-Light Enhancement via Physical Quadruple Priors
This prior serves as the bridge between normal and low-light images.
Learning Position-Aware Implicit Neural Network for Real-World Face Inpainting
Face inpainting requires the model to have a precise global understanding of the facial position structure.
ViCo: Engaging Video Comment Generation with Human Preference Rewards
Experiments on ViCo-20k show that the comments generated by our ViCo model exhibit the best performance in terms of both quantitative and qualitative results, particularly when engagement is considered.
Improving Diversity in Zero-Shot GAN Adaptation with Semantic Variations
Due to the data absence, the textual description of the target domain and the vision-language models, e. g., CLIP, are utilized to effectively guide the generator.
MobileVidFactory: Automatic Diffusion-Based Social Media Video Generation for Mobile Devices from Text
In the basic generation, we take advantage of the pretrained image diffusion model, and adapt it to a high-quality open-domain vertical video generator for mobile devices.
SINC: Self-Supervised In-Context Learning for Vision-Language Tasks
To this end, we raise a question: ``How can we enable in-context learning without relying on the intrinsic in-context ability of large language models?".
Learning Profitable NFT Image Diffusions via Multiple Visual-Policy Guided Reinforcement Learning
The proposed framework consists of a large language model (LLM), a diffusion-based image generator, and a series of visual rewards by design.
MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images
To generate videos, we extend the capabilities of a pretrained text-to-image diffusion model through a two-stage process.
Transferring Foundation Models for Generalizable Robotic Manipulation
In this paper, we propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models, to condition robot manipulation tasks.
AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation
We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks, such as making a smiley face using building blocks.
Solving Diffusion ODEs with Optimal Boundary Conditions for Better Image Super-Resolution
Diffusion models, as a kind of powerful generative model, have given impressive results on image super-resolution (SR) tasks.
Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation
Moreover, to fully unlock model capabilities for high-quality video generation and promote the development of the field, we curate a large-scale and open-source video dataset called HD-VG-130M.
Ranked #1 on
Text-to-Video Generation
on WebVid
NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation.
Learning Data-Driven Vector-Quantized Degradation Model for Animation Video Super-Resolution
Existing real-world video super-resolution (VSR) methods focus on designing a general degradation pipeline for open-domain videos while ignoring data intrinsic characteristics which strongly limit their performance when applying to some specific domains (eg., animation videos).
Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation
Language-guided image generation has achieved great success nowadays by using diffusion models.
Learning Spatiotemporal Frequency-Transformer for Low-Quality Video Super-Resolution
Video Super-Resolution (VSR) aims to restore high-resolution (HR) videos from low-resolution (LR) videos.
Ranked #4 on
Video Super-Resolution
on REDS4- 4x upscaling
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
To generate joint audio-video pairs, we propose a novel Multi-Modal Diffusion model (i. e., MM-Diffusion), with two-coupled denoising autoencoders.
Weakly-supervised Pre-training for 3D Human Pose Estimation via Perspective Knowledge
Based on MCPC, we propose a weakly-supervised pre-training (WSP) strategy to distinguish the depth relationship between two points in an image.
Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning
Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks.
Ranked #2 on
Video Retrieval
on QuerYD
(using extra training data)
Fine-Grained Image Style Transfer with Visual Transformers
Such a design usually destroys the spatial information of the input images and fails to transfer fine-grained style patterns into style transfer results.
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment
and 2) how to mitigate the impact of these factors?
Ranked #2 on
Video Retrieval
on MSR-VTT-1kA
(using extra training data)
AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation
To address this issue, we propose a Prompt-based Cross-Modal Generation Framework (PCM-Frame) to leverage two powerful pre-trained models, including CLIP and StyleGAN.
4D LUT: Learnable Context-Aware 4D Lookup Table for Image Enhancement
In particular, we first introduce a lightweight context encoder and a parameter encoder to learn a context map for the pixel-level category and a group of image-adaptive coefficients, respectively.
Ranked #7 on
Image Enhancement
on MIT-Adobe 5k
(SSIM on proRGB metric)
Language-Guided Face Animation by Recurrent StyleGAN-based Generator
Specifically, we propose a recurrent motion generator to extract a series of semantic and motion information from the language and feed it along with visual information to a pre-trained StyleGAN to generate high-quality frames.
Exploring Anchor-based Detection for Ego4D Natural Language Query
In this paper we provide the technique report of Ego4D natural language query challenge in CVPR 2022.
GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-training
In contrast to the mainstream VLP methods, we highlight that two routinely applied steps during pre-training have crucial impact on the performance of the pre-trained model: in-batch hard negative sampling for image-text matching (ITM) and assigning the large masking probability for the masked language modeling (MLM).
Learning Spatiotemporal Frequency-Transformer for Compressed Video Super-Resolution
First, we divide a video frame into patches, and transform each patch into DCT spectral maps in which each channel represents a frequency band.
Ranked #5 on
Video Super-Resolution
on REDS4- 4x upscaling
Expanding Language-Image Pretrained Models for General Video Recognition
Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios.
Ranked #9 on
Zero-Shot Action Recognition
on Kinetics
TinyViT: Fast Pretraining Distillation for Small Vision Transformers
It achieves a top-1 accuracy of 84. 8% on ImageNet-1k with only 21M parameters, being comparable to Swin-B pretrained on ImageNet-21k while using 4. 2 times fewer parameters.
Ranked #135 on
Image Classification
on ImageNet
TTVFI: Learning Trajectory-Aware Transformer for Video Frame Interpolation
In particular, we formulate the warped features with inconsistent motions as query tokens, and formulate relevant regions in a motion trajectory from two original consecutive frames into keys and values.
Degradation-Guided Meta-Restoration Network for Blind Super-Resolution
The extractor estimates the degradations in LR inputs and guides the meta-restoration modules to predict restoration parameters for different degradations on-the-fly.
MiniViT: Compressing Vision Transformers with Weight Multiplexing
The central idea of MiniViT is to multiplex the weights of consecutive transformer blocks.
Ranked #213 on
Image Classification
on ImageNet
Learning Trajectory-Aware Transformer for Video Super-Resolution
Existing approaches usually align and aggregate video frames from limited adjacent frames (e. g., 5 or 7 frames), which prevents these approaches from satisfactory results.
Ranked #4 on
Video Super-Resolution
on UDM10 - 4x upscaling
Searching the Search Space of Vision Transformer
Vision Transformer has shown great visual representation power in substantial vision tasks such as recognition and detection, and thus been attracting fast-growing efforts on manually designing more effective architectures.
Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
To enable VL pre-training, we jointly optimize the HD-VILA model by a hybrid Transformer that learns rich spatiotemporal features, and a multimodal Transformer that enforces interactions of the learned video features with diversified texts.
Ranked #16 on
Video Retrieval
on MSR-VTT
Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers
Given a sequence of style tokens, the TokenGAN is able to control the image synthesis by assigning the styles to the content tokens by attention mechanism with a Transformer.
A Picture is Worth a Thousand Words: A Unified System for Diverse Captions and Rich Images Generation
In this work, we demonstrate such an AI creation system to produce both diverse captions and rich images.
Learning Fine-Grained Motion Embedding for Landscape Animation
To tackle this problem, we propose a model named FGLA to generate high-quality and realistic videos by learning Fine-Grained motion embedding for Landscape Animation.
Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessment
And the reference space is optimized to capture deep image priors that are useful for quality assessment.
Domain-Aware Universal Style Transfer
To this end, we design a novel domainness indicator that captures the domainness value from the texture and structural features of reference images.
Reference-based Defect Detection Network
To solve the partial visual confusion issue, we propose to leverage the carried context information of context reference, which is the concentric bigger box of each region proposal, to perform more accurate region classification and regression.
Rethinking and Improving Relative Position Encoding for Vision Transformer
We then propose new relative position encoding methods dedicated to 2D images, called image RPE (iRPE).
Ranked #152 on
Object Detection
on COCO minival
AutoFormer: Searching Transformers for Visual Recognition
Specifically, the performance of these subnets with weights inherited from the supernet is comparable to those retrained from scratch.
Ranked #1 on
Fine-Grained Image Classification
on Oxford 102 Flowers
(Top 1 Accuracy metric)
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
To tackle this, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment.
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training
To tackle this, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment.
LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search
Object tracking has achieved significant progress over the past few years.
Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
As region-based visual features usually represent parts of an image, it is challenging for existing vision-language models to fully understand the semantics from paired natural languages.
Ranked #5 on
Visual Entailment
on SNLI-VE val
3D Human Body Reshaping with Anthropometric Modeling
First, we calculate full-body anthropometric parameters from limited user inputs by imputation technique, and thus essential anthropometric parameters for 3D body reshaping can be obtained.
Aggregated Contextual Transformations for High-Resolution Image Inpainting
For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task.
Ranked #9 on
Image Inpainting
on Places2
One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking
In this paper, we propose a one-shot neural ensemble architecture search (NEAS) solution that addresses the two challenges.
Learning Spatio-Temporal Transformer for Visual Tracking
In this paper, we present a new tracking architecture with an encoder-decoder transformer as the key component.
Ranked #21 on
Visual Object Tracking
on TrackingNet
Multi-Scale 2D Temporal Adjacent Networks for Moment Localization with Natural Language
It is a challenging problem because a target moment may take place in the context of other temporal moments in the untrimmed video.
Learning Semantic-aware Normalization for Generative Adversarial Networks
Such a model disentangles latent factors according to the semantic of feature channels by channel-/group- wise fusion of latent codes and feature channels.
Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
The experiments on ImageNet verify such path distillation method can improve the convergence ratio and performance of the hypernetwork, as well as boosting the training of subnetworks.
Revisiting Anchor Mechanisms for Temporal Action Localization
To address this problem, this paper proposes a novel anchor-free action localization module that assists action localization by temporal points.
Learning Joint Spatial-Temporal Transformations for Video Inpainting
In this paper, we propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting.
Ranked #5 on
Seeing Beyond the Visible
on KITTI360-EX
Cyclic Differentiable Architecture Search
First, the search network generates an initial architecture for evaluation, and the weights of the evaluation network are optimized.
Ranked #18 on
Neural Architecture Search
on NAS-Bench-201, CIFAR-10
Ocean: Object-aware Anchor-free Tracking
In this paper, we propose a novel object-aware anchor-free network to address this issue.
Ranked #2 on
Visual Object Tracking
on VOT2019
Learning Texture Transformer Network for Image Super-Resolution
In this paper, we propose a novel Texture Transformer Network for Image Super-Resolution (TTSR), in which the LR and Ref images are formulated as queries and keys in a transformer, respectively.
NTIRE 2020 Challenge on Perceptual Extreme Super-Resolution: Methods and Results
This paper reviews the NTIRE 2020 challenge on perceptual extreme super-resolution with focus on proposed solutions and results.
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks.
Learning Sparse 2D Temporal Adjacent Networks for Temporal Action Localization
In this report, we introduce the Winner method for HACS Temporal Action Localization Challenge 2019.
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
We address the problem of retrieving a specific moment from an untrimmed video by a query sentence.
Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences
A storyboard is a sequence of images to illustrate a story containing multiple sentences, which has been a key process to create different story products.
Learning Deep Bilinear Transformation for Fine-grained Image Representation
However, the computational cost to learn pairwise interactions between deep feature channels is prohibitively expensive, which restricts this powerful transformation to be used in deep neural networks.
Learning Rich Image Region Representation for Visual Question Answering
We propose to boost VQA by leveraging more powerful feature extractors by improving the representation ability of both visual and text features and the ensemble of models.
360-Indoor: Towards Learning Real-World Objects in 360° Indoor Equirectangular Images
In this paper, our goal is to provide a standard dataset to facilitate the vision and machine learning communities in 360{\deg} domain.
WSOD2: Learning Bottom-up and Top-down Objectness Distillation forWeakly-supervised Object Detection
We study on weakly-supervised object detection (WSOD)which plays a vital role in relieving human involvement fromobject-level annotations.
Ranked #12 on
Weakly Supervised Object Detection
on PASCAL VOC 2007
WSOD^2: Learning Bottom-up and Top-down Objectness Distillation for Weakly-supervised Object Detection
We study on weakly-supervised object detection (WSOD) which plays a vital role in relieving human involvement from object-level annotations.
Learn to Scale: Generating Multipolar Normalized Density Maps for Crowd Counting
Dense crowd counting aims to predict thousands of human instances from an image, by calculating integrals of a density map over image pixels.
Activitynet 2019 Task 3: Exploring Contexts for Dense Captioning Events in Videos
The overall system achieves the state-of-the-art performance on the dense-captioning events in video task with 9. 91 METEOR score on the challenge testing set.
From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots
However, a picture tells a thousand words, which makes multi-lingual sentences pivoted by the same image noisy as mutual translations and thus hinders the translation model learning.
Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting
As the missing content can be filled by attention transfer from deep to shallow in a pyramid fashion, both visual and semantic coherence for image inpainting can be ensured.
Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition
Learning subtle yet discriminative features (e. g., beak and eyes for a bird) plays a significant role in fine-grained image recognition.
Ranked #1 on
Fine-Grained Image Classification
on iNaturalist
Fine-Grained Image Classification
Fine-Grained Image Recognition
Deep Attention Neural Tensor Network for Visual Question Answering
First, we model one of the pairwise interaction (e. g., image and question) by bilinear features, which is further encoded with the third dimension (e. g., answer) to be a triplet by bilinear tensor product.
Image Inspired Poetry Generation in XiaoIce
This is one of the few attempts to generate poetry from images.
DA-GAN: Instance-Level Image Translation by Deep Attention Generative Adversarial Networks
Specifically, we jointly learn a deep attention encoder, and the instance-level correspondences could be consequently discovered through attending on the learned instances.
Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training
Extensive experiments are conducted with 8K images, among which 1. 5K image are randomly picked for evaluation.
DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)
Specifically, we jointly learn a deep attention encoder, and the instancelevel correspondences could be consequently discovered through attending on the learned instance pairs.
Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
Most existing approaches adopt the pipeline of representing an image via pre-trained CNNs, and then using the uninterpretable CNN features in conjunction with the question to predict the answer.
Self-view Grounding Given a Narrated 360° Video
The negative log reconstruction loss of the reverse sentence (referred to as "irrelevant loss") is jointly minimized to encourage the reverse sentence to be different from the given sentence.
Learning Multi-Attention Convolutional Neural Network for Fine-Grained Image Recognition
Two losses are proposed to guide the multi-task learning of channel grouping and part classification, which encourages MA-CNN to generate more discriminative parts from feature channels and learn better fine-grained features from parts in a mutual reinforced way.
Ranked #23 on
Fine-Grained Image Classification
on CUB-200-2011
Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition
The learning at each scale consists of a classification sub-network and an attention proposal sub-network (APN).
Fine-Grained Image Classification
Fine-Grained Image Recognition
+1
Multi-Level Attention Networks for Visual Question Answering
To solve the challenges, we propose a multi-level attention network for visual question answering that can simultaneously reduce the semantic gap by semantic attention and benefit fine-grained spatial inference by visual attention.
Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
The domain critic assesses whether the generated sentences are indistinguishable from sentences in the target domain.
Storytelling of Photo Stream with Bidirectional Multi-thread Recurrent Neural Network
Second, by using sGRU as basic units, the BMRNN is trained to align the local storylines into the global sequential timeline.
Relaxing From Vocabulary: Robust Weakly-Supervised Deep Learning for Vocabulary-Free Image Tagging
The development of deep learning has empowered machines with comparable capability of recognizing limited image categories to human beings.
