Search Results for author:
Found 44 papers, 29 papers with code
COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation
This dataset sets a new benchmark for evaluating models on joint panoptic segmentation and grounded captioning tasks, addressing the need for high-quality, detailed image-text annotations in multi-modal learning.
Dual Diffusion for Unified Image Generation and Understanding
Diffusion models have gained tremendous success in text-to-image generation, yet still lag behind with visual understanding tasks, an area dominated by autoregressive vision-language models.
Fast Prompt Alignment for Text-to-Image Generation
Text-to-image generation has advanced rapidly, yet aligning complex textual prompts with generated visuals remains challenging, especially with intricate object relationships and fine-grained details.
VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos
We present a framework for learning to generate background music from video inputs.
LSVOS Challenge Report: Large-scale Complex and Long Video Object Segmentation
Despite the promising performance of current video segmentation models on existing benchmarks, these models still struggle with complex scenes.
Autoregressive Pretraining with Mamba in Vision
The vision community has started to build with the recently developed state space model, Mamba, as the new backbone for a range of tasks.
Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters
Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore.
Video Recognition in Portrait Mode
While existing datasets mainly comprise landscape mode videos, our paper seeks to introduce portrait mode videos to the research community and highlight the unique challenges associated with this video format.
Shot2Story20K: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
A human need to capture both the event in every shot and associate them together to understand the story behind it.
Ranked #1 on
video narration captioning
on Shot2Story20K
Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decoupling
This paper proposes Video-Teller, a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment to significantly enhance the video-to-text generation task.
Selective Feature Adapter for Dense Vision Transformers
Specifically, SFA consists of external adapters and internal adapters which are sequentially operated over a transformer model.
The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filtering
DataComp is a new benchmark dedicated to evaluating different methods for data filtering.
Learning Dynamic Query Combinations for Transformer-based Object Detection and Segmentation
Transformer-based detection and segmentation methods use a list of learned detection queries to retrieve information from the transformer network and learn to predict the location and category of one specific object from each query.
Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
Vision-language models such as CLIP learn a generic text-image embedding from large-scale training data.
Exploring the Role of Audio in Video Captioning
Recent focus in video captioning has been on designing architectures that can consume both video and text modalities, and using large-scale video datasets with text transcripts for pre-training, such as HowTo100M.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
$R^{2}$Former: Unified $R$etrieval and $R$eranking Transformer for Place Recognition
Visual Place Recognition (VPR) estimates the location of query images by matching them with images in a reference database.
FAQ: Feature Aggregated Queries for Transformer-based Video Object Detectors
With Transformerbased object detectors getting a better performance on the image domain tasks, recent works began to extend those methods to video object detection.
R2Former: Unified Retrieval and Reranking Transformer for Place Recognition
Visual Place Recognition (VPR) estimates the location of query images by matching them with images in a reference database.
Revisiting Training-free NAS Metrics: An Efficient Training-based Method
In this paper, we revisit these training-free metrics and find that: (1) the number of parameters (\#Param), which is the most straightforward training-free metric, is overlooked in previous works but is surprisingly effective, (2) recent training-free metrics largely rely on the \#Param information to rank networks.
Dynamic Proposals for Efficient Object Detection
Object detection is a basic computer vision task to loccalize and categorize objects in a given image.
Robust High-Resolution Video Matting with Temporal Guidance
We introduce a robust, real-time, high-resolution human video matting method that achieves new state-of-the-art performance.
HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
Last, we proposed an efficient fine-grained search strategy to train HR-NAS, which effectively explores the search space, and finds optimal architectures given various tasks and computation resources.
Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization
Post-training quantization methods use a set of calibration data to compute quantization ranges for network parameters and activations.
Progressive Temporal Feature Alignment Network for Video Inpainting
To achieve this goal, it is necessary to find correspondences from neighbouring frames to faithfully hallucinate the unknown content.
Learning Versatile Neural Architectures by Propagating Network Codes
(4) Thorough studies of NCP on inter-, cross-, and intra-tasks highlight the importance of cross-task neural architecture design, i. e., multitask neural architectures and architecture transferring between different tasks.
DeepViT: Towards Deeper Vision Transformer
In this paper, we show that, unlike convolution neural networks (CNNs)that can be improved by stacking more convolutional layers, the performance of ViTs saturate fast when scaled to be deeper.
Ranked #464 on
Image Classification
on ImageNet
AutoSpace: Neural Architecture Search with Less Human Interference
Current neural architecture search (NAS) algorithms still require expert knowledge and effort to design a search space for network construction.
CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation
Video instance segmentation is a complex task in which we need to detect, segment, and track each object for any given video.
Ranked #32 on
Video Instance Segmentation
on YouTube-VIS validation
FracBits: Mixed Precision Quantization via Fractional Bit-Widths
Model quantization helps to reduce model size and latency of deep neural networks.
Neural Architecture Search for Lightweight Non-Local Networks
However, it has been rarely explored to embed the NL blocks in mobile neural networks, mainly due to the following challenges: 1) NL blocks generally have heavy computation cost which makes it difficult to be applied in applications where computational resources are limited, and 2) it is an open problem to discover an optimal configuration to embed NL blocks into mobile neural networks.
Ranked #60 on
Neural Architecture Search
on ImageNet
Towards Efficient Training for Neural Network Quantization
To deal with this problem, we propose a simple yet effective technique, named scale-adjusted training (SAT), to comply with the discovered rules and facilitates efficient training.
AdaBits: Neural Network Quantization with Adaptive Bit-Widths
With our proposed techniques applied on a bunch of models including MobileNet-V1/V2 and ResNet-50, we demonstrate that bit-width of weights and activations is a new option for adaptively executable deep neural networks, offering a distinct opportunity for improved accuracy-efficiency trade-off as well as instant adaptation according to the platform constraints in real-world applications.
AtomNAS: Fine-Grained End-to-End Neural Architecture Search
We propose a fine-grained search space comprised of atomic blocks, a minimal search unit that is much smaller than the ones used in recent NAS algorithms.
Ranked #61 on
Neural Architecture Search
on ImageNet
Rethinking Neural Network Quantization
To deal with this problem, we propose a simple yet effective technique, named scale-adjusted training (SAT), to comply with the discovered rules and facilitates efficient training.
Weakly Supervised Body Part Segmentation with Pose based Part Priors
The core idea is first converting the sparse weak labels such as keypoints to the initial estimate of body part masks, and then iteratively refine the part mask predictions.
Video Instance Segmentation
The goal of this new task is simultaneous detection, segmentation and tracking of instances in videos.
Ranked #40 on
Video Instance Segmentation
on YouTube-VIS validation
Context-Aware Zero-Shot Recognition
We present a novel problem setting in zero-shot learning, zero-shot object recognition and detection in the context.
Streamlined Dense Video Captioning
Dense video captioning is an extremely challenging task since accurate and coherent description of events in a video requires holistic understanding of video contents as well as contextual reasoning of individual events.
Slimmable Neural Networks
Instead of training individual networks with different width configurations, we train a shared network with switchable batch normalization.
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
End-to-end sequential learning to explore spatialtemporal features for video segmentation is largely limited by the scale of available video segmentation datasets, i. e., even the largest video segmentation dataset only contains 90 short video clips.
YouTube-VOS: Sequence-to-Sequence Video Object Segmentation
End-to-end sequential learning to explore spatial-temporal features for video segmentation is largely limited by the scale of available video segmentation datasets, i. e., even the largest video segmentation dataset only contains 90 short video clips.
Ranked #12 on
Video Object Segmentation
on YouTube-VOS 2018
(F-Measure (Unseen) metric)
Efficient Video Object Segmentation via Network Modulation
Video object segmentation targets at segmenting a specific object throughout a video sequence, given only an annotated first frame.
Ranked #1 on
One-shot visual object segmentation
on YouTube-VOS 2018
(Jaccard (Seen) metric)
Dense Captioning with Joint Inference and Visual Context
The goal is to densely detect visual concepts (e. g., objects, object parts, and interactions between them) from images, labeling each with a short descriptive phrase.
A Large-Scale Car Dataset for Fine-Grained Categorization and Verification
Updated on 24/09/2015: This update provides preliminary experiment results for fine-grained classification on the surveillance data of CompCars.
Ranked #6 on
Fine-Grained Image Classification
on CompCars
