Search Results for author:
Found 47 papers, 21 papers with code
VCoME: Verbal Video Composition with Multimodal Editing Effects
To address this task, we propose VCoME, a general framework that employs a large multimodal model to generate editing effects for video composition.
Hierarchical Memory for Long Video QA
This paper describes our champion solution to the LOVEU Challenge @ CVPR'24, Track 1 (Long Video VQA).
Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams
Our model is able to process extremely long video streams in real-time and respond to user queries simultaneously.
Ranked #1 on
Zero-Shot Video Question Answer
on MSRVTT-QA
The SkatingVerse Workshop & Challenge: Methods and Results
The SkatingVerse Workshop & Challenge aims to encourage research in developing novel and accurate methods for human action understanding.
VISTA-LLAMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens
This amplifies the effect of visual tokens on text generation especially when the relative distance is longer between visual and text tokens.
Video Recognition in Portrait Mode
While existing datasets mainly comprise landscape mode videos, our paper seeks to introduce portrait mode videos to the research community and highlight the unique challenges associated with this video format.
Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens
This amplifies the effect of visual tokens on text generation, especially when the relative distance is longer between visual and text tokens.
Ranked #2 on
Question Answering
on NExT-QA (Open-ended VideoQA)
PixelLM: Pixel Reasoning with Large Multimodal Model
PixelLM excels across various pixel-level image reasoning and understanding tasks, outperforming well-established methods in multiple benchmarks, including MUSE, single- and multi-referring segmentation.
Selective Feature Adapter for Dense Vision Transformers
Specifically, SFA consists of external adapters and internal adapters which are sequentially operated over a transformer model.
Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling
To bridge the physical and virtual worlds for rapidly developed VR/AR applications, the ability to realistically drive 3D full-body avatars is of great significance.
COSA: Concatenated Sample Pretrained Vision-Language Foundation Model
Due to the limited scale and quality of video-text training corpus, most vision-language foundation models employ image-text datasets for pretraining and primarily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations.
Ranked #1 on
TGIF-Frame
on TGIF-QA
(using extra training data)
Delving Deeper into Data Scaling in Masked Image Modeling
The study reveals that: 1) MIM can be viewed as an effective method to improve the model capacity when the scale of the training data is relatively small; 2) Strong reconstruction targets can endow the models with increased capacities on downstream tasks; 3) MIM pre-training is data-agnostic under most scenarios, which means that the strategy of sampling pre-training data is non-critical.
VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending
Towards this goal, we propose a novel video-text pre-training method dubbed VLAB: Video Language pre-training by feature Adapting and Blending, which transfers CLIP representations to video pre-training tasks and develops unified video multimodal models for a wide range of video-text tasks.
Ranked #1 on
Visual Question Answering (VQA)
on MSVD-QA
(using extra training data)
MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval
The first is a Temporal Adaptation Module that is incorporated in the video branch to introduce global and local temporal contexts.
Temporal Perceiving Video-Language Pre-training
Specifically, text-video localization consists of moment retrieval, which predicts start and end boundaries in videos given the text description, and text localization which matches the subset of texts with the video features.
CMAE-V: Contrastive Masked Autoencoders for Video Action Recognition
Contrastive Masked Autoencoder (CMAE), as a new self-supervised framework, has shown its potential of learning expressive feature representations in visual image recognition.
Revisiting Training-free NAS Metrics: An Efficient Training-based Method
In this paper, we revisit these training-free metrics and find that: (1) the number of parameters (\#Param), which is the most straightforward training-free metric, is overlooked in previous works but is surprisingly effective, (2) recent training-free metrics largely rely on the \#Param information to rank networks.
Noise Self-Regression: A New Learning Paradigm to Enhance Low-Light Images Without Task-Related Data
Specifically, we propose Noise SElf-Regression (NoiSER) without access to any task-related data, simply learns a convolutional neural network equipped with an instance-normalization layer by taking a random noise image, $\mathcal{N}(0,\sigma^2)$ for each pixel, as both input and output for each training pair, and then the low-light image is fed to the trained network for predicting the normal-light image.
OSIC: A New One-Stage Image Captioner Coined
To obtain rich features, we use the Swin Transformer to calculate multi-level features, and then feed them into a novel dynamic multi-sight embedding module to exploit both global structure and local texture of input images.
Exploring Domain Incremental Video Highlights Detection with the LiveFood Benchmark
However, most VHD methods are based on the closed world assumption, i. e., a fixed number of highlight categories is defined in advance and all training data are available beforehand.
Contrastive Masked Autoencoders are Stronger Vision Learners
The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart.
AutoTransition: Learning to Recommend Video Transition Effects
First we learn the embedding of video transitions through a video transition classification task.
Video Salient Object Detection via Contrastive Features and Attention Modules
Video salient object detection aims to find the most visually distinctive objects in a video.
Conflict-Averse Gradient Descent for Multi-task Learning
The goal of multi-task learning is to enable more efficient learning than single task learning by sharing model structures for a diverse set of tasks.
HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
Last, we proposed an efficient fine-grained search strategy to train HR-NAS, which effectively explores the search space, and finds optimal architectures given various tasks and computation resources.
Refiner: Refining Self-attention for Vision Transformers
Vision Transformers (ViTs) have shown competitive accuracy in image classification tasks compared with CNNs.
Ranked #183 on
Image Classification
on ImageNet
One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning
Deep learning models in large-scale machine learning systems are often continuously trained with enormous data from production environments.
All Tokens Matter: Token Labeling for Training Better Vision Transformers
In this paper, we present token labeling -- a new training objective for training high-performance vision transformers (ViTs).
Ranked #3 on
Efficient ViTs
on ImageNet-1K (With LV-ViT-S)
DeepViT: Towards Deeper Vision Transformer
In this paper, we show that, unlike convolution neural networks (CNNs)that can be improved by stacking more convolutional layers, the performance of ViTs saturate fast when scaled to be deeper.
Ranked #466 on
Image Classification
on ImageNet
AutoSpace: Neural Architecture Search with Less Human Interference
Current neural architecture search (NAS) algorithms still require expert knowledge and effort to design a search space for network construction.
Human-centric Spatio-Temporal Video Grounding With Visual Transformers
HC-STVG is a video grounding task that requires both spatial (where) and temporal (when) localization.
Neural Architecture Search for Lightweight Non-Local Networks
However, it has been rarely explored to embed the NL blocks in mobile neural networks, mainly due to the following challenges: 1) NL blocks generally have heavy computation cost which makes it difficult to be applied in applications where computational resources are limited, and 2) it is an open problem to discover an optimal configuration to embed NL blocks into mobile neural networks.
Ranked #60 on
Neural Architecture Search
on ImageNet
RC-DARTS: Resource Constrained Differentiable Architecture Search
In this paper, we propose the resource constrained differentiable architecture search (RC-DARTS) method to learn architectures that are significantly smaller and faster while achieving comparable accuracy.
AtomNAS: Fine-Grained End-to-End Neural Architecture Search
We propose a fine-grained search space comprised of atomic blocks, a minimal search unit that is much smaller than the ones used in recent NAS algorithms.
Ranked #61 on
Neural Architecture Search
on ImageNet
Neural Epitome Search for Architecture-Agnostic Network Compression
The recent WSNet [1] is a new model compression method through sampling filterweights from a compact set and has demonstrated to be effective for 1D convolutionneural networks (CNNs).
WSNet: Learning Compact and Efficient Networks with Weight Sampling
We present a new approach and a novel architecture, termed WSNet, for learning compact and efficient deep neural networks.
WSNet: Compact and Efficient Networks Through Weight Sampling
We present a new approach and a novel architecture, termed WSNet, for learning compact and efficient deep neural networks.
Predicting Scene Parsing and Motion Dynamics in the Future
The ability of predicting the future is important for intelligent systems, e. g. autonomous vehicles and robots to plan early and make decisions accordingly.
Dual Path Networks
In this work, we present a simple, highly efficient and modularized Dual Path Network (DPN) for image classification which presents a new topology of connection paths internally.
Deep Self-Taught Learning for Weakly Supervised Object Localization
To overcome this issue, we propose a deep self-taught learning approach, which makes the detector learn the object-level features reliable for acquiring tight positive samples and afterwards re-train itself based on them.
Training Group Orthogonal Neural Networks with Privileged Information
Learning rich and diverse representations is critical for the performance of deep convolutional neural networks (CNNs).
Video Scene Parsing with Predictive Feature Learning
In this way, the network can effectively learn to capture video dynamics and temporal context, which are critical clues for video scene parsing, without requiring extra manual annotations.
Multi-Path Feedback Recurrent Neural Network for Scene Parsing
In this paper, we consider the scene parsing problem and propose a novel Multi-Path Feedback recurrent neural network (MPF-RNN) for parsing scene images.
Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
In this paper, we propose a layer-wise discriminative learning method to enhance the discriminative capability of a deep network by allowing its layers to work collaboratively for classification.
Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
In this paper, we propose an iterative hard thresholding (IHT) approach to train Skinny Deep Neural Networks (SDNNs).
Deep Learning with S-shaped Rectified Linear Activation Units
Rectified linear activation units are important components for state-of-the-art deep convolutional networks.
