Search Results for author:
Found 49 papers, 31 papers with code
Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle.
Ranked #24 on
Visual Question Answering (VQA)
on MSRVTT-QA
(using extra training data)
Revealing Single Frame Bias for Video-and-Language Learning
Training an effective video-and-language model intuitively requires multiple frames as model inputs.
Ranked #5 on
Video Retrieval
on SSv2-template retrieval
(using extra training data)
Unifying Vision-and-Language Tasks via Text Generation
On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models.
Ranked #3 on
Image Captioning
on nocaps val
SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery
Furthermore, we design a simple and flexible SR branch to learn HR feature representations that can discriminate small objects from vast backgrounds with low-resolution (LR) input, thus further improving the detection accuracy.
Ranked #3 on
Object Detection
on VEDAI
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries
Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w. r. t.
Ranked #12 on
Highlight Detection
on QVHighlights
Detecting Moments and Highlights in Videos via Natural Language Queries
Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w. r. t.
Ranked #6 on
Video Grounding
on QVHighlights
MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph.
Ranked #5 on
Video Captioning
on ActivityNet Captions
TVQA: Localized, Compositional Video Question Answering
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks.
Ranked #4 on
Video Question Answering
on SUTD-TrafficQA
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval
The queries are also labeled with query types that indicate whether each of them is more related to video or subtitle or both, allowing for in-depth analysis of the dataset and the methods that built on top of it.
Ranked #2 on
Video Retrieval
on TVR
TVQA+: Spatio-Temporal Grounding for Video Question Answering
We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos.
Ranked #6 on
Video Question Answering
on TVQA
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction.
VindLU: A Recipe for Effective Video-and-Language Pretraining
Furthermore, our model also obtains state-of-the-art video question-answering results on ActivityNet-QA, MSRVTT-QA, MSRVTT-MC and TVQA.
Ranked #2 on
Video Retrieval
on Condensed Movies
(using extra training data)
AFPN: Asymptotic Feature Pyramid Network for Object Detection
Multi-scale features are of great importance in encoding objects with scale variance in object detection tasks.
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task.
Vision Transformers are Parameter-Efficient Audio-Visual Learners
To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT.
Ranked #4 on
Audio-visual Question Answering
on MUSIC-AVQA
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e. g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations.
Ranked #10 on
Action Recognition
on Diving-48
What is More Likely to Happen Next? Video-and-Language Future Event Prediction
Given a video with aligned dialogue, people can often infer what is more likely to happen next.
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
We present Perceiver-VL, a vision-and-language framework that efficiently handles high-dimensional multimodal inputs such as long videos and text.
ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound
We introduce an audiovisual method for long-range text-to-video retrieval.
MTVR: Multilingual Moment Retrieval in Videos
We introduce mTVR, a large-scale multilingual video moment retrieval dataset, containing 218K English and Chinese queries from 21. 8K TV show video clips.
Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching
More concretely, we first design a structure called guided quantization self-distillation (GQSD), which is an innovative idea for realizing lightweight through the synergy of quantization and distillation.
Ranked #1 on
Object Detection
on VEDAI
RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios
We introduce RESIN-11, a new schema-guided event extraction&prediction framework that can be applied to a large variety of newsworthy scenarios.
Transcoded Video Restoration by Temporal Spatial Auxiliary Network
In most video platforms, such as Youtube, and TikTok, the played videos usually have undergone multiple video encodings such as hardware encoding by recording devices, software encoding by video editing apps, and single/multiple video transcoding by video application servers.
Toward Stable, Interpretable, and Lightweight Hyperspectral Super-Resolution
Specifically, we create a positive cycle between fusion and degradation estimation under a new probabilistic framework.
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization
Second, to alleviate the temporal misalignment issue, our method incorporates an entropy minimization-based constrained attention loss, to encourage the model to automatically focus on the correct caption from a pool of candidate ASR captions.
Edge-competing Pathological Liver Vessel Segmentation with Limited Labels
The diagnosis of MVI needs discovering the vessels that contain hepatocellular carcinoma cells and counting their number in each vessel, which depends heavily on experiences of the doctor, is largely subjective and time-consuming.
EfficientMFD: Towards More Efficient Multimodal Synchronous Fusion Detection
Multimodal image fusion and object detection play a vital role in autonomous driving.
Distribution-aware Interactive Attention Network and Large-scale Cloud Recognition Benchmark on FY-4A Satellite Image
Accurate cloud recognition and warning are crucial for various applications, including in-flight support, weather forecasting, and climate research.
Multimodal Informative ViT: Information Aggregation and Distribution for Hyperspectral and LiDAR Classification
Additionally, the information distribution flow (IDF) in MIVit enhances performance-awareness by distributing global classification information across different modalities' feature maps.
Mid-level Representation Enhancement and Graph Embedded Uncertainty Suppressing for Facial Expression Recognition
On the other hand, GUS is introduced to suppress the feature ambiguity in the representation space.
Facial Expression Recognition
Facial Expression Recognition (FER)
+1
Physics Inspired Criterion for Pruning-Quantization Joint Learning
Pruning-quantization joint learning always facilitates the deployment of deep neural networks (DNNs) on resource-constrained edge devices.
TripletGAN: Training Generative Model with Triplet Loss
As an effective way of metric learning, triplet loss has been widely used in many deep learning tasks, including face recognition and person-ReID, leading to many states of the arts.
Selective Zero-Shot Classification with Augmented Attributes
We propose a selective zero-shot classifier based on both the human defined and the automatically discovered residual attributes.
A Novel Multi-Step Finite-State Automaton for Arbitrarily Deterministic Tsetlin Machine Learning
However, TMs rely heavily on energy-costly random number generation to stochastically guide a team of Tsetlin Automata to a Nash Equilibrium of the TM game.
MCM-aware Twin-least-square GAN for Hyperspectral Anomaly Detection
Hyperspectral anomaly detection under high-dimensional data and interference of deteriorated bands without any prior information has been challenging and attracted close attention in the exploration of the unknown in real scenarios.
Sparse Coding-inspired GAN for Weakly Supervised Hyperspectral Anomaly Detection
For solving these problems, this paper proposes a sparse coding-inspired generative adversarial network (GAN) for weakly supervised HAD, named sparseHAD.
One-sample Guided Object Representation Disassembling
In this paper, we introduce the One-sample Guided Object Representation Disassembling (One-GORD) method, which only requires one annotated sample for each object category to learn disassembled object representation from unannotated images.
Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models
We hope our Adversarial VQA dataset can shed new light on robustness study in the community and serve as a valuable benchmark for future work.
Boundary Knowledge Translation based Reference Semantic Segmentation
In this paper, we introduce a novel Reference semantic segmentation Network (Ref-Net) to conduct visual boundary knowledge translation.
Mutual-Complementing Framework for Nuclei Detection and Segmentation in Pathology Image
The existing two unsupervised methods are prone to failure on degenerated samples.
LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval
In this work, we propose LoopITR, which combines them in the same network for joint learning.
CNN LEGO: Disassembling and Assembling Convolutional Neural Network
the feature maps are adopted to locate the critical features in each layer.
Toward matrix multiplication for deep learning inference on the Xilinx Versal
The remarkable positive impact of Deep Neural Networks on many Artificial Intelligence (AI) tasks has led to the development of various high performance algorithms as well as specialized processors and accelerators.
ViT-Calibrator: Decision Stream Calibration for Vision Transformer
In this work, we propose a new paradigm dubbed Decision Stream Calibration that boosts the performance of general Vision Transformers.
A Loopback Network for Explainable Microvascular Invasion Classification
The loopback between two branches enables the category label to supervise the cell locating branch to learn the locating ability for cancerous areas.
SAR-Net: Multi-scale Direction-aware SAR Network via Global Information Fusion
Deep learning has driven significant progress in object detection using Synthetic Aperture Radar (SAR) imagery.
SwiMDiff: Scene-wide Matching Contrastive Learning with Diffusion Constraint for Remote Sensing Image
Effectively leveraging this data through self-supervised learning (SSL) is vital in the field of remote sensing.
Angle Robustness Unmanned Aerial Vehicle Navigation in GNSS-Denied Scenarios
Due to the inability to receive signals from the Global Navigation Satellite System (GNSS) in extreme conditions, achieving accurate and robust navigation for Unmanned Aerial Vehicles (UAVs) is a challenging task.
SPAR: Personalized Content-Based Recommendation via Long Engagement Attention
Leveraging users' long engagement histories is essential for personalized content recommendations.
