Search Results for author:
Found 39 papers, 14 papers with code
DailyMAE: Towards Pretraining Masked Autoencoders in One Day
Recently, masked image modeling (MIM), an important self-supervised learning (SSL) method, has drawn attention for its effectiveness in learning data representation from unlabeled data.
Text-to-Audio Generation Synchronized with Videos
Extensive evaluations on the AudioCaps and T2AV-Bench demonstrate that our T2AV sets a new standard for video-aligned TTA generation in ensuring visual alignment and temporal consistency.
Audio-Synchronized Visual Animation
We hope our established benchmark can open new avenues for controllable visual generation.
LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning
Visual Prompt Tuning (VPT) techniques have gained prominence for their capacity to adapt pre-trained Vision Transformers (ViTs) to downstream visual tasks using specialized learnable tokens termed as prompts.
We Choose to Go to Space: Agent-driven Human and Multi-Robot Collaboration in Microgravity
We present SpaceAgents-1, a system for learning human and multi-robot collaboration (HMRC) strategies under microgravity conditions.
Fast Training of Diffusion Transformer with Extreme Masking for 3D Point Clouds Generation
Motivated by the inherent redundancy of 3D compared to 2D, we propose FastDiT-3D, a novel masked diffusion transformer tailored for efficient 3D point cloud generation, which greatly reduces training costs.
Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling
Thus, to address the computational complexity, we propose an alternative procedure that factorizes the local representations before representing audio-visual interactions.
Linker-Tuning: Optimizing Continuous Prompts for Heterodimeric Protein Prediction
Predicting the structure of interacting chains is crucial for understanding biological systems and developing new drugs.
Beyond Accuracy: Statistical Measures and Benchmark for Evaluation of Representation from Self-Supervised Learning
Recently, self-supervised metric learning has raised attention for the potential to learn a generic distance function.
MultiIoT: Towards Large-scale Multisensory Learning for the Internet of Things
The Internet of Things (IoT), the network integrating billions of smart physical devices embedded with sensors, software, and communication technologies for the purpose of connecting and exchanging data with other devices and systems, is a critical and rapidly expanding component of our modern world.
Exploring Data Augmentations on Self-/Semi-/Fully- Supervised Pre-trained Models
Data augmentation has become a standard component of vision pre-trained models to capture the invariance between augmented views.
Tree of Uncertain Thoughts Reasoning for Large Language Models
These local uncertainties, intrinsic to LLMs given their potential for diverse responses, remain a significant concern in the reasoning process.
Class-Incremental Grouping Network for Continual Audio-Visual Learning
Our CIGN leverages learnable audio-visual class tokens and audio-visual grouping to continually aggregate class-aware features.
Masked Momentum Contrastive Learning for Zero-shot Semantic Understanding
Self-supervised pretraining (SSP) has emerged as a popular technique in machine learning, enabling the extraction of meaningful feature representations without labelled data.
Audio-Visual Class-Incremental Learning
We demonstrate that joint audio-visual modeling can improve class-incremental learning, but current methods fail to preserve semantic similarity between audio and visual features as incremental step grows.
DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
Recent Diffusion Transformers (e. g., DiT) have demonstrated their powerful effectiveness in generating high-quality 2D images.
Ranked #1 on
Point Cloud Generation
on ShapeNet Car
A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition
The ability to accurately recognize, localize and separate sound sources is fundamental to any audio-visual perception task.
DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment
In this work, we propose a novel and personalized text-to-sound generation approach with visual alignment based on latent diffusion models, namely DiffAVA, that can simply fine-tune lightweight visual-text alignment modules with frozen modality-specific encoders to update visual-aligned text embeddings as the condition.
AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation
In this work, we propose a simple yet effective audio-visual localization and segmentation framework based on the Segment Anything Model, namely AV-SAM, that can generate sounding object masks corresponding to the audio.
CAVL: Learning Contrastive and Adaptive Representations of Vision and Language
Visual and linguistic pre-training aims to learn vision and language representations together, which can be transferred to visual-linguistic downstream tasks.
Audio-Visual Grouping Network for Sound Localization from Mixtures
Sound source localization is a typical and challenging task that predicts the location of sound sources in a video.
Variantional autoencoder with decremental information bottleneck for disentanglement
One major challenge of disentanglement learning with variational autoencoders is the trade-off between disentanglement and reconstruction fidelity.
Rethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlation
Particularly, in the classification down-stream tasks with linear probes, our proposed method outperforms the state-of-the-art instance-wise and prototypical contrastive learning methods on the ImageNet-100 dataset by 2. 96% and the ImageNet-1K dataset by 2. 46% under the same settings of batch size and epochs.
A Closer Look at Weakly-Supervised Audio-Visual Source Localization
We also propose a new approach for visual sound source localization that addresses both these problems.
Siamese Prototypical Contrastive Learning
One of the drawbacks of CSL is that the loss term requires a large number of negative samples to provide better mutual information bound ideally.
Object-wise Masked Autoencoders for Fast Pre-training
Furthermore, we investigate the inter-object and intra-object relationship and find that the latter is crucial for self-supervised pre-training.
Unitail: Detecting, Reading, and Matching in Retail Scene
To make full use of computer vision technology in stores, it is required to consider the actual needs that fit the characteristics of the retail scene.
Ranked #1 on
Dense Object Detection
on SKU-110K
Point3D: tracking actions as moving points with 3D CNNs
Our Point3D consists of a Point Head for action localization and a 3D Head for action classification.
Localizing Visual Sounds the Easy Way
Unsupervised audio-visual source localization aims at localizing visible sound sources in a video without relying on ground-truth localization for training.
Multi-Scale Self-Contrastive Learning with Hard Negative Mining for Weakly-Supervised Query-based Video Grounding
Secondly, since some predicted frames (i. e., boundary frames) are relatively coarse and exhibit similar appearance to their adjacent frames, we propose a coarse-to-fine contrastive learning paradigm to learn more discriminative frame-wise representations for distinguishing the false positive frames.
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots.
Context Autoencoder for Self-Supervised Representation Learning
The pretraining tasks include two tasks: masked representation prediction - predict the representations for the masked patches, and masked patch reconstruction - reconstruct the masked patches.
Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types
The core problem is to model how regulatory elements interact with each other and its variability across different cell types.
Representation Disentanglement in Generative Models with Contrastive Learning
Recent works apply the contrastive learning on the discriminator of the Generative Adversarial Networks, and there exists little work on exploring if contrastive learning can be applied on encoders to learn disentangled representations.
Piecing and Chipping: An effective solution for the information-erasing view generation in Self-supervised Learning
In self-supervised learning frameworks, deep networks are optimized to align different views of an instance that contains the similar visual semantic information.
Multi-modal Self-supervised Pre-training for Large-scale Genome Data
In this work, we propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT.
Learning by Examples Based on Multi-level Optimization
Learning by examples, which learns to solve a new problem by looking into how similar problems are solved, is an effective learning method in human learning.
Towards Improving Spatiotemporal Action Recognition in Videos
Spatiotemporal action recognition deals with locating and classifying actions in videos.
Automatic Speech Verification Spoofing Detection
Automatic speech verification (ASV) is the technology to determine the identity of a person based on their voice.
