Search Results for author:
Found 52 papers, 27 papers with code
Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound
The quantification of audio aesthetics remains a complex challenge in audio processing, primarily due to its subjective nature, which is influenced by human perception and cultural context.
Molecular Graph Contrastive Learning with Line Graph
Trapped by the label scarcity in molecular property prediction and drug design, graph contrastive learning (GCL) came forward.
Audiobox TTA-RAG: Improving Zero-Shot and Few-Shot Text-To-Audio with Retrieval-Augmented Generation
This work focuses on improving Text-To-Audio (TTA) generation on zero-shot and few-shot settings (i. e. generating unseen or uncommon audio events).
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following
To address this challenge, we introduce MDCure, a scalable and effective instruction data generation framework to enhance the MD capabilities of LLMs without the computational cost of pre-training or reliance on human-annotated data.
MusicFlow: Cascaded Flow Matching for Text Guided Music Generation
Based on self-supervised representations to bridge between text descriptions and music audios, we construct two flow matching networks to model the conditional distribution of semantic and acoustic features.
Movie Gen: A Cast of Media Foundation Models
Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation.
NC-NCD: Novel Class Discovery for Node Classification
SWORD achieves this by employing a self-training strategy to learn new categories and preventing the forgetting of old categories through the joint use of feature prototypes and knowledge distillation.
High Fidelity Text-Guided Music Editing via Single-Stage Flow Matching
We adapt the ReNoise latent inversion method to flow matching and compare it with the original implementation and naive denoising diffusion implicit model (DDIM) inversion on a variety of music editing prompts.
Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning
As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations.
XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception
It is designed to maximize the benefits of limited multilingual AV pre-training data, by building on top of audio-only multilingual pre-training and simplifying existing pre-training schemes.
Towards Privacy-Aware Sign Language Translation at Scale
A major impediment to the advancement of sign language translation (SLT) is data scarcity.
Ranked #1 on
Gloss-free Sign Language Translation
on How2Sign
(using extra training data)
Gloss-free Sign Language Translation
Sign Language Translation
+1
UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding
Vision-language foundation models, represented by Contrastive Language-Image Pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks.
Ranked #1 on
Open Vocabulary Panoptic Segmentation
on ADE20K
Open Vocabulary Panoptic Segmentation
Open Vocabulary Semantic Segmentation
+2
Audiobox: Unified Audio Generation with Natural Language Prompts
Research communities have made great progress over the past year advancing the performance of large scale audio generative models for a single modality (speech, sound, or music) through adopting more powerful generative models and scaling data.
Ranked #1 on
Audio Generation
on AudioCaps
(FAD metric)
AiluRus: A Scalable ViT Framework for Dense Prediction
Specifically, at the intermediate layer of the ViT, we utilize a spatial-aware density-based clustering algorithm to select representative tokens from the token sequence.
Generative Pre-training for Speech with Flow Matching
Generative models have gained more and more attention in recent years for their remarkable success in tasks that required estimating and sampling data distribution to generate high-fidelity synthetic data.
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs.
Ranked #2 on
Text-to-Image Generation
on COCO
Toward American Sign Language Processing in the Real World: Data, Tasks, and Methods
To address the problem of searching for fingerspelled keywords in raw sign language videos, we propose a novel method that jointly localizes and matches fingerspelling segments to text.
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization).
ActionPrompt: Action-Guided 3D Human Pose Estimation With Text and Pose Prompting
Recent 2D-to-3D human pose estimation (HPE) utilizes temporal consistency across sequences to alleviate the depth ambiguity problem but ignore the action related prior knowledge hidden in the pose sequence.
Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners
In order to better obtain both discrimination and diversity, we propose a simple but effective Hybrid Distillation strategy, which utilizes both the supervised/CL teacher and the MIM teacher to jointly guide the student model.
Prompt to GPT-3: Step-by-Step Thinking Instructions for Humor Generation
Artificial intelligence has made significant progress in natural language processing, with models like GPT-3 demonstrating impressive capabilities.
Scaling Speech Technology to 1,000+ Languages
Expanding the language coverage of speech technology has the potential to improve access to information for many more people.
SEGA: Structural Entropy Guided Anchor View for Graph Contrastive Learning
In contrastive learning, the choice of ``view'' controls the information that the representation captures and influences the performance of the model.
Rethinking Visual Prompt Learning as Masked Visual Token Modeling
To explore prompt learning on the generative pre-trained visual model, as well as keeping the task consistency, we propose Visual Prompt learning as masked visual Token Modeling (VPTM) to transform the downstream visual classification into the pre-trained masked visual token prediction.
MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages.
Audio-Visual Speech Recognition
Robust Speech Recognition
+5
Pose-Oriented Transformer with Uncertainty-Guided Refinement for 2D-to-3D Human Pose Estimation
There has been a recent surge of interest in introducing transformers to 3D human pose estimation (HPE) due to their powerful capabilities in modeling long-term dependencies.
Visual Story Generation Based on Emotion and Keywords
Automated visual story generation aims to produce stories with corresponding illustrations that exhibit coherence, progression, and adherence to characters' emotional development.
ReVISE: Self-Supervised Speech Resynthesis With Visual Input for Universal and Generalized Speech Regeneration
Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR.
Adapting Shortcut With Normalizing Flow: An Efficient Tuning Framework for Visual Recognition
To mitigate the computational and storage demands, recent research has explored Parameter-Efficient Fine-Tuning (PEFT), which focuses on tuning a minimal number of parameters for efficient adaptation.
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
Moreover, we utilize self-supervised audio-visual speech model to initialize P-AVSR.
Ranked #1 on
Speech Recognition
on EasyCom
Comparative layer-wise analysis of self-supervised speech models
We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks.
u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality
By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-art modality-specific models.
Open-Domain Sign Language Translation Learned from Online Video
Existing work on sign language translation - that is, translation from sign language videos into sentences in a written language - has focused mainly on (1) data collected in a controlled environment or (2) data in a specific domain, which limits the applicability to real-world settings.
Ranked #2 on
Gloss-free Sign Language Translation
on OpenASL
Gloss-free Sign Language Translation
Sign Language Translation
+1
Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT
This paper investigates self-supervised pre-training for audio-visual speaker representation learning where a visual stream showing the speaker's mouth area is used alongside speech as inputs.
Searching for fingerspelled content in American Sign Language
This is an important task since significant content in sign language is often conveyed via fingerspelling, and to our knowledge the task has not been studied before.
Robust Self-Supervised Audio-Visual Speech Recognition
Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe.
Ranked #6 on
Audio-Visual Speech Recognition
on LRS3-TED
(using extra training data)
Audio-Visual Speech Recognition
Automatic Speech Recognition
+6
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
The lip-reading WER is further reduced to 26. 9% when using all 433 hours of labeled data from LRS3 and combined with self-training.
Ranked #3 on
Speech Recognition
on LRS3-TED
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Hierarchical Graph Networks for 3D Human Pose Estimation
Recent 2D-to-3D human pose estimation works tend to utilize the graph structure formed by the topology of the human skeleton.
Ranked #47 on
3D Human Pose Estimation
on MPI-INF-3DHP
(AUC metric)
Multi-dataset Pretraining: A Unified Model for Semantic Segmentation
This is achieved by first pretraining the network via the proposed pixel-to-prototype contrastive loss over multiple datasets regardless of their taxonomy labels, and followed by fine-tuning the pretrained model over specific dataset as usual.
Fingerspelling Detection in American Sign Language
We propose a benchmark and a suite of evaluation metrics, some of which reflect the effect of detection on the downstream fingerspelling recognition task.
Domain wall topological entanglement entropy
We derive a universal correction to the ground-state entanglement entropy, which is equal to the logarithm of the total quantum dimension of a set of superselection sectors localized on the domain wall.
Strongly Correlated Electrons High Energy Physics - Theory Quantum Physics
Whole-Word Segmental Speech Recognition with Acoustic Word Embeddings
We find that word error rate can be reduced by a large margin by pre-training the acoustic segment representation with AWEs, and additional (smaller) gains can be obtained by pre-training the word prediction layer with AGWEs.
A Cross-Task Analysis of Text Span Representations
Many natural language processing (NLP) tasks involve reasoning with textual spans, including question answering, entity recognition, and coreference resolution.
Few-shot acoustic event detection via meta-learning
We study few-shot acoustic event detection (AED) in this paper.
Latency-Aware Differentiable Neural Architecture Search
However, these methods suffer the difficulty in optimizing network, so that the searched network is often unfriendly to hardware.
Fingerspelling recognition in the wild with iterative visual attention
In this paper we focus on recognition of fingerspelling sequences in American Sign Language (ASL) videos collected in the wild, mainly from YouTube and Deaf social media.
Compression of Acoustic Event Detection Models With Quantized Distillation
Acoustic Event Detection (AED), aiming at detecting categories of events based on audio signals, has found application in many intelligent systems.
Compression of Acoustic Event Detection Models with Low-rank Matrix Factorization and Quantization Training
In this paper, we present a compression approach based on the combination of low-rank matrix factorization and quantization training, to reduce complexity for neural network based acoustic event detection (AED) models.
Semi-supervised Acoustic Event Detection based on tri-training
This paper presents our work of training acoustic event detection (AED) models using unlabeled dataset.
On the Contributions of Visual and Textual Supervision in Low-Resource Semantic Speech Retrieval
Recent work has shown that speech paired with images can be used to learn semantically meaningful speech representations even without any textual supervision.
American Sign Language fingerspelling recognition in the wild
As the first attempt at fingerspelling recognition in the wild, this work is intended to serve as a baseline for future work on sign language recognition in realistic conditions.
Multitask training with unlabeled data for end-to-end sign language fingerspelling recognition
We introduce a model for fingerspelling recognition that addresses these issues.
