Search Results for author:
Found 42 papers, 27 papers with code
xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations
We introduce xbench, a dynamic, profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity.
CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment
Second, we resolve conflicting optimization goals by separating contrastive and reconstruction objectives through dedicated global tokens.
UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation
We show that with the appropriate design choices for pre-training, one can jointly learn a representation encoder and generative audio decoder that can be applied to both types of tasks.
State-Space Large Audio Language Models
Large Audio Language Models (LALM) combine the audio perception models and the Large Language Models (LLM) and show a remarkable ability to reason about the input audio, infer the meaning, and understand the intent.
A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generation
The discrete and low-frequency nature of neural codecs introduced a new way to generate speech with token-based models.
AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models
Recent advancements in Large Language Models (LLMs) have demonstrated great success in many Natural Language Processing (NLP) tasks.
Revise, Reason, and Recognize: LLM-Based Emotion Recognition via Emotion-Specific Prompts and ASR Error Correction
Furthermore, we propose a Revise-Reason-Recognize prompting pipeline for robust LLM-based emotion recognition from spoken language with ASR errors.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition
Given recent advances in generative AI technology, a key question is how large language models (LLMs) can enhance acoustic modeling tasks using text decoding results from a frozen, pretrained automatic speech recognition (ASR) model.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners
To the best of our knowledge, it is the first SSM that outperforms the Transformers on AudioSet and achieves an mAP of 48. 9; and 2) We designed a new test called Audio Needle In A Haystack (Audio NIAH).
Ranked #29 on
Audio Classification
on AudioSet
Automatic Prediction of Amyotrophic Lateral Sclerosis Progression using Longitudinal Speech Transformer
Automatic prediction of amyotrophic lateral sclerosis (ALS) disease progression provides a more efficient and objective alternative than manual approaches.
Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation
Since videos are harder to obtain than audio, the video training data of AVSR models is usually limited to a few thousand hours.
Ranked #1 on
Audio-Visual Speech Recognition
on LRS2
(using extra training data)
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+5
Generic Knowledge Boosted Pre-training For Remote Sensing Images
Deep learning models are essential for scene classification, change detection, land cover segmentation, and other remote sensing image understanding tasks.
Joint Audio and Speech Understanding
Humans are surrounded by audio signals that include both speech and non-speech sounds.
Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning
How can we perform computations over natural language representations to solve tasks that require symbolic and numeric reasoning?
ToonTalker: Cross-Domain Face Reenactment
Moreover, since no paired data is provided, we propose a novel cross-domain training scheme using data from two domains with the designed analogy constraint.
Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
For the first module, we leverage an off-the-shelf video retrieval system and extract video depths as motion structure.
TaleCrafter: Interactive Story Visualization with Multiple Characters
Accurate Story visualization requires several necessary elements, such as identity consistency across frames, the alignment between plain text and visual content, and a reasonable layout of objects in images.
SAIL: Search-Augmented Instruction Learning
Large language models (LLMs) have been significantly improved by instruction fine-tuning, but still lack transparency and the ability to utilize up-to-date knowledge and information.
Listen, Think, and Understand
On the other hand, modern large language models (LLMs) exhibit emerging reasoning ability but they lack audio perception capabilities.
Ranked #3 on
Music Question Answering
on MusicQA
(using extra training data)
3D GAN Inversion with Facial Symmetry Prior
It is natural to associate 3D GANs with GAN inversion methods to project a real image into the generator's latent space, allowing free-view consistent synthesis and editing, referred as 3D GAN inversion.
MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model
Multimodal semantic understanding often has to deal with uncertainty, which means the obtained messages tend to refer to multiple targets.
Contrastive Audio-Visual Masked Autoencoder
In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities.
Ranked #1 on
Audio Tagging
on AudioSet
(using extra training data)
Rethinking Knowledge Distillation via Cross-Entropy
Furthermore, we smooth students' target output to treat it as the soft target for training without teachers and propose a teacher-free new KD loss (tf-NKD).
UAVM: Towards Unifying Audio and Visual Models
Conventional audio-visual models have independent audio and video branches.
Ranked #2 on
Multi-modal Classification
on AudioSet
(using extra training data)
Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition
Recognizing human non-speech vocalizations is an important task and has broad applications such as automatic sound transcription and health condition monitoring.
Ranked #1 on
Audio Classification
on VocalSound
Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment
Automatic pronunciation assessment is an important technology to help self-directed language learners.
Ranked #3 on
Phone-level pronunciation scoring
on speechocean762
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Attentions Help CNNs See Better: Attention-based Hybrid Image Quality Assessment Network
Image quality assessment (IQA) algorithm aims to quantify the human perception of image quality.
Ranked #1 on
Image Quality Assessment
on MSU FR VQA Database
MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment
No-Reference Image Quality Assessment (NR-IQA) aims to assess the perceptual quality of images in accordance with human subjective perception.
Ranked #8 on
Video Quality Assessment
on MSU SR-QA Dataset
CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification
Audio classification is an active research area with a wide range of applications.
Focal and Global Knowledge Distillation for Detectors
Global distillation rebuilds the relation between different pixels and transfers it from teachers to students, compensating for missing global information in focal distillation.
SSAST: Self-Supervised Audio Spectrogram Transformer
However, pure Transformer models tend to require more training data compared to CNNs, and the success of the AST relies on supervised pretraining that requires a large amount of labeled data and a complex training pipeline, thus limiting the practical usage of AST.
Ranked #1 on
Spoken Command Recognition
on Speech Command v2
AST: Audio Spectrogram Transformer
In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels.
Ranked #2 on
Audio Classification
on Speech Commands
PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation
Audio tagging is an active research area and has a wide range of applications.
Ranked #6 on
Audio Classification
on FSD50K
(using extra training data)
Detecting Replay Attacks Using Multi-Channel Audio: A Neural Network-Based Method
With the rapidly growing number of security-sensitive systems that use voice as the primary input, it becomes increasingly important to address these systems' potential vulnerability to replay attacks.
Second-order Non-local Attention Networks for Person Re-identification
Recent efforts have shown promising results for person re-identification by designing part-based architectures to allow a neural network to learn discriminative representations from semantically coherent parts.
Real-Time Adversarial Attacks
In recent years, many efforts have demonstrated that modern machine learning algorithms are vulnerable to adversarial attacks, where small, but carefully crafted, perturbations on the input can make them fail.
ReMASC: Realistic Replay Attack Corpus for Voice Controlled Systems
This paper introduces a new database of voice recordings with the goal of supporting research on vulnerabilities and protection of voice-controlled systems (VCSs).
Towards Learning Fine-Grained Disentangled Representations from Speech
Learning disentangled representations of high-dimensional data is currently an active research area.
Topic Modeling Based Multi-modal Depression Detection
Major depressive disorder is a common mental disorder that affects almost 7% of the adult U. S. population.
An Overview of Vulnerabilities of Voice Controlled Systems
These systems have been shown to be vulnerable to various types of voice spoofing attacks.
How do deep convolutional neural networks learn from raw audio waveforms?
Prior work on speech and audio processing has demonstrated the ability to obtain excellent performance when learning directly from raw audio waveforms using convolutional neural networks (CNNs).
Crafting Adversarial Examples For Speech Paralinguistics Applications
Computational paralinguistic analysis is increasingly being used in a wide range of cyber applications, including security-sensitive applications such as speaker verification, deceptive speech detection, and medical diagnostics.
