Search Results for author:
Found 31 papers, 22 papers with code
Generic Knowledge Boosted Pre-training For Remote Sensing Images
Deep learning models are essential for scene classification, change detection, land cover segmentation, and other remote sensing image understanding tasks.
Joint Audio and Speech Understanding
Humans are surrounded by audio signals that include both speech and non-speech sounds.
Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning
How can we perform computations over natural language representations to solve tasks that require symbolic and numeric reasoning?
ToonTalker: Cross-Domain Face Reenactment
Moreover, since no paired data is provided, we propose a novel cross-domain training scheme using data from two domains with the designed analogy constraint.
Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
For the first module, we leverage an off-the-shelf video retrieval system and extract video depths as motion structure.
TaleCrafter: Interactive Story Visualization with Multiple Characters
Accurate Story visualization requires several necessary elements, such as identity consistency across frames, the alignment between plain text and visual content, and a reasonable layout of objects in images.
SAIL: Search-Augmented Instruction Learning
Large language models (LLMs) have been significantly improved by instruction fine-tuning, but still lack transparency and the ability to utilize up-to-date knowledge and information.
Listen, Think, and Understand
On the other hand, modern large language models (LLMs) exhibit emerging reasoning ability but they lack audio perception capabilities.
Ranked #3 on
Music Question Answering
on MusicQA
(using extra training data)
3D GAN Inversion with Facial Symmetry Prior
It is natural to associate 3D GANs with GAN inversion methods to project a real image into the generator's latent space, allowing free-view consistent synthesis and editing, referred as 3D GAN inversion.
MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model
Multimodal semantic understanding often has to deal with uncertainty, which means the obtained messages tend to refer to multiple targets.
Contrastive Audio-Visual Masked Autoencoder
In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities.
Ranked #1 on
Audio Tagging
on AudioSet
(using extra training data)
Rethinking Knowledge Distillation via Cross-Entropy
Furthermore, we smooth students' target output to treat it as the soft target for training without teachers and propose a teacher-free new KD loss (tf-NKD).
UAVM: Towards Unifying Audio and Visual Models
Conventional audio-visual models have independent audio and video branches.
Ranked #2 on
Multi-modal Classification
on AudioSet
(using extra training data)
Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment
Automatic pronunciation assessment is an important technology to help self-directed language learners.
Ranked #2 on
Phone-level pronunciation scoring
on speechocean762
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition
Recognizing human non-speech vocalizations is an important task and has broad applications such as automatic sound transcription and health condition monitoring.
Ranked #1 on
Audio Classification
on VocalSound
Attentions Help CNNs See Better: Attention-based Hybrid Image Quality Assessment Network
Image quality assessment (IQA) algorithm aims to quantify the human perception of image quality.
Ranked #1 on
Image Quality Assessment
on MSU FR VQA Database
MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment
No-Reference Image Quality Assessment (NR-IQA) aims to assess the perceptual quality of images in accordance with human subjective perception.
Ranked #8 on
Video Quality Assessment
on MSU SR-QA Dataset
CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification
Audio classification is an active research area with a wide range of applications.
Focal and Global Knowledge Distillation for Detectors
Global distillation rebuilds the relation between different pixels and transfers it from teachers to students, compensating for missing global information in focal distillation.
Ranked #1 on
Knowledge Distillation
on MS COCO
SSAST: Self-Supervised Audio Spectrogram Transformer
However, pure Transformer models tend to require more training data compared to CNNs, and the success of the AST relies on supervised pretraining that requires a large amount of labeled data and a complex training pipeline, thus limiting the practical usage of AST.
Ranked #1 on
Spoken Command Recognition
on Speech Command v2
AST: Audio Spectrogram Transformer
In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels.
Ranked #1 on
Audio Classification
on Speech Commands
PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation
Audio tagging is an active research area and has a wide range of applications.
Ranked #6 on
Audio Classification
on FSD50K
(using extra training data)
Detecting Replay Attacks Using Multi-Channel Audio: A Neural Network-Based Method
With the rapidly growing number of security-sensitive systems that use voice as the primary input, it becomes increasingly important to address these systems' potential vulnerability to replay attacks.
Second-order Non-local Attention Networks for Person Re-identification
Recent efforts have shown promising results for person re-identification by designing part-based architectures to allow a neural network to learn discriminative representations from semantically coherent parts.
Real-Time Adversarial Attacks
In recent years, many efforts have demonstrated that modern machine learning algorithms are vulnerable to adversarial attacks, where small, but carefully crafted, perturbations on the input can make them fail.
ReMASC: Realistic Replay Attack Corpus for Voice Controlled Systems
This paper introduces a new database of voice recordings with the goal of supporting research on vulnerabilities and protection of voice-controlled systems (VCSs).
Towards Learning Fine-Grained Disentangled Representations from Speech
Learning disentangled representations of high-dimensional data is currently an active research area.
Topic Modeling Based Multi-modal Depression Detection
Major depressive disorder is a common mental disorder that affects almost 7% of the adult U. S. population.
An Overview of Vulnerabilities of Voice Controlled Systems
These systems have been shown to be vulnerable to various types of voice spoofing attacks.
How do deep convolutional neural networks learn from raw audio waveforms?
Prior work on speech and audio processing has demonstrated the ability to obtain excellent performance when learning directly from raw audio waveforms using convolutional neural networks (CNNs).
Crafting Adversarial Examples For Speech Paralinguistics Applications
Computational paralinguistic analysis is increasingly being used in a wide range of cyber applications, including security-sensitive applications such as speaker verification, deceptive speech detection, and medical diagnostics.
