Search Results for author:
Found 86 papers, 25 papers with code
A Tree-Structured Decoder for Image-to-Markup Generation
Recent encoder-decoder approaches typically employ string decoders to convert images into serialized strings for image-to-markup.
Joint Optimization of Communication Enhancement and Location Privacy Protection in RIS-Assisted Underwater Communication System
However, in open underwater environments, the location of the source node is highly susceptible to being obtained by eavesdropping nodes through correlation analysis, leading to the issue of location privacy in underwater communication systems, which has been overlooked by many studies.
EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion
Diffusion models have revolutionized the field of talking head generation, yet still face challenges in expressiveness, controllability, and stability in long-time generation.
MVANet: Multi-Stage Video Attention Network for Sound Event Localization and Detection with Source Distance Estimation
We propose a novel output representation that combines the DOA with distance of sound sources by calculating the real Cartesian coordinates to address the newly introduced source distance estimation (SDE) task in the Detection and Classification of Acoustic Scenes and Events (DCASE) 2024 Challenge.
DCF-DS: Deep Cascade Fusion of Diarization and Separation for Speech Recognition under Realistic Single-Channel Conditions
First, we sequentially integrate the NSD and SS modules within a joint training framework, enabling the separation module to leverage speaker time boundaries from the diarization module effectively.
Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention
In multimodal sentiment analysis, collecting text data is often more challenging than video or audio due to higher annotation costs and inconsistent automatic speech recognition (ASR) quality.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation
To address these challenges, we present DAWN (Dynamic frame Avatar With Non-autoregressive diffusion), a framework that enables all-at-once generation of dynamic-length video sequences.
The USTC-NERCSLIP Systems for the CHiME-8 MMCSG Challenge
In the two-person conversation scenario with one wearing smart glasses, transcribing and displaying the speaker's content in real-time is an intriguing application, providing a priori information for subsequent tasks such as translation and comprehension.
See then Tell: Enhancing Key Information Extraction with Vision Grounding
Positioned at the outset of the answer text, the <see> token allows the model to first see--observing the regions of the image related to the input question--and then tell--providing articulated textual responses.
Incorporating Spatial Cues in Modular Speaker Diarization for Multi-channel Multi-party Meetings
This paper proposes a three-stage modular system to enhance single-channel neural speaker diarization systems and recognition performance by utilizing spatial cues from multi-channel speech to provide more accurate initialization for each stage of neural speaker diarization (NSD) decoding: (1) Overlap detection and continuous speech separation (CSS) on multi-channel speech are used to obtain cleaner single speaker speech segments for clustering, followed by the first NSD decoding pass.
DocMamba: Efficient Document Pre-training with State Space Model
In recent years, visually-rich document understanding has attracted increasing attention.
Findings of the 2024 Mandarin Stuttering Event Detection and Automatic Speech Recognition Challenge
The StutteringSpeech Challenge focuses on advancing speech technologies for people who stutter, specifically targeting Stuttering Event Detection (SED) and Automatic Speech Recognition (ASR) in Mandarin.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
The USTC-NERCSLIP Systems for the CHiME-8 NOTSOFAR-1 Challenge
This technical report outlines our submission system for the CHiME-8 NOTSOFAR-1 Challenge.
Topological GCN for Improving Detection of Hip Landmarks from B-Mode Ultrasound Images
In this work, we propose a novel hip landmark detection model by integrating the Topological GCN (TGCN) with an Improved Conformer (TGCN-ICF) into a unified frame-work to improve detection performance.
NAMER: Non-Autoregressive Modeling for Handwritten Mathematical Expression Recognition
Recently, Handwritten Mathematical Expression Recognition (HMER) has gained considerable attention in pattern recognition for its diverse applications in document understanding.
Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios
Evaluation results on the Detection and Classification of Acoustic Scenes and Events (DCASE) 2023 Challenge data set demonstrate significant improvements in SELD performances.
Quantum Compiling with Reinforcement Learning on a Superconducting Processor
To effectively implement quantum algorithms on noisy intermediate-scale quantum (NISQ) processors is a central task in modern quantum technology.
SRFUND: A Multi-Granularity Hierarchical Structure Reconstruction Benchmark in Form Understanding
Accurately identifying and organizing textual content is crucial for the automation of document processing in the field of form understanding.
AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection
The rapid advancements in speech technologies over the past two decades have led to human-level performance in tasks like automatic speech recognition (ASR) for fluent speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A Variance-Preserving Interpolation Approach for Diffusion Models with Applications to Single Channel Speech Enhancement and Recognition
In this paper, we propose a variance-preserving interpolation framework to improve diffusion models for single-channel speech enhancement (SE) and automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation
In recent years, diffusion-based text-to-music (TTM) generation has gained prominence, offering an innovative approach to synthesizing musical content from textual descriptions.
Ranked #1 on
Music Generation
on Song Describer Dataset
SEMv3: A Fast and Robust Approach to Table Separation Line Detection
The `"split-and-merge" paradigm is a pivotal approach to parse table structure, where the table separation line detection is crucial.
Multitask frame-level learning for few-shot sound event detection
This paper focuses on few-shot Sound Event Detection (SED), which aims to automatically recognize and classify sound events with limited samples.
A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition
In this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason.
Bidirectional Trained Tree-Structured Decoder for Handwritten Mathematical Expression Recognition
The Handwritten Mathematical Expression Recognition (HMER) task is a critical branch in the field of OCR.
Continuous Modeling of the Denoising Process for Speech Enhancement Based on Deep Learning
The starting state is noisy speech and the ending state is clean speech.
Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding with Sequence-to-Sequence Architecture
We propose a novel neural speaker diarization system using memory-aware multi-speaker embedding with sequence-to-sequence architecture (NSD-MS2S), which integrates the strengths of memory-aware multi-speaker embedding (MA-MSE) and sequence-to-sequence (Seq2Seq) architecture, leading to improvement in both efficiency and performance.
The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction
This pioneering effort aims to set the first benchmark for the AVTSE task, offering fresh insights into enhancing the ac-curacy of back-end speech recognition systems through AVTSE in challenging and real acoustic environments.
Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023
Three different structures based on attention-guided feature gathering (AFG) are designed for deep feature fusion.
The USTC-NERCSLIP Systems for the CHiME-7 DASR Challenge
This technical report details our submission system to the CHiME-7 DASR Challenge, which focuses on speaker diarization and speech recognition under complex multi-speaker scenarios.
Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder
In this paper, we propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework.
Audio-Visual Speech Recognition
Automatic Speech Recognition
+2
Count, Decode and Fetch: A New Approach to Handwritten Chinese Character Error Correction
This poses a challenge when dealing with an unseen misspelled character, as the decoder may generate an IDS sequence that matches a seen character instead.
Semi-supervised multi-channel speaker diarization with cross-channel attention
Most neural speaker diarization systems rely on sufficient manual training data labels, which are hard to collect under real-world scenarios.
Variance-Preserving-Based Interpolation Diffusion Models for Speech Enhancement
The goal of this study is to implement diffusion models for speech enhancement (SE).
HRDoc: Dataset and Baseline Method Toward Hierarchical Reconstruction of Document Structures
Moreover, we proposed an encoder-decoder-based hierarchical document structure parsing system (DSPS) to tackle this problem.
SEMv2: Table Separation Line Detection Based on Instance Segmentation
Table structure recognition is an indispensable element for enabling machines to comprehend tables.
Multimodal Tree Decoder for Table of Contents Extraction in Document Images
Next, to parse the hierarchical relationship between the heading entities, a tree-structured decoder is designed.
Gradient and Channel Aware Dynamic Scheduling for Over-the-Air Computation in Federated Edge Learning Systems
To further improve the efficiency of wireless data aggregation and model learning, over-the-air computation (AirComp) is emerging as a promising solution by using the superposition characteristics of wireless channels.
Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function
In this paper, we propose a deep learning based multi-speaker direction of arrival (DOA) estimation with audio and visual signals by using permutation-free loss function.
Vision-Language Adaptive Mutual Decoder for OOV-STR
However, in real-world scenarios, out-of-vocabulary (OOV) words are of great importance and SOTA recognition models usually perform poorly on OOV settings.
Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization
Compared with perfect data, quantization poses fundamental challenges on loss of data accuracy, which further impacts the convergence of the algorithms.
TDv2: A Novel Tree-Structured Decoder for Offline Mathematical Expression Recognition
However previous tree decoders converted the tree structure labels into a fixed and ordered sequence, which could not make full use of the diversified expression of tree labels.
Multimodal Pre-training Based on Graph Attention Network for Document Understanding
Its main task is to automatically read, understand, and analyze documents.
A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification
In this paper, we propose two techniques, namely joint modeling and data augmentation, to improve system performances for audio-visual scene classification (AVSC).
A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning
Audio-only-based wake word spotting (WWS) is challenging under noisy conditions due to environmental interference in signal transmission.
The USTC-Ximalaya system for the ICASSP 2022 multi-channel multi-party meeting transcription (M2MeT) challenge
We propose two improvements to target-speaker voice activity detection (TS-VAD), the core component in our proposed speaker diarization system that was submitted to the 2022 Multi-Channel Multi-Party Meeting Transcription (M2MeT) challenge.
Underwater Differential Game: Finite-Time Target Hunting Task with Communication Delay
Simulations show that underwater disturbances have a large impact on the system considering communication delay.
SDN-based Resource Allocation in Edge and Cloud Computing Systems: An Evolutionary Stackelberg Differential Game Approach
Based on this dynamic access model, a Stackelberg differential game based cloud computing resource sharing mechanism is proposed to facilitate the resource trading between the cloud computing service provider (CCP) and different edge computing service providers (ECPs).
Target-speaker Voice Activity Detection with Improved I-Vector Estimation for Unknown Number of Speaker
Target-speaker voice activity detection (TS-VAD) has recently shown promising results for speaker diarization on highly overlapped speech.
Split, embed and merge: An accurate table structure recognizer
However, due to the complexity and diversity in their structure and style, it is very difficult to parse the tabular data into the structured format which machines can understand easily, especially for complex tables.
Ranked #9 on
Table Recognition
on PubTabNet
Separation Guided Speaker Diarization in Realistic Mismatched Conditions
We propose a separation guided speaker diarization (SGSD) approach by fully utilizing a complementarity of speech separation and speaker clustering.
A Lottery Ticket Hypothesis Framework for Low-Complexity Device-Robust Neural Acoustic Scene Classification
We propose a novel neural model compression strategy combining data augmentation, knowledge transfer, pruning, and quantization for device-robust acoustic scene classification (ASC).
USTC-NELSLIP System Description for DIHARD-III Challenge
This system description describes our submission system to the Third DIHARD Speech Diarization Challenge.
Lip-reading with Hierarchical Pyramidal Convolution and Self-Attention
In this paper, we propose a novel deep learning architecture to improving word-level lip-reading.
Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition
The audio-video based emotion recognition aims to classify a given video into basic emotions.
Facial Expression Recognition (FER)
Video Emotion Recognition
The Third DIHARD Diarization Challenge
DIHARD III was the third in a series of speaker diarization challenges intended to improve the robustness of diarization systems to variability in recording equipment, noise conditions, and conversational domain.
Frequency Gating: Improved Convolutional Neural Networks for Speech Enhancement in the Time-Frequency Domain
One of the strengths of traditional convolutional neural networks (CNNs) is their inherent translational invariance.
Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis
Multi-speaker speech recognition of unsegmented recordings has diverse applications such as meeting transcription and automatic subtitle generation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
A Two-Stage Approach to Device-Robust Acoustic Scene Classification
To improve device robustness, a highly desirable key feature of a competitive data-driven acoustic scene classification (ASC) system, a novel two-stage system based on fully convolutional neural networks (CNNs) is proposed.
Ranked #1 on
Acoustic Scene Classification
on TAU Urban Acoustic Scenes 2019
(using extra training data)
A Noise-Aware Memory-Attention Network Architecture for Regression-Based Speech Enhancement
We propose a novel noise-aware memory-attention network (NAMAN) for regression-based speech enhancement, aiming at improving quality of enhanced speech in unseen noise conditions.
Correlating Subword Articulation with Lip Shapes for Embedding Aware Audio-Visual Speech Enhancement
We first extract visual embedding from lip frames using a pre-trained phone or articulation place recognizer for visual-only EASE (VEASE).
On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression
In this paper, we exploit the properties of mean absolute error (MAE) as a loss function for the deep neural network (DNN) based vector-to-vector regression.
Analyzing Upper Bounds on Mean Absolute Errors for Deep Neural Network Based Vector-to-Vector Regression
In this paper, we show that, in vector-to-vector regression utilizing deep neural networks (DNNs), a generalized loss of mean absolute error (MAE) between the predicted and expected feature vectors is upper bounded by the sum of an approximation error, an estimation error, and an optimization error.
An Acoustic Segment Model Based Segment Unit Selection Approach to Acoustic Scene Classification with Partial Utterances
In contrast to building scene models with whole utterances, the ASM-removed sub-utterances, i. e., acoustic utterances without stop acoustic segments, are then used as inputs to the AlexNet-L back-end for final classification.
Device-Robust Acoustic Scene Classification Based on Two-Stage Categorization and Data Augmentation
On Task 1b development data set, we achieve an accuracy of 96. 7\% with a model size smaller than 500KB.
Stroke Constrained Attention Network for Online Handwritten Mathematical Expression Recognition
For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract point-level features from input traces in online mode and employs a CNN encoder to extract pixel-level features from input images in offline mode, then use stroke constrained information to convert them into online and offline stroke-level features.
Joint Architecture and Knowledge Distillation in CNN for Chinese Text Recognition
Finally, the knowledge distillation with multiple losses is adopted to improve performance of the compact CNN.
Speaker detection in the wild: Lessons learned from JSALT 2019
This paper presents the problems and solutions addressed at the JSALT workshop when using a single microphone for speaker detection in adverse scenarios.
Audio and Speech Processing Sound
Challenge of Spatial Cognition for Deep Learning
Given the success of the deep convolutional neural networks (DCNNs) in applications of visual recognition and classification, it would be tantalizing to test if DCNNs can also learn spatial concepts, such as straightness, convexity, left/right, front/back, relative size, aspect ratio, polygons, etc., from varied visual examples of these concepts that are simple and yet vital for spatial reasoning.
Adaptive Period Embedding for Representing Oriented Objects in Aerial Images
While almost all previous object detectors for aerial images directly regress the angle of objects, they use complex rules to calculate the angle, and their performance is limited by the rule design.
Ranked #41 on
Object Detection In Aerial Images
on DOTA
(using extra training data)
The Second DIHARD Diarization Challenge: Dataset, task, and baselines
This paper introduces the second DIHARD challenge, the second in a series of speaker diarization challenges intended to improve the robustness of diarization systems to variation in recording equipment, noise conditions, and conversational domain.
Multi-Task Learning with High-Order Statistics for X-vector based Text-Independent Speaker Verification
The x-vector based deep neural network (DNN) embedding systems have demonstrated effectiveness for text-independent speaker verification.
Deep Neural Network Embeddings with Gating Mechanisms for Text-Independent Speaker Verification
In this paper, gating mechanisms are applied in deep neural network (DNN) training for x-vector-based text-independent speaker verification.
Deep Fusion: An Attention Guided Factorized Bilinear Pooling for Audio-video Emotion Recognition
Although there is no consensus on a definition, human emotional states usually can be apperceived by auditory and visual systems.
Writer-Aware CNN for Parsimonious HMM-Based Offline Handwritten Chinese Text Recognition
Recently, the hybrid convolutional neural network hidden Markov model (CNN-HMM) has been introduced for offline handwritten Chinese text recognition (HCTR) and has achieved state-of-the-art performance.
TextMountain: Accurate Scene Text Detection via Instance Segmentation
In inference stage, each pixel at the mountain foot needs to search the path to the mountaintop and this process can be efficiently completed in parallel, yielding the efficiency of our method compared with others.
Parsimonious HMMs for Offline Handwritten Chinese Text Recognition
Recently, hidden Markov models (HMMs) have achieved promising results for offline handwritten Chinese text recognition.
DenseRAN for Offline Handwritten Chinese Character Recognition
Recently, great success has been achieved in offline handwritten Chinese character recognition by using deep learning methods.
Attention Based Fully Convolutional Network for Speech Emotion Recognition
In this paper, we present a novel attention based fully convolutional network for speech emotion recognition.
Sliding Line Point Regression for Shape Robust Scene Text Detection
Specifically, we first generate the smallest rectangular box including the text with region proposal network (RPN), then isometrically regress the points on the edge of text by using the vertically and horizontally sliding lines.
Ranked #17 on
Scene Text Detection
on SCUT-CTW1500
Trajectory-based Radical Analysis Network for Online Handwritten Chinese Character Recognition
The RNN decoder aims at generating the caption by detecting radicals and spatial structures through an attention model.
Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition
Handwritten mathematical expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols.
A GRU-based Encoder-Decoder Approach with Attention for Online Handwritten Mathematical Expression Recognition
In this study, we present a novel end-to-end approach based on the encoder-decoder framework with the attention mechanism for online handwritten mathematical expression recognition (OHMER).
Radical analysis network for zero-shot learning in printed Chinese character recognition
Chinese characters have a huge set of character categories, more than 20, 000 and the number is still increasing as more and more novel characters continue being created.
Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition
We employ a convolutional neural network encoder that takes HME images as input as the watcher and employ a recurrent neural network decoder equipped with an attention mechanism as the parser to generate LaTeX sequences.
Multi-Objective Learning and Mask-Based Post-Processing for Deep Neural Network Based Speech Enhancement
We propose a multi-objective framework to learn both secondary targets not directly related to the intended task of speech enhancement (SE) and the primary target of the clean log-power spectra (LPS) features to be used directly for constructing the enhanced speech signals.
Sound
