Search Results for author:
Found 72 papers, 24 papers with code
Fuxi-DA: A Generalized Deep Learning Data Assimilation Framework for Assimilating Satellite Observations
Data assimilation (DA), as an indispensable component within contemporary Numerical Weather Prediction (NWP) systems, plays a crucial role in generating the analysis that significantly impacts forecast performance.
Ultrafast Adaptive Primary Frequency Tuning and Secondary Frequency Identification for S/S WPT system
The swift response of SCC and two-step perturb-and-observe algorithm mitigate output disturbances, thereby expediting the frequency tuning process.
INSTRAUG: Automatic Instruction Augmentation for Multimodal Instruction Fine-tuning
Fine-tuning large language models (LLMs) on multi-task instruction-following data has been proven to be a powerful learning paradigm for improving their zero-shot capabilities on new tasks.
Retrieval Augmented End-to-End Spoken Dialog Models
We recently developed SLM, a joint speech and language model, which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability intrinsic to the pretrained LLM.
Extending Context Window of Large Language Models via Semantic Compression
Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses.
SLM: Bridge the thin gap between speech and text foundation models
We present a joint Speech and Language Model (SLM), a multitask, multilingual, and dual-modal model that takes advantage of pretrained foundational speech and language models.
High Perceptual Quality Wireless Image Delivery with Denoising Diffusion Models
We consider the image transmission problem over a noisy wireless channel via deep learning-based joint source-channel coding (DeepJSCC) along with a denoising diffusion probabilistic model (DDPM) at the receiver.
Multimodal Modeling For Spoken Language Identification
Spoken language identification refers to the task of automatically predicting the spoken language in a given utterance.
Self-Adaptive Sampling for Efficient Video Question-Answering on Image--Text Models
Video question-answering is a fundamental task in the field of video understanding.
AudioPaLM: A Large Language Model That Can Speak and Listen
AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2.
Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding
Large Language Models (LLMs) have been applied in the speech domain, often incurring a performance drop due to misaligned between speech and language representations.
Label Aware Speech Representation Learning For Language Identification
In this paper, we propose a novel framework of combining self-supervised representation learning with the language label information for the pre-training task.
LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
The constituent samples of LibriTTS-R are identical to those of LibriTTS, with only the sound quality improved.
Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction
Aspect Sentiment Triplet Extraction (ASTE) is a subtask of Aspect-Based Sentiment Analysis (ABSA) that considers each opinion term, their expressed sentiment, and the corresponding aspect targets.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+2
Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations
Experiments show that Miipher (i) is robust against various audio degradation and (ii) enable us to train a high-quality text-to-speech (TTS) model from restored speech samples collected from the Web.
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Noise2Music: Text-conditioned Music Generation with Diffusion Models
We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts.
Ranked #2 on
Text-to-Music Generation
on MusicCaps
Efficient Domain Adaptation for Speech Foundation Models
The FM encoder adapter and decoder are then finetuned to the target domain with a small amount of supervised in-domain data.
Speech Aware Dialog System Technology Challenge (DSTC11)
The research on this topic is stymied by the lack of a public corpus.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
An Interpretable Neuron Embedding for Static Knowledge Distillation
The visualization of semantic vectors allows for a qualitative explanation of the neural network.
Accelerating RNN-T Training and Inference Using CTC guidance
We propose a novel method to accelerate training and inference process of recurrent neural network transducer (RNN-T) based on the guidance from a co-trained connectionist temporal classification (CTC) model.
SAT: Improving Semi-Supervised Text Classification with Simple Instance-Adaptive Self-Training
Self-training methods have been explored in recent years and have exhibited great performance in improving semi-supervised learning.
MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences
Existing multimodal tasks mostly target at the complete input modality setting, i. e., each modality is either complete or completely missing in both training and test sets.
DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification
This paper proposes a simple yet effective interpolation-based data augmentation approach termed DoubleMix, to improve the robustness of models in text classification.
SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning
With the boom of e-commerce, Multimodal Review Helpfulness Prediction (MRHP), which aims to sort product reviews according to the predicted helpfulness scores has become a research hotspot.
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge.
Ranked #1 on
Text-to-Image Generation
on LAION COCO
Accented Speech Recognition: Benchmarking, Pre-training, and Diverse Data
However, there are not enough data sets for accented speech, and for the ones that are already available, more training approaches need to be explored to improve the quality of accented speech recognition.
Unsupervised Data Selection via Discrete Speech Representation for ASR
Self-supervised learning of speech representations has achieved impressive results in improving automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Co-training Transformer with Videos and Images Improves Action Recognition
We term this approach as Co-training Videos and Images for Action Recognition (CoVeR).
Ranked #8 on
Action Classification
on MiT
(using extra training data)
A Tensor-BTD-based Modulation for Massive Unsourced Random Access
In this letter, we propose a novel tensor-based modulation scheme for massive unsourced random access.
Harmonic Retrieval with $L_1$-Tucker Tensor Decomposition
We also develop a novel subcarrier recovery method for the proposed model.
TAPL: Dynamic Part-based Visual Tracking via Attention-guided Part Localization
Holistic object representation-based trackers suffer from performance drop under large appearance change such as deformation and occlusion.
Universal Paralinguistic Speech Representations Using Self-Supervised Conformers
Many speech applications require understanding aspects beyond the words being spoken, such as recognizing emotion, detecting whether the speaker is wearing a mask, or distinguishing real from synthetic speech.
Multi-trends Enhanced Dynamic Micro-video Recommendation
In this paper, we propose the DMR framework to explicitly model dynamic multi-trends of users' current preference and make predictions based on both the history and future potential trends.
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio.
Ranked #1 on
Speech Recognition
on Common Voice
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis
In this work, we propose a framework named MultiModal InfoMax (MMIM), which hierarchically maximizes the Mutual Information (MI) in unimodal input pairs (inter-modality) and between multimodal fusion result and unimodal input in order to maintain task-related information through multimodal fusion.
Ranked #5 on
Multimodal Sentiment Analysis
on CMU-MOSI
W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training
In particular, when compared to published models such as conformer-based wav2vec~2. 0 and HuBERT, our model shows~5\% to~10\% relative WER reduction on the test-clean and test-other subsets.
Ranked #1 on
Speech Recognition
on LibriSpeech test-clean
(using extra training data)
Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis
Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data.
Supervised Contrastive Learning for Accented Speech Recognition
Neural network based speech recognition systems suffer from performance degradation due to accented speech, especially unfamiliar accents.
Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models
To improve streaming models, a recent study [1] proposed to distill a non-streaming teacher model on unsupervised utterances, and then train a streaming student using the teachers' predictions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Exploring Targeted Universal Adversarial Perturbations to End-to-end ASR Models
To attack RNN-T, we find prepending perturbation is more effective than the additive perturbation, and can mislead the models to predict the same short target on utterances of arbitrary length.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
A Better and Faster End-to-End Model for Streaming ASR
To address this, we explore replacing the LSTM layers in the encoder of our E2E model with Conformer layers [4], which has shown good improvements for ASR.
Audio and Speech Processing Sound
Superconductor-metal quantum transition at the EuO-KTaO3 interface
Superconductivity has been one of the most fascinating quantum states of matter for over several decades.
Superconductivity Mesoscale and Nanoscale Physics Materials Science
Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data
We propose a novel and effective learning method by leveraging a non-streaming ASR model as a teacher to generate transcripts on an arbitrarily large data set, which is then used to distill knowledge into streaming ASR models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization
FastEmit also improves streaming ASR accuracy from 4. 4%/8. 9% to 3. 1%/7. 5% WER, meanwhile reduces 90th percentile latency from 210 ms to only 30 ms on LibriSpeech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset.
Ranked #1 on
Speech Recognition
on LibriSpeech test-clean
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Answer-checking in Context: A Multi-modal FullyAttention Network for Visual Question Answering
From the human perspective, to answer a visual question, one needs to read the question and then refer to the image to generate an answer.
Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
Streaming automatic speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible, while full-context ASR waits for the completion of a full speech utterance before emitting completed hypotheses.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering
Positional information of text is underused and there is a lack of evidence for the generated answer.
Optical Character Recognition
Optical Character Recognition (OCR)
+2
Dialogue Relation Extraction with Document-level Heterogeneous Graph Attention Networks
This graph is fed to a graph attention network for context propagation among relevant nodes, which effectively captures the dialogue context.
Ranked #7 on
Dialog Relation Extraction
on DialogRE
(F1c (v1) metric)
Improved Noisy Student Training for Automatic Speech Recognition
Noisy student training is an iterative self-training method that leverages augmentation to improve network performance.
Ranked #5 on
Speech Recognition
on LibriSpeech test-clean
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Conformer: Convolution-augmented Transformer for Speech Recognition
Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs).
Ranked #12 on
Speech Recognition
on LibriSpeech test-other
(using extra training data)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
We demonstrate that on the widely used LibriSpeech benchmark, ContextNet achieves a word error rate (WER) of 2. 1%/4. 6% without external language model (LM), 1. 9%/4. 1% with LM and 2. 9%/7. 0% with only 10M parameters on the clean/noisy LibriSpeech test sets.
Ranked #12 on
Speech Recognition
on LibriSpeech test-clean
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions
On a long-form YouTube test set, when the nonstreaming RNN-T model is trained with shorter segments of data, the proposed combination improves word error rate (WER) from 22. 3% to 14. 8%; when the streaming RNN-T model trained on short Search queries, the proposed techniques improve WER on the YouTube set from 67. 0% to 25. 3%.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Streaming Object Detection for 3-D Point Clouds
This built-in data capture latency is artificial, and based on treating the point cloud as a camera image in order to leverage camera-inspired architectures.
FFusionCGAN: An end-to-end fusion method for few-focus images using conditional GAN in cytopathological digital slides
However, although existing image fusion techniques, including traditional algorithms and deep learning-based algorithms, can generate high-quality fused images, they need multiple images with different focus depths in the same field of view.
Scalability in Perception for Autonomous Driving: Waymo Open Dataset
In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset.
A comparison of end-to-end models for long-form speech recognition
In this paper, we both investigate and improve the performance of end-to-end models on long-form transcription.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Statistical Inference in Mean-Field Variational Bayes
We conduct non-asymptotic analysis on the mean-field variational inference for approximating posterior distributions in complex Bayesian models that may involve latent variables.
StarNet: Targeted Computation for Object Detection in Point Clouds
We show how our redesign---namely using only local information and using sampling instead of learned proposals---leads to a significantly more flexible and adaptable system: we demonstrate how we can vary the computational cost of a single trained StarNet without retraining, and how we can target proposals towards areas of interest with priors and heuristics.
A Strategy of MR Brain Tissue Images' Suggestive Annotation Based on Modified U-Net
How to choose appropriate training dataset from limited labeled dataset rather than the whole also has great significance in saving training time.
Image Super-Resolution via Dual-State Recurrent Networks
Advances in image super-resolution (SR) have recently benefited significantly from rapid developments in deep neural networks.
Ranked #42 on
Image Super-Resolution
on BSD100 - 4x upscaling
Learning $3$D-FilterMap for Deep Convolutional Neural Networks
Due to the weight sharing scheme, the parameter size of the $3$D-FilterMap is much smaller than that of the filters to be learned in the conventional convolution layer when $3$D-FilterMap generates the same number of filters.
Dilated Recurrent Neural Networks
To provide a theory-based quantification of the architecture's advantages, we introduce a memory capacity measure, the mean recurrent length, which is more suitable for RNNs with long skip connections than existing measures.
Ranked #24 on
Sequential Image Classification
on Sequential MNIST
A Learning-Based Approach for Lane Departure Warning Systems with a Personalized Driver Model
Second, based on this model, we develop an online model-based prediction algorithm to predict the forthcoming vehicle trajectory and judge whether the driver will demonstrate an LDB or a DCB.
Robust Single Image Super-Resolution via Deep Networks With Sparse Prior
We demonstrate that a sparse coding model particularly designed for SR can be incarnated as a neural network with the merit of end-to-end optimization over training data.
Seq-NMS for Video Object Detection
Video object detection is challenging because objects that are easily detected in one frame may be difficult to detect in another frame within the same clip.
Deep Networks for Image Super-Resolution with Sparse Prior
We show that a sparse coding model particularly designed for super-resolution can be incarnated as a neural network, and trained in a cascaded structure from end to end.
Learning Semantic Relationships for Better Action Retrieval in Images
Human actions capture a wide variety of interactions between people and objects.
Self-Tuned Deep Super Resolution
Deep learning has been successfully applied to image super resolution (SR).
An Analysis of Unsupervised Pre-training in Light of Recent Advances
We discover unsupervised pre-training, as expected, helps when the ratio of unsupervised to supervised samples is high, and surprisingly, hurts when the ratio is low.
Ranked #92 on
Image Classification
on STL-10
