Search Results for author:
Found 22 papers, 8 papers with code
InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training
In this paper, we introduce a simple and scalable training method called InSerter, which stands for Interleaved Speech-Text Representation Pre-training.
Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuray
Establishing the long-context capability of large vision-language models is crucial for video understanding, high-resolution image understanding, multi-modal agents and reasoning.
LUCY: Linguistic Understanding and Control Yielding Early Stage of Her
The film Her features Samantha, a sophisticated AI audio agent who is capable of understanding both linguistic and paralinguistic information in human speech and delivering real-time responses that are natural, informative and sensitive to emotional subtleties.
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing interaction.
Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
Our main contribution is that the speech input and output modalities can be easily connected to a textual LLM while keeping the LLM's parameters frozen throughout the training process.
VITA: Towards Open-Source Interactive Omni Multimodal LLM
The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas.
QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
The challenge focuses on the uncertainty quantification of medical image segmentation which considers the omnipresence of inter-rater variability in imaging datasets.
Interacting Particle Systems on Networks: joint inference of the network and the interaction kernel
Modeling multi-agent systems on networks is a fundamental challenge in a wide variety of disciplines.
Optimal minimax rate of learning interaction kernels
Our tLSE method offers a straightforward approach for establishing the optimal minimax rate for models with either local or nonlocal dependency.
FedSN: A Federated Learning Framework over Heterogeneous LEO Satellite Networks
To this end, we propose FedSN as a general FL framework to tackle the above challenges, and fully explore data diversity on LEO satellites.
Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture
We believe that the combination is complementary and able to solve the inherent difficulties of using one modality input, including occlusions, extreme lighting/texture, and out-of-view for visual mocap and global drifts for inertial mocap.
Ranked #1 on
3D Human Pose Estimation
on AIST++
DCCRN-KWS: an audio bias based model for noise robust small-footprint keyword spotting
Real-world complex acoustic environments especially the ones with a low signal-to-noise ratio (SNR) will bring tremendous challenges to a keyword spotting (KWS) system.
Two Stage Contextual Word Filtering for Context bias in Unified Streaming and Non-streaming Transducer
In this paper, we propose an efficient approach to obtain a high quality contextual list for a unified streaming/non-streaming based E2E model.
A Data-Adaptive Prior for Bayesian Learning of Kernels in Operators
However, a fixed non-degenerate prior leads to a divergent posterior mean when the observation noise becomes small, if the data induces a perturbation in the eigenspace of zero eigenvalues of the inversion operator.
Biomedical image analysis competitions: The state of current participation practice
Of these, 84% were based on standard architectures.
Decentralized Task Offloading in Edge Computing: A Multi-User Multi-Armed Bandit Approach
Mobile edge computing facilitates users to offload computation tasks to edge servers for meeting their stringent delay requirements.
Mean Field Equilibrium in Multi-Armed Bandit Game with Continuous Reward
In this paper, we study the mean field bandit game with a continuous reward function.
WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit
In this paper, we propose an open source, production first, and production ready speech recognition toolkit called WeNet in which a new two-pass approach is implemented to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model.
Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition
In this paper, we present a novel two-pass approach to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model.
Ranked #12 on
Speech Recognition
on AISHELL-1
Cascade RNN-Transducer: Syllable Based Streaming On-device Mandarin Speech Recognition with a Syllable-to-Character Converter
End-to-end models are favored in automatic speech recognition (ASR) because of its simplified system structure and superior performance.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
The SLT 2021 children speech recognition challenge: Open datasets, rules and baselines
Automatic speech recognition (ASR) has been significantly advanced with the use of deep learning and big data.
Sound Audio and Speech Processing
IEEE SLT 2021 Alpha-mini Speech Challenge: Open Datasets, Tracks, Rules and Baselines
In this challenge, we open source a sizable speech, keyword, echo and noise corpus for promoting data-driven methods, particularly deep-learning approaches on KWS and SSL.
Sound Audio and Speech Processing
