Search Results for author:
Found 15 papers, 3 papers with code
Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection
Although Large Language Models (LLMs) have shown promise for human-like conversations, they are primarily pre-trained on text data.
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
Combining the visual modality with pretrained language models has been surprisingly effective for simple descriptive tasks such as image captioning.
Speech Summarization using Restricted Self-Attention
End-to-end modeling of speech summarization models is challenging due to memory and compute constraints arising from long input audio sequences.
How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language
Towards this end, we introduce How2Sign, a multimodal and multiview continuous American Sign Language (ASL) dataset, consisting of a parallel corpus of more than 80 hours of sign language videos and a set of corresponding modalities including speech, English transcripts, and depth.
Towards Understanding ASR Error Correction for Medical Conversations
Domain Adaptation for Automatic Speech Recognition (ASR) error correction via machine translation is a useful technique for improving out-of-domain outputs of pre-trained ASR systems to obtain optimal results for specific in-domain tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
ASR Error Correction and Domain Adaptation Using Machine Translation
We propose a simple technique to perform domain adaptation for ASR error correction via machine translation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Multimodal Abstractive Summarization for How2 Videos
In this paper, we study abstractive summarization for open-domain videos.
Ranked #1 on
Text Summarization
on How2
Learned In Speech Recognition: Contextual Acoustic Word Embeddings
End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not require a lexicon.
Learning from Multiview Correlations in Open-Domain Videos
This shows it is possible to learn reliable representations across disparate, unaligned and noisy modalities, and encourages using the proposed approach on larger datasets.
Multimodal Grounding for Sequence-to-Sequence Speech Recognition
Specifically, in our previous work, we propose a multistep visual adaptive training approach which improves the accuracy of an audio-based Automatic Speech Recognition (ASR) system.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
How2: A Large-scale Dataset for Multimodal Language Understanding
In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Acoustic-to-Word Recognition with Sequence-to-Sequence Models
We present effective methods to train Sequence-to-Sequence models for direct word-level recognition (and character-level recognition) and show an absolute improvement of 4. 4-5. 0\% in Word Error Rate on the Switchboard corpus compared to prior work.
End-to-End Multimodal Speech Recognition
Transcription or sub-titling of open-domain videos is still a challenging domain for Automatic Speech Recognition (ASR) due to the data's challenging acoustics, variable signal processing and the essentially unrestricted domain of the data.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Linguistic unit discovery from multi-modal inputs in unwritten languages: Summary of the "Speaking Rosetta" JSALT 2017 Workshop
We summarize the accomplishments of a multi-disciplinary workshop exploring the computational and scientific issues surrounding the discovery of linguistic units (subwords and words) in a language without orthography.
Combining LSTM and Latent Topic Modeling for Mortality Prediction
There is a great need for technologies that can predict the mortality of patients in intensive care units with both high accuracy and accountability.
