Search Results for author:
Found 39 papers, 17 papers with code
Enhancing Audiovisual Speech Recognition through Bifocal Preference Optimization
Audiovisual Automatic Speech Recognition (AV-ASR) aims to improve speech recognition accuracy by leveraging visual signals.
Two-in-One: Unified Multi-Person Interactive Motion Generation by Latent Diffusion Transformer
Multi-person interactive motion generation, a critical yet under-explored domain in computer character animation, poses significant challenges such as intricate modeling of inter-human interactions beyond individual motions and generating two motions with huge differences from one text condition.
BSharedRAG: Backbone Shared Retrieval-Augmented Generation for the E-commerce Domain
Retrieval Augmented Generation (RAG) system is important in domains such as e-commerce, which has many long-tail entities and frequently updated information.
ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech
Neural codecs have become crucial to recent speech and audio generation research.
Robust Audiovisual Speech Recognition Models with Mixture-of-Experts
Moreover, to incorporate visual information effectively, we inject visual information into the ASR model through a mixture-of-experts module.
See or Guess: Counterfactually Regularized Image Captioning
Image captioning, which generates natural language descriptions of the visual information in an image, is a crucial task in vision-language research.
Towards Effective and Efficient Continual Pre-training of Large Language Models
To make the CPT approach more traceable, this paper presents a technical report for continually pre-training Llama-3 (8B), which significantly enhances the Chinese language ability and scientific reasoning ability of the backbone model.
YuLan: An Open-source Large Language Model
This paper presents the development of YuLan, a series of open-source LLMs with $12$ billion parameters.
SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition
Existing speech language models typically utilize task-dependent prompt tokens to unify various speech tasks in a single model.
Intelligent Virtual Assistants with LLM-based Process Automation
This paper proposes a novel LLM-based virtual assistant that can automatically perform multi-step operations within mobile apps based on high-level user requests.
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
By conducting a comprehensive empirical study, we find that instructions focused on complex visual reasoning tasks are particularly effective in improving the performance of MLLMs on evaluation benchmarks.
Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models
In this paper, we introduce Parrot, a solution aiming to enhance multi-turn instruction following for LLMs.
ViCo: Engaging Video Comment Generation with Human Preference Rewards
Experiments on ViCo-20k show that the comments generated by our ViCo model exhibit the best performance in terms of both quantitative and qualitative results, particularly when engagement is considered.
Transferring Foundation Models for Generalizable Robotic Manipulation
In this paper, we propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models, to condition robot manipulation tasks.
User Behavior Simulation with Large Language Model based Agents
Simulating high quality user behavior data has always been a fundamental problem in human-centered applications, where the major difficulty originates from the intricate mechanism of human decision process.
AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation
We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks, such as making a smiley face using building blocks.
ComedicSpeech: Text To Speech For Stand-up Comedies in Low-Resource Scenarios
Considering comedians have diverse personal speech styles, including personal prosody, rhythm, and fillers, it requires real-world datasets and strong speech style modeling capabilities, which brings challenges.
TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World
Experimental results indicate that the models incorporating large language models (LLM) can generate more diverse responses, while the model utilizing knowledge graphs to introduce external knowledge performs the best overall.
Text2Poster: Laying out Stylized Texts on Retrieved Images
Poster generation is a significant task for a wide range of applications, which is often time-consuming and requires lots of manual editing and artistic experience.
TeViS:Translating Text Synopses to Video Storyboards
In this paper, we propose a new task called Text synopsis to Video Storyboard (TeViS) which aims to retrieve an ordered sequence of images as the video storyboard to visualize the text synopsis.
VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing
In this paper, we propose a machine translation system tailored for the task of video dubbing, which directly considers the speech duration of each token in translation, to match the length of source and target speech.
Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning
Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks.
Ranked #2 on
Video Retrieval
on QuerYD
(using extra training data)
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment
and 2) how to mitigate the impact of these factors?
Ranked #2 on
Video Retrieval
on MSR-VTT-1kA
(using extra training data)
Multi-Modal Experience Inspired AI Creation
To model such human capabilities, in this paper, we define and solve a novel AI creation problem based on human experiences.
Self-supervised Context-aware Style Representation for Expressive Speech Synthesis
In this paper, we propose a novel framework for learning style representation from abundant plain text in a self-supervised manner.
AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios
We model the speaker characteristics systematically to improve the generalization on new speakers.
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
Class-aware Sounding Objects Localization via Audiovisual Correspondence
To address this problem, we propose a two-stage step-by-step learning framework to localize and recognize sounding objects in complex audiovisual scenarios using only the correspondence between audio and vision.
Towards artificial general intelligence via a multimodal foundation model
To overcome this limitation and take a solid step towards artificial general intelligence (AGI), we develop a foundation model pre-trained with huge multimodal data, which can be quickly adapted for various downstream cognitive tasks.
Stylistic Retrieval-based Dialogue System with Unparallel Training Data
The ability of a dialog system to express consistent language style during conversations has a direct, positive impact on its usability and on user satisfaction.
Pre-Trained Models: Past, Present and Future
Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI).
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
We further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model.
Ranked #1 on
Image Retrieval
on RUC-CAS-WenLan
ScriptWriter: Narrative-Guided Script Generation
In dialogue systems, it would also be useful to drive dialogues by a dialogue plan.
"Love is as Complex as Math": Metaphor Generation System for Social Chatbot
By embedding the framework into a chatbot system, we then enables the chatbot to communicate with users using figurative language.
Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences
A storyboard is a sequence of images to illustrate a story containing multiple sentences, which has been a key process to create different story products.
Composing a Picture Book by Automatic Story Understanding and Visualization
Pictures can enrich storytelling experiences.
Image Inspired Poetry Generation in XiaoIce
This is one of the few attempts to generate poetry from images.
A World of Difference: Divergent Word Interpretations among People
We propose an approach that quantifies semantic differences in interpretations among different groups of people.
