Search Results for author:
Found 81 papers, 36 papers with code
ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing
While end-to-end video-to-audio generation has greatly improved, producing high-fidelity audio that authentically captures the nuances of visual content remains challenging.
Graceful Forgetting in Generative Language Models
To bridge this gap, in this paper we propose a novel framework, Learning With Forgetting (LWF), to achieve graceful forgetting in generative language models.
MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
Each item in the benchmark demands multi-step deep reasoning beyond surface-level understanding.
J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judge
The current focus of AI research is shifting from emphasizing model training towards enhancing evaluation quality, a transition that is crucial for driving further advancements in AI systems.
SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
Moreover, to assess the effectiveness of song aesthetic evaluation, we conduct experiments using SongEval to predict aesthetic scores and demonstrate better performance than existing objective evaluation metrics in predicting human-perceived musical quality.
CMD: Controllable Multiview Diffusion for 3D Editing and Progressive Generation
This conditional multiview diffusion model not only allows the generation of 3D models part by part but also enables local editing of 3D models according to the local revision of the input image without changing other 3D parts.
Co$^{3}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion
Considering the asymmetric body dynamics of two speakers, our framework is built upon two cooperative generation branches conditioned on separated speaker audio.
DB-GNN: Dual-Branch Graph Neural Network with Multi-Level Contrastive Learning for Jointly Identifying Within- and Cross-Frequency Coupled Brain Networks
Secondly, to more fully perceive the global information of cross-frequency and within-frequency coupling, the global perception branch of DB-GNN adopts a Transformer architecture.
OmniAudio: Generating Spatial Audio from 360-Degree Video
To generate spatial audio from 360-degree video, we propose a novel framework OmniAudio, which leverages self-supervised pre-training using both spatial audio data (in FOA format) and large-scale non-spatial data.
AudioX: Diffusion Transformer for Anything-to-Audio Generation
Audio and music generation have emerged as crucial tasks in many applications, yet existing approaches face significant limitations: they operate in isolation without unified capabilities across modalities, suffer from scarce high-quality, multi-modal training data, and struggle to effectively integrate diverse inputs.
YuE: Scaling Open Foundation Models for Long-Form Music Generation
We tackle the task of long-form music generation--particularly the challenging \textbf{lyrics-to-song} problem--by introducing YuE, a family of open foundation models based on the LLaMA2 architecture.
Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
Recent advancements in large language models (LLMs) have driven significant progress in zero-shot text-to-speech (TTS) synthesis.
LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement
In this paper, we introduce LLaSE-G1, a LLaMA-based language model that incentivizes generalization capabilities for speech enhancement.
Delta Decompression for MoE-based LLMs Compression
Based on observations of expert diversity, we decompose their weights into a shared base weight and unique delta weights.
Audio-FLAN: A Preliminary Release
Audio-FLAN lays the foundation for unified audio-language models that can seamlessly handle both understanding (e. g., transcription, comprehension) and generation (e. g., speech, music, sound) tasks across a wide range of audio domains in a zero-shot manner.
VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer
Crafting magic and illusions is one of the most thrilling aspects of filmmaking, with visual effects (VFX) serving as the powerhouse behind unforgettable cinematic experiences.
Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
Recent advances in text-based large language models (LLMs), particularly in the GPT series and the o1 model, have demonstrated the effectiveness of scaling both training-time and inference-time compute.
Every Angle Is Worth A Second Glance: Mining Kinematic Skeletal Structures from Multi-view Joint Cloud
Multi-person motion capture over sparse angular observations is a challenging problem under interference from both self- and mutual-occlusions.
PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing
It is found that directly applying multiview diffusion on single-view human images leads to severe geometric distortions, especially on generated faces.
SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model
The differences between human talking and singing limit the performance of existing talking face video generation models when applied to singing.
Foundation Cures Personalization: Recovering Facial Personalized Models' Prompt Consistency
Facial personalization represents a crucial downstream task in the domain of text-to-image generation.
LEADRE: Multi-Faceted Knowledge Enhanced LLM Empowered Display Advertisement Recommender System
Display advertising provides significant value to advertisers, publishers, and users.
EVA: An Embodied World Model for Future Video Anticipation
It leverages the complementary strengths of pre-trained vision-language and video generation models, enabling them to function as a world model in embodied scenarios.
FlashAudio: Rectified Flows for Fast and High-Fidelity Text-to-Audio Generation
To alleviate the inefficient timesteps allocation and suboptimal distribution of noise, FlashAudio optimizes the time distribution of rectified flow with Bifocal Samplers and proposes immiscible flow to minimize the total distance of data-noise pairs in a batch vias assignment.
Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
However, when it comes to stereo audio generation, the soundscapes often have a complex scene of multiple objects and directions.
Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer
For more precise and fine-grained melody control, we introduce a novel top-$k$ constant-Q Transform representation as the melody prompt, reducing ambiguity compared to previous representations (e. g., chroma), particularly for music with multiple tracks or a wide range of pitch values.
You Know What I'm Saying: Jailbreak Attack via Implicit Reference
While recent advancements in large language model (LLM) alignment have enabled the effective identification of malicious objectives involving scene nesting and keyword rewriting, our study reveals that these methods remain inadequate at detecting malicious objectives expressed through context within nested harmless objectives.
PSHuman: Photorealistic Single-view Human Reconstruction using Cross-Scale Diffusion
Detailed and photorealistic 3D human modeling is essential for various applications and has seen tremendous progress.
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
By enhancing the semantic ability of the codec, X-Codec significantly reduces WER in speech synthesis tasks and extends these benefits to non-speech applications, including music and sound generation.
AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems
However, configuring an MAS for a task remains challenging, with performance only observable post-execution.
Importance Weighting Can Help Large Language Models Self-Improve
Large language models (LLMs) have shown remarkable capability in numerous tasks and applications.
NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models
This approach allows the model to more precisely adapt to specific tasks while maintaining a compact parameter space.
STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
In this paper, we present the first structural binarization method for LLM compression to less than 1-bit precision.
Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation
Yet scant research explores the details of how these LLMs perform on advanced music understanding and conditioned generation, especially from the multi-step reasoning perspective, which is a critical aspect in the conditioned, editable, and interactive human-computer co-creation process.
MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions
To fulfill this gap, we present MMTrail, a large-scale multi-modality video-language dataset incorporating more than 20M trailer clips with visual captions, and 2M high-quality clips with multimodal captions.
VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling
In this work, we systematically study music generation conditioned solely on the video.
LLMs Meet Multimodal Generation and Editing: A Survey
With the recent advancement in large language models (LLMs), there is a growing interest in combining LLMs with multimodal learning.
CoCoGesture: Toward Coherent Co-speech 3D Gesture Generation in the Wild
Here, we construct the audio ControlNet through a trainable copy of our pre-trained diffusion model.
VAE-Var: Variational-Autoencoder-Enhanced Variational Assimilation
Data assimilation refers to a set of algorithms designed to compute the optimal estimate of a system's state by refining the prior prediction (known as background states) using observed data.
FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generation
By intensive experimental studies, we demonstrate that the proposed method can generate better samples than SingSong, and accelerate the generation by at least 30 times.
ComposerX: Multi-Agent Symbolic Music Composition with LLMs
Music composition represents the creative side of humanity, and itself is a complex task that requires abilities to understand and generate information with long dependency and harmony constraints.
FlashSpeech: Efficient Zero-Shot Speech Synthesis
The generation processes of FlashSpeech can be achieved efficiently with one or two sampling steps while maintaining high audio quality and high similarity to the audio prompt for zero-shot speech generation.
Information Re-Organization Improves Reasoning in Large Language Models
In this paper, we propose an information re-organization (InfoRE) method before proceeding with the reasoning to enhance the reasoning ability of LLMs.
MuPT: A Generative Symbolic Music Pretrained Transformer
In this paper, we explore the application of Large Language Models (LLMs) to the pre-training of music.
RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation
To this end, we propose learning to Refine Query for Retrieval Augmented Generation (RQ-RAG) in this paper, endeavoring to enhance the model by equipping it with capabilities for explicit rewriting, decomposition, and disambiguation.
Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm
Applying such paradigm, we construct an evaluation benchmark in the field of urology, including a LCP, a SPs dataset, and an automated RAE.
ChatMusician: Understanding and Generating Music Intrinsically with LLM
It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language.
Ads Recommendation in a Collapsed and Entangled World
We present Tencent's ads recommendation system and examine the challenges and practices of learning appropriate recommendation representations.
RJUA-MedDQA: A Multimodal Benchmark for Medical Document Question Answering and Clinical Reasoning
Recent advancements in Large Language Models (LLMs) and Large Multi-modal Models (LMMs) have shown potential in various medical applications, such as Intelligent Medical Diagnosis.
CoMoSVC: Consistency Model-based Singing Voice Conversion
The diffusion-based Singing Voice Conversion (SVC) methods have achieved remarkable performances, producing natural audios with high similarity to the target timbre.
FM-OV3D: Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detection
In this work, we propose FM-OV3D, a method of Foundation Model-based Cross-modal Knowledge Blending for Open-Vocabulary 3D Detection, which improves the open-vocabulary localization and recognition abilities of 3D model by blending knowledge from multiple pre-trained foundation models, achieving true open-vocabulary without facing constraints from original 3D datasets.
FengWu-4DVar: Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation
Weather forecasting is a crucial yet highly challenging task.
RJUA-QA: A Comprehensive QA Dataset for Urology
We introduce RJUA-QA, a novel medical dataset for question answering (QA) and reasoning with clinical evidence, contributing to bridge the gap between general large language models (LLMs) and medical-specific LLM applications.
Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation
In addition, the lack of large-scale available datasets with emotional transition speech and corresponding 3D human gestures also limits the addressing of this task.
Continual Learning with Dirichlet Generative-based Rehearsal
Recent advancements in data-driven task-oriented dialogue systems (ToDs) struggle with incremental learning due to computational constraints and time-consuming issues.
PUMGPT: A Large Vision-Language Model for Product Understanding
E-commerce platforms benefit from accurate product understanding to enhance user experience and operational efficiency.
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost.
LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT
We introduce LyricWhiz, a robust, multilingual, and zero-shot automatic lyrics transcription method achieving state-of-the-art performance on various lyrics transcription datasets, even in challenging genres such as rock and metal.
MARBLE: Music Audio Representation Benchmark for Universal Evaluation
This is evident in the limited work on deep music representations, the scarcity of large-scale datasets, and the absence of a universal and community-driven benchmark.
ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
Note that, our method can be regarded as a novel transfer paradigm for large-scale models, delivering promising results in adaptation to continually changing distributions.
NAS-FM: Neural Architecture Search for Tunable and Interpretable Sound Synthesis based on Frequency Modulation
Since expert knowledge is hard to acquire, it hinders the flexibility to quickly design and tune digital synthesizers for diverse sounds.
Insert or Attach: Taxonomy Completion via Box Embedding
Taxonomy completion, enriching existing taxonomies by inserting new concepts as parents or attaching them as children, has gained significant interest.
CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
In this paper, we propose a "Co"nsistency "Mo"del-based "Speech" synthesis method, CoMoSpeech, which achieve speech synthesis through a single diffusion sampling step while achieving high audio quality.
RECIST Weakly Supervised Lesion Segmentation via Label-Space Co-Training
As an essential indicator for cancer progression and treatment response, tumor size is often measured following the response evaluation criteria in solid tumors (RECIST) guideline in CT slices.
Pathway to Future Symbiotic Creativity
This report presents a comprehensive view of our vision on the development path of the human-machine symbiotic art creation.
Efficient Climate Simulation via Machine Learning Method
We develop a user-friendly platform NeuroGCM for efficiently developing hybrid modeling in climate simulation.
Masked Spatial-Spectral Autoencoders Are Excellent Hyperspectral Defenders
Deep learning methodology contributes a lot to the development of hyperspectral image (HSI) analysis community.
Transferable Physical Attack against Object Detection with Separable Attention
Transferable adversarial attack is always in the spotlight since deep learning models have been demonstrated to be vulnerable to adversarial samples.
Stereo Matching with Cost Volume based Sparse Disparity Propagation
Stereo matching is crucial for binocular stereo vision.
metaCAT: A Metadata-based Task-oriented Chatbot Annotation Tool
Creating high-quality annotated dialogue corpora is challenging.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Few-shot Object Detection with Self-adaptive Attention Network for Remote Sensing Images
Particularly, in order to fit the object detection settings, our proposed few-shot detector concentrates on the relations that lie in the level of objects instead of the full image with the assistance of Self-Adaptive Attention Network (SAAN).
Few-shot Object Detection with Feature Attention Highlight Module in Remote Sensing Images
Then the object-specific features are delivered to the two-stage detection backend for the detection results.
Neural Kalman Filtering for Speech Enhancement
Statistical signal processing based speech enhancement methods adopt expert knowledge to design the statistical models and linear filters, which is complementary to the deep neural network (DNN) based methods which are data-driven.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
SkipConvNet: Skip Convolutional Neural Network for Speech Dereverberation using Optimally Smoothed Spectral Mapping
The reliability of using fully convolutional networks (FCNs) has been successfully demonstrated by recent studies in many speech applications.
Re-examining the Solar Axion Explanation for the XENON1T Excess
Meanwhile, they can also scatter with the atoms through the inverse Primakoff process via the axion-photon coupling, which emits a photon and mimics the electronic recoil signals.
High Energy Physics - Phenomenology High Energy Physics - Experiment
An Adaptive Remote Stochastic Gradient Method for Training Neural Networks
RSG is further combined with adaptive methods to construct ARSG for acceleration.
Aspect Based Sentiment Analysis with Gated Convolutional Networks
Aspect based sentiment analysis (ABSA) can provide more detailed information than general sentiment analysis, because it aims to predict the sentiment polarities of the given aspects or entities in text.
Aspect-Based Sentiment Analysis
Aspect Category Sentiment Analysis
Serendipity in dark photon searches
Searches for dark photons provide serendipitous discovery potential for other types of vector particles.
High Energy Physics - Phenomenology High Energy Physics - Experiment
MTNA: A Neural Multi-task Model for Aspect Category Classification and Aspect Term Extraction On Restaurant Reviews
Online reviews are valuable resources not only for consumers to make decisions before purchase, but also for providers to get feedbacks for their services or commodities.
PBODL : Parallel Bayesian Online Deep Learning for Click-Through Rate Prediction in Tencent Advertising System
Then we extend the model family to a variety of bayesian online models with increasing feature embedding capabilities, such as Sparse-MLP, FM-MLP and FFM-MLP.
