Search Results for author:
Found 110 papers, 40 papers with code
Improving Deep Embedded Clustering via Learning Cluster-level Representations
In this paper, we propose a novel DEC model, which we named the deep embedded clustering model with cluster-level representation learning (DECCRL) to jointly learn cluster and instance level representations.
SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model
The differences between human talking and singing limit the performance of existing talking face video generation models when applied to singing.
Fire-Image-DenseNet (FIDN) for predicting wildfire burnt area using remote sensing data
Here, we develop a deep-learning-based predictive model, Fire-Image-DenseNet (FIDN), that uses spatial features derived from both near real-time and reanalysis data on the environmental and meteorological drivers of wildfire.
Foundation Cures Personalization: Recovering Facial Personalized Models' Prompt Consistency
Facial personalization represents a crucial downstream task in the domain of text-to-image generation.
EVA: An Embodied World Model for Future Video Anticipation
Yet, applying the world model for accurate video prediction is quite challenging due to the complex and dynamic intentions of the various scenes in practice.
Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation
However, when it comes to stereo audio generation, the soundscapes often have a complex scene of multiple objects and directions.
You Know What I'm Saying: Jailbreak Attack via Implicit Reference
While recent advancements in large language model (LLM) alignment have enabled the effective identification of malicious objectives involving scene nesting and keyword rewriting, our study reveals that these methods remain inadequate at detecting malicious objectives expressed through context within nested harmless objectives.
PSHuman: Photorealistic Single-view Human Reconstruction using Cross-Scale Diffusion
Detailed and photorealistic 3D human modeling is essential for various applications and has seen tremendous progress.
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
By enhancing the semantic ability of the codec, X-Codec significantly reduces WER in speech synthesis tasks and extends these benefits to non-speech applications, including music and sound generation.
AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems
However, configuring an MAS for a task remains challenging, with performance only observable post-execution.
Importance Weighting Can Help Large Language Models Self-Improve
Large language models (LLMs) have shown remarkable capability in numerous tasks and applications.
NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models
This approach allows the model to more precisely adapt to specific tasks while maintaining a compact parameter space.
STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
In this paper, we present the first structural binarization method for LLM compression to less than 1-bit precision.
Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation
Yet scant research explores the details of how these LLMs perform on advanced music understanding and conditioned generation, especially from the multi-step reasoning perspective, which is a critical aspect in the conditioned, editable, and interactive human-computer co-creation process.
MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions
To fulfill this gap, we present MMTrail, a large-scale multi-modality video-language dataset incorporating more than 20M trailer clips with visual captions, and 2M high-quality clips with multimodal captions.
Discovering symbolic expressions with parallelized tree search
Symbolic regression plays a crucial role in modern scientific research thanks to its capability of discovering concise and interpretable mathematical expressions from data.
Multi-View Large Reconstruction Model via Geometry-Aware Positional Encoding and Attention
Despite recent advancements in the Large Reconstruction Model (LRM) demonstrating impressive results, when extending its input from single image to multiple images, it exhibits inefficiencies, subpar geometric and texture quality, as well as slower convergence speed than expected.
VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling
In this work, we systematically study music generation conditioned solely on the video.
LLMs Meet Multimodal Generation and Editing: A Survey
With the recent advancement in large language models (LLMs), there is a growing interest in combining LLMs with multimodal learning.
CoCoGesture: Toward Coherent Co-speech 3D Gesture Generation in the Wild
Here, we construct the audio ControlNet through a trainable copy of our pre-trained diffusion model.
Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
For example, compared to a conventionally trained 7B model using 300B tokens, our $G_{\text{stack}}$ model converges to the same loss with 194B tokens, resulting in a 54. 6\% speedup.
Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention
Specifically, these methods assume that the input images should comply with a predefined camera type, e. g. a perspective camera with a fixed focal length, leading to distorted shapes when the assumption fails.
FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generation
By intensive experimental studies, we demonstrate that the proposed method can generate better samples than SingSong, and accelerate the generation by at least 30 times.
ComposerX: Multi-Agent Symbolic Music Composition with LLMs
Music composition represents the creative side of humanity, and itself is a complex task that requires abilities to understand and generate information with long dependency and harmony constraints.
FlashSpeech: Efficient Zero-Shot Speech Synthesis
The generation processes of FlashSpeech can be achieved efficiently with one or two sampling steps while maintaining high audio quality and high similarity to the audio prompt for zero-shot speech generation.
RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation
To this end, we propose learning to Refine Query for Retrieval Augmented Generation (RQ-RAG) in this paper, endeavoring to enhance the model by equipping it with capabilities for explicit rewriting, decomposition, and disambiguation.
ChatMusician: Understanding and Generating Music Intrinsically with LLM
It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language.
Data and Physics driven Deep Learning Models for Fast MRI Reconstruction: Fundamentals and Methodologies
Magnetic Resonance Imaging (MRI) is a pivotal clinical diagnostic tool, yet its extended scanning times often compromise patient comfort and image quality, especially in volumetric, temporal and quantitative scans.
CoMoSVC: Consistency Model-based Singing Voice Conversion
The diffusion-based Singing Voice Conversion (SVC) methods have achieved remarkable performances, producing natural audios with high similarity to the target timbre.
Freeze the backbones: A Parameter-Efficient Contrastive Approach to Robust Medical Vision-Language Pre-training
Modern healthcare often utilises radiographic images alongside textual reports for diagnostics, encouraging the use of Vision-Language Self-Supervised Learning (VL-SSL) with large pre-trained models to learn versatile medical vision representations.
A Survey of Reasoning with Foundation Models
Reasoning, a crucial ability for complex problem-solving, plays a pivotal role in various real-world settings such as negotiation, medical diagnosis, and criminal investigation.
T3D: Towards 3D Medical Image Understanding through Vision-Language Pre-training
This underlines T3D's potential in representation learning for 3D medical image analysis.
Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation
In addition, the lack of large-scale available datasets with emotional transition speech and corresponding 3D human gestures also limits the addressing of this task.
M$^{2}$Chat: Empowering VLM for Multimodal LLM Interleaved Text-Image Generation
Moreover, to further enhance the effectiveness of $M^{3}Adapter$ while preserving the coherence of semantic context comprehension, we introduce a two-stage $M^{3}FT$ fine-tuning strategy.
AI Alignment: A Comprehensive Survey
The former aims to make AI systems aligned via alignment training, while the latter aims to gain evidence about the systems' alignment and govern them appropriately to avoid exacerbating misalignment risks.
Efficient deep data assimilation with sparse observations and time-varying sensors
We introduce a novel variational DA scheme, named Voronoi-tessellation Inverse operator for VariatIonal Data assimilation (VIVID), that incorporates a DL inverse operator into the assimilation objective function.
Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Models
EndoKED automates the transformation of raw colonoscopy records into image datasets with pixel-level annotation.
zkFL: Zero-Knowledge Proof-based Gradient Aggregation for Federated Learning
Federated learning (FL) is a machine learning paradigm, which enables multiple and decentralized clients to collaboratively train a model under the orchestration of a central aggregator.
Continual Learning with Dirichlet Generative-based Rehearsal
Recent advancements in data-driven task-oriented dialogue systems (ToDs) struggle with incremental learning due to computational constraints and time-consuming issues.
Video-Instrument Synergistic Network for Referring Video Instrument Segmentation in Robotic Surgery
Our method is verified on these datasets, and experimental results exhibit that the VIS-Net can significantly outperform existing state-of-the-art referring segmentation methods.
A generative model for surrogates of spatial-temporal wildfire nowcasting
The model is tested in the ecoregion of a recent massive wildfire event in California, known as the Chimney fire.
The Potential and Pitfalls of using a Large Language Model such as ChatGPT or GPT-4 as a Clinical Assistant
Recent studies have demonstrated promising performance of ChatGPT and GPT-4 on several medical domain tasks.
On the Effectiveness of Speech Self-supervised Learning for Music
Our findings suggest that training with music data can generally improve performance on MIR tasks, even when models are trained using paradigms designed for speech.
LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT
We introduce LyricWhiz, a robust, multilingual, and zero-shot automatic lyrics transcription method achieving state-of-the-art performance on various lyrics transcription datasets, even in challenging genres such as rock and metal.
MARBLE: Music Audio Representation Benchmark for Universal Evaluation
This is evident in the limited work on deep music representations, the scarcity of large-scale datasets, and the absence of a universal and community-driven benchmark.
MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
Although SSL has been proven effective in speech and audio, its application to music audio has yet to be thoroughly explored.
Interactive Natural Language Processing
Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP, aimed at addressing limitations in existing frameworks while aligning with the ultimate goals of artificial intelligence.
NAS-FM: Neural Architecture Search for Tunable and Interpretable Sound Synthesis based on Frequency Modulation
Since expert knowledge is hard to acquire, it hinders the flexibility to quickly design and tune digital synthesizers for diverse sounds.
Long-lead forecasts of wintertime air stagnation index in southern China using oceanic memory effects
Stagnant weather condition is one of the major contributors to air pollution as it is favorable for the formation and accumulation of pollutants.
CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
In this paper, we propose a "Co"nsistency "Mo"del-based "Speech" synthesis method, CoMoSpeech, which achieve speech synthesis through a single diffusion sampling step while achieving high audio quality.
Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review
Data Assimilation (DA) and Uncertainty quantification (UQ) are extensively used in analysing and reducing error propagation in high-dimensional spatial-temporal dynamics.
Bayesian at heart: Towards autonomic outflow estimation via generative state-space modelling of heart rate dynamics
Recent research is revealing how cognitive processes are supported by a complex interplay between the brain and the rest of the body, which can be investigated by the analysis of physiological features such as breathing rhythms, heart rate, and skin conductance.
Pathway to Future Symbiotic Creativity
This report presents a comprehensive view of our vision on the development path of the human-machine symbiotic art creation.
A Dual-Masked Auto-Encoder for Robust Motion Capture with Spatial-Temporal Skeletal Token Completion
In order to demonstrate the proposed model's capability in dealing with severe data loss scenarios, we contribute a high-accuracy and challenging motion capture dataset of multi-person interactions with severe occlusion.
Suggestive Annotation of Brain MR Images with Gradient-guided Sampling
We evaluate the framework on two different brain image analysis tasks, namely brain tumour segmentation and whole brain segmentation.
Medical Scientific Table-to-Text Generation with Human-in-the-Loop under the Data Sparsity Constraint
Structured (tabular) data in the preclinical and clinical domains contains valuable information about individuals and an efficient table-to-text summarization system can drastically reduce manual efforts to condense this data into reports.
A Scalable Workflow to Build Machine Learning Classifiers with Clinician-in-the-Loop to Identify Patients in Specific Diseases
This paper proposes a scalable workflow which leverages both structured data and unstructured textual notes from EHRs with techniques including NLP, AutoML and Clinician-in-the-Loop mechanism to build machine learning classifiers to identify patients at scale with given diseases, especially those who might currently be miscoded or missed by ICD codes.
Unsupervised Numerical Reasoning to Extract Phenotypes from Clinical Text by Leveraging External Knowledge
Extracting phenotypes from clinical text has been shown to be useful for a variety of clinical use cases such as identifying patients with rare diseases.
Receding Neuron Importances for Structured Pruning
To tackle this issue, we introduce a simple BatchNorm variation with bounded scaling parameters, based on which we design a novel regularisation term that suppresses only neurons with low importance.
Label Dependent Attention Model for Disease Risk Prediction Using Multimodal Electronic Health Records
In this paper, we propose a label dependent attention model LDAM to 1) improve the interpretability by exploiting Clinical-BERT (a biomedical language model pre-trained on a large clinical corpus) to encode biomedically meaningful features and labels jointly; 2) extend the idea of joint embedding to the processing of time-series data, and develop a multi-modal learning framework for integrating heterogeneous information from medical notes and time-series health status indicators.
Label-dependent and event-guided interpretable disease risk prediction using EHRs
Thirdly, both label-dependent and event-guided representations are integrated to make a robust prediction, in which the interpretability is enabled by the attention weights over words from medical notes.
QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
In this study, we explore and evaluate a score developed during the BraTS 2019 and BraTS 2020 task on uncertainty quantification (QU-BraTS) and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation.
OmiTrans: generative adversarial networks based omics-to-omics translation framework
With the rapid development of high-throughput experimental technologies, different types of omics (e. g., genomics, epigenomics, transcriptomics, proteomics, and metabolomics) data can be produced from clinical samples.
Self-Supervised Detection of Contextual Synonyms in a Multi-Class Setting: Phenotype Annotation Use Case
Contextualised word embeddings is a powerful tool to detect contextual synonyms.
Clinical Utility of the Automatic Phenotype Annotation in Unstructured Clinical Notes: ICU Use Cases
We propose the automatic annotation of phenotypes from clinical notes as a method to capture essential information, which is complementary to typically used vital signs and laboratory test results, to predict outcomes in the Intensive Care Unit (ICU).
Joint Motion Correction and Super Resolution for Cardiac Segmentation via Latent Optimisation
In cardiac magnetic resonance (CMR) imaging, a 3D high-resolution segmentation of the heart is essential for detailed description of its anatomical structures.
XOmiVAE: an interpretable deep learning model for cancer classification using high-dimensional omics data
To the best of our knowledge, XOmiVAE is one of the first activation level-based interpretable deep learning models explaining novel clusters generated by VAE.
Adversarial autoencoders and adversarial LSTM for improved forecasts of urban air pollution simulations
Our two-step method integrates a Principal Components Analysis (PCA) based adversarial autoencoder (PC-AAE) with adversarial Long short-term memory (LSTM) networks.
MOAI: A methodology for evaluating the impact of indoor airflow in the transmission of COVID-19
Epidemiology models play a key role in understanding and responding to the COVID-19 pandemic.
Product semantics translation from brain activity via adversarial learning
To modify a design semantic of a given product from personalised brain activity via adversarial learning, in this work, we propose a deep generative transformation model to modify product semantics from the brain signal.
Verifying Design through Generative Visualization of Neural Activities
A recurrent neural network is used as the encoder to learn latent representation from electroencephalogram (EEG) signals, recorded while subjects looked at 50 categories of images.
A General Framework for Revealing Human Mind with auto-encoding GANs
For this reason, we propose an end-to-end brain decoding framework which translates brain activity into an image by latent space alignment.
Digital twins based on bidirectional LSTM and GAN for modelling the COVID-19 pandemic
Here we introduce two digital twins of a SEIRS model applied to an idealised town.
OmiEmbed: a unified multi-task deep learning framework for multi-omics data
To tackle this problem and pave the way for machine learning aided precision medicine, we proposed a unified multi-task deep learning framework named OmiEmbed to capture biomedical information from high-dimensional omics data with the deep embedding and downstream task modules.
A Blockchain-based Trust System for Decentralised Applications: When trustless needs trust
Blockchain technology has been envisaged to commence an era of decentralised applications and services (DApps) without the need for a trusted intermediary.
Cryptography and Security Distributed, Parallel, and Cluster Computing
Adversarially trained LSTMs on reduced order models of urban air pollution simulations
This adversarially trained LSTM-based approach is used on the ROM in order to produce faster forecasts of the air pollution tracer.
Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
Furthermore, in the era of the Internet of Things and big data in which data is essentially distributed, transferring a vast amount of data to a data centre for processing seems to be a cumbersome solution.
Suggestive Annotation of Brain Tumour Images with Gradient-guided Sampling
Machine learning has been widely adopted for medical image analysis in recent years given its promising performance in image segmentation and classification tasks.
Deep Generative Model-based Quality Control for Cardiac MRI Segmentation
Our approach provides a real-time and model-agnostic quality control for cardiac MRI segmentation, which has the potential to be integrated into clinical image analysis workflows.
An Epidemiological Modelling Approach for Covid19 via Data Assimilation
We compare and discuss model results which conducts updates as new observations become available.
Efficient Deep Representation Learning by Adaptive Latent Space Sampling
Supervised deep learning requires a large amount of training samples with annotations (e. g. label class for classification task, pixel- or voxel-wised label map for segmentation tasks), which are expensive and time-consuming to obtain.
Urban air pollution forecasts generated from latent space representation
Once the PCA is applied on the original model solution, a Fully-Connected AE is trained on the full-rank PCs.
Suggestive Labelling for Medical Image Analysis by Adaptive Latent Space Sampling
Supervised deep learning for medical imaging analysis requires a large amount of training samples with annotations (e. g. label class for classification task, pixel- or voxel-wised label map for medical segmentation tasks), which are expensive and time-consuming to obtain.
Regularizing Deep Multi-Task Networks using Orthogonal Gradients
Deep neural networks are a promising approach towards multi-task learning because of their capability to leverage knowledge across domains and learn general purpose representations.
Biologically inspired architectures for sample-efficient deep reinforcement learning
Deep reinforcement learning requires a heavy price in terms of sample efficiency and overparameterization in the neural networks used for function approximation.
Automatic Brain Tumour Segmentation and Biophysics-Guided Survival Prediction
Gliomas are the most common malignant brain tumourswith intrinsic heterogeneity.
Unsupervised Annotation of Phenotypic Abnormalities via Semantic Latent Representations on Electronic Health Records
The extraction of phenotype information which is naturally contained in electronic health records (EHRs) has been found to be useful in various clinical informatics applications such as disease diagnosis.
How many weights are enough : can tensor factorization learn efficient policies ?
Deep reinforcement learning requires a heavy price in terms of sample efficiency and overparameterization in the neural networks used for function approximation.
Transfer Learning from Partial Annotations for Whole Brain Segmentation
Brain MR image segmentation is a key task in neuroimaging studies.
Integrated Multi-omics Analysis Using Variational Autoencoders: Application to Pan-cancer Classification
The training procedure of OmiVAE is comprised of an unsupervised phase without the classifier and a supervised phase with the classifier.
TeraVR empowers precise reconstruction of complete 3-D neuronal morphology in the whole brain
Neuron morphology is recognized as a key determinant of cell type, yet the quantitative profiling of a mammalian neuron’s complete three-dimensional (3-D) morphology remains arduous when the neuron has complex arborization and long projection.
Self-Supervised Learning for Cardiac MR Image Segmentation by Anatomical Position Prediction
In the recent years, convolutional neural networks have transformed the field of medical image analysis due to their capacity to learn discriminative image features for a variety of classification and regression tasks.
Static Activation Function Normalization
Recent seminal work at the intersection of deep neural networks practice and random matrix theory has linked the convergence speed and robustness of these networks with the combination of random weight initialization and nonlinear activation function in use.
Integrating Semantic Knowledge to Tackle Zero-shot Text Classification
Insufficient or even unavailable training data of emerging classes is a big challenge of many classification tasks, including text classification.
Combining learning rate decay and weight decay with complexity gradient descent - Part I
The role of $L^2$ regularization, in the specific case of deep neural networks rather than more traditional machine learning models, is still not fully elucidated.
Deep Sequence Learning with Auxiliary Information for Traffic Prediction
Predicting traffic conditions from online route queries is a challenging task as there are many complicated interactions over the roads and crowds involved.
Ranked #1 on
Traffic Prediction
on Q-Traffic
Generative Creativity: Adversarial Learning for Bionic Design
Bionic design refers to an approach of generative creativity in which a target object (e. g. a floor lamp) is designed to contain features of biological source objects (e. g. flowers), resulting in creative biologically-inspired design.
Dropping Activation Outputs with Localized First-layer Deep Network for Enhancing User Privacy and Data Security
However, current architecture of deep networks suffers the privacy issue that users need to give out their data to the model (typically hosted in a server or a cluster on Cloud) for training or prediction.
TensorLayer: A Versatile Library for Efficient Deep Learning Development
Deep learning has enabled major advances in the fields of computer vision, natural language processing, and multimedia among many others.
Semantic Image Synthesis via Adversarial Learning
In this paper, we propose a way of synthesizing realistic images directly with natural language description, which has many useful applications, e. g. intelligent image manipulation.
Deep De-Aliasing for Fast Compressive Sensing MRI
Fast Magnetic Resonance Imaging (MRI) is highly in demand for many clinical applications in order to reduce the scanning cost and improve the patient experience.
Automatic Brain Tumor Detection and Segmentation Using U-Net Based Fully Convolutional Networks
In this context, a reliable fully automatic segmentation method for the brain tumor segmentation is necessary for an efficient measurement of the tumor extent.
The Deep Poincaré Map: A Novel Approach for Left Ventricle Segmentation
Our method is evaluated on two datasets, namely the Sunnybrook Cardiac Dataset (SCD) and data from the STACOM 2011 LV segmentation challenge.
I2T2I: Learning Text to Image Synthesis with Textual Data Augmentation
We demonstrate that %the capability of our method to understand the sentence descriptions, so as to I2T2I can generate better multi-categories images using MSCOCO than the state-of-the-art.
DeepSleepNet: a Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG
This demonstrated that, without changing the model architecture and the training algorithm, our model could automatically learn features for sleep stage scoring from different raw single-channel EEGs from different datasets without utilizing any hand-engineered features.
Ranked #4 on
Sleep Stage Detection
on MASS SS3
Unsupervised Image-to-Image Translation with Generative Adversarial Networks
It's useful to automatically transform an image from its original form to some synthetic form (style, partial contents, etc.
Mixed Neural Network Approach for Temporal Sleep Stage Classification
Use of this recording configuration with neural network deconvolution promises to make clinically indicated home sleep studies practical.
Automatic Sleep Stage Scoring with Single-Channel EEG Using Convolutional Neural Networks
We used convolutional neural networks (CNNs) for automatic sleep stage scoring based on single-channel electroencephalography (EEG) to learn task-specific filters for classification without using prior domain knowledge.
DropNeuron: Simplifying the Structure of Deep Neural Networks
We proposed regularisers which support a simple mechanism of dropping neurons during a network training process.
