Search Results for author:
Found 32 papers, 8 papers with code
DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment
In this work, we propose a novel and personalized text-to-sound generation approach with visual alignment based on latent diffusion models, namely DiffAVA, that can simply fine-tune lightweight visual-text alignment modules with frozen modality-specific encoders to update visual-aligned text embeddings as the condition.
Mixture of personality improved Spiking actor network for efficient multi-agent cooperation
Inspired by this two-step psychology theory, we propose a biologically plausible mixture of personality (MoP) improved spiking actor network (SAN), whereby a determinantal point process is used to simulate the complex formation and integration of different types of personality in MoP, and dynamic and spiking neurons are incorporated into the SAN for the efficient reinforcement learning.
X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
(3) Integrating multiple modalities: all single-modal encoders are aligned with the LLM through X2L interfaces to integrate multimodal capabilities into the LLM.
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning
First, we learn the general concept of the input images by converting them to a textual token with a learnable image encoder.
Matching-based Term Semantics Pre-training for Spoken Patient Query Understanding
Medical Slot Filling (MSF) task aims to convert medical queries into structured information, playing an essential role in diagnosis dialogue systems.
Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation
Large-scale pre-trained language models (PLMs) with powerful language modeling capabilities have been widely used in natural language processing.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Dynamic Event-Triggered Discrete-Time Linear Time-Varying System with Privacy-Preservation
Moreover, the privacy of the system is analyzed to ensure the security of the real data.
SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Color Editing
Recently, large pretrained models (e. g., BERT, StyleGAN, CLIP) show great knowledge transfer and generalization capability on various downstream tasks within their domains.
Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem
Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols, and convert the paradigm of the speech separation/enhancement related tasks from regression to classification.
Anomaly Crossing: New Horizons for Video Anomaly Detection as Cross-domain Few-shot Learning
Video anomaly detection aims to identify abnormal events that occurred in videos.
SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing
Recently, large pretrained models (e. g., BERT, StyleGAN, CLIP) have shown great knowledge transfer and generalization capability on various downstream tasks within their domains.
Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions
The deep learning based time-domain models, e. g. Conv-TasNet, have shown great potential in both single-channel and multi-channel speech enhancement.
An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Learning to Generate Scene Graph from Natural Language Supervision
To bridge the gap between images and texts, we leverage an off-the-shelf object detector to identify and localize object instances, match labels of detected regions to concepts parsed from captions, and thus create "pseudo" labels for learning scene graph.
Learning by Planning: Language-Guided Global Image Editing
Recently, language-guided global image editing draws increasing attention with growing application potentials.
A Simple Baseline for Weakly-Supervised Scene Graph Generation
We investigate the weakly-supervised scene graph generation, which is a challenging task since no correspondence of label and object is provided.
Language-Guided Global Image Editing via Cross-Modal Cyclic Mechanism
Given the cycle, we propose several free augmentation strategies to help our model understand various editing requests given the imbalanced dataset.
Audio-visual Speech Separation with Adversarially Disentangled Visual Representation
In our model, we use the face detector to detect the number of speakers in the scene and use visual information to avoid the permutation problem.
A Benchmark and Baseline for Language-Driven Image Editing
To solve this new task, we first present a new language-driven image editing dataset that supports both local and global editing with editing operation and mask annotations.
Cubic Spline Smoothing Compensation for Irregularly Sampled Sequences
The marriage of recurrent neural networks and neural ordinary differential networks (ODE-RNN) is effective in modeling irregularly-observed sequences.
Actor-Action Video Classification CSC 249/449 Spring 2020 Challenge Report
This technical report summarizes submissions and compiles from Actor-Action video classification challenge held as a final project in CSC 249/449 Machine Vision course (Spring 2020) at University of Rochester
Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals
This model additionally has a simple and efficient stop criterion for the end of the transduction, making it able to infer the variable number of output sequences.
Ranked #2 on
Speech Separation
on WSJ0-5mix
Speaker-Conditional Chain Model for Speech Separation and Extraction
With the predicted speaker information from whole observation, our model is helpful to solve the problem of conventional speech separation and speaker extraction for multi-round long recordings.
Audio and Speech Processing Sound
Learning Continuous-Time Dynamics by Stochastic Differential Networks
Learning continuous-time stochastic dynamics is a fundamental and essential problem in modeling sporadic time series, whose observations are irregular and sparse in both time and dimension.
Neural Speaker Diarization with Speaker-Wise Chain Rule
Speaker diarization is an essential step for processing multi-speaker audio.
Not All Frames Are Equal: Weakly-Supervised Video Grounding With Contextual Similarity and Visual Clustering Losses
We invest the problem of weakly-supervised video grounding, where only video-level sentences are provided.
GAN-EM: GAN based EM learning framework
To overcome such limitation, we propose a GAN based EM learning framework that can maximize the likelihood of images and estimate the latent variables with only the constraint of L-Lipschitz continuity.
Concept Learning through Deep Reinforcement Learning with Memory-Augmented Neural Networks
In this paper, we present a memory-augmented neural network which is motivated by the process of human concept learning.
Audio-Visual Event Localization in Unconstrained Videos
In this paper, we introduce a novel problem of audio-visual event localization in unconstrained videos.
Combining Lexical and Semantic-based Features for Answer Sentence Selection
As the result of the evaluation shows, our solution provides a valuable and brief model which could be used in modelling question answering or sentence semantic relevance.
Hierarchical Memory Networks for Answer Selection on Unknown Words
Recently, end-to-end memory networks have shown promising results on Question Answering task, which encode the past facts into an explicit memory and perform reasoning ability by making multiple computational steps on the memory.
