Search Results for author:
Found 37 papers, 11 papers with code
X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages
(3) Integrating multiple modalities: all single-modal encoders are aligned with the LLM through X2L interfaces to integrate multimodal capabilities into the LLM.
Neural Speaker Diarization with Speaker-Wise Chain Rule
Speaker diarization is an essential step for processing multi-speaker audio.
Audio-Visual Event Localization in Unconstrained Videos
In this paper, we introduce a novel problem of audio-visual event localization in unconstrained videos.
Learning to Generate Scene Graph from Natural Language Supervision
To bridge the gap between images and texts, we leverage an off-the-shelf object detector to identify and localize object instances, match labels of detected regions to concepts parsed from captions, and thus create "pseudo" labels for learning scene graph.
Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation
Large-scale pre-trained language models (PLMs) have shown great potential in natural language processing tasks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Hierarchical Memory Networks for Answer Selection on Unknown Words
Recently, end-to-end memory networks have shown promising results on Question Answering task, which encode the past facts into an explicit memory and perform reasoning ability by making multiple computational steps on the memory.
Learning by Planning: Language-Guided Global Image Editing
Recently, language-guided global image editing draws increasing attention with growing application potentials.
Matching-based Term Semantics Pre-training for Spoken Patient Query Understanding
Medical Slot Filling (MSF) task aims to convert medical queries into structured information, playing an essential role in diagnosis dialogue systems.
Actor-Action Video Classification CSC 249/449 Spring 2020 Challenge Report
This technical report summarizes submissions and compiles from Actor-Action video classification challenge held as a final project in CSC 249/449 Machine Vision course (Spring 2020) at University of Rochester
A Knowledge-enhanced Two-stage Generative Framework for Medical Dialogue Information Extraction
However, these generative methods output a whole sequence consisting of term-status pairs in one stage and ignore integrating prior knowledge, which demands a deeper understanding to model the relationship between terms and infer the status of each term.
Concept Learning through Deep Reinforcement Learning with Memory-Augmented Neural Networks
In this paper, we present a memory-augmented neural network which is motivated by the process of human concept learning.
GAN-EM: GAN based EM learning framework
To overcome such limitation, we propose a GAN based EM learning framework that can maximize the likelihood of images and estimate the latent variables with only the constraint of L-Lipschitz continuity.
Combining Lexical and Semantic-based Features for Answer Sentence Selection
As the result of the evaluation shows, our solution provides a valuable and brief model which could be used in modelling question answering or sentence semantic relevance.
Not All Frames Are Equal: Weakly-Supervised Video Grounding With Contextual Similarity and Visual Clustering Losses
We invest the problem of weakly-supervised video grounding, where only video-level sentences are provided.
Learning Continuous-Time Dynamics by Stochastic Differential Networks
Learning continuous-time stochastic dynamics is a fundamental and essential problem in modeling sporadic time series, whose observations are irregular and sparse in both time and dimension.
Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals
This model additionally has a simple and efficient stop criterion for the end of the transduction, making it able to infer the variable number of output sequences.
Ranked #3 on
Speech Separation
on WSJ0-4mix
Speaker-Conditional Chain Model for Speech Separation and Extraction
With the predicted speaker information from whole observation, our model is helpful to solve the problem of conventional speech separation and speaker extraction for multi-round long recordings.
Audio and Speech Processing Sound
Cubic Spline Smoothing Compensation for Irregularly Sampled Sequences
The marriage of recurrent neural networks and neural ordinary differential networks (ODE-RNN) is effective in modeling irregularly-observed sequences.
A Benchmark and Baseline for Language-Driven Image Editing
To solve this new task, we first present a new language-driven image editing dataset that supports both local and global editing with editing operation and mask annotations.
Audio-visual Speech Separation with Adversarially Disentangled Visual Representation
In our model, we use the face detector to detect the number of speakers in the scene and use visual information to avoid the permutation problem.
A Simple Baseline for Weakly-Supervised Scene Graph Generation
We investigate the weakly-supervised scene graph generation, which is a challenging task since no correspondence of label and object is provided.
Language-Guided Global Image Editing via Cross-Modal Cyclic Mechanism
Given the cycle, we propose several free augmentation strategies to help our model understand various editing requests given the imbalanced dataset.
An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions
The deep learning based time-domain models, e. g. Conv-TasNet, have shown great potential in both single-channel and multi-channel speech enhancement.
SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing
Recently, large pretrained models (e. g., BERT, StyleGAN, CLIP) have shown great knowledge transfer and generalization capability on various downstream tasks within their domains.
Anomaly Crossing: New Horizons for Video Anomaly Detection as Cross-domain Few-shot Learning
Video anomaly detection aims to identify abnormal events that occurred in videos.
Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem
Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols, and convert the paradigm of the speech separation/enhancement related tasks from regression to classification.
SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Color Editing
Recently, large pretrained models (e. g., BERT, StyleGAN, CLIP) show great knowledge transfer and generalization capability on various downstream tasks within their domains.
Dynamic Event-Triggered Discrete-Time Linear Time-Varying System with Privacy-Preservation
Moreover, the privacy of the system is analyzed to ensure the security of the real data.
InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning
First, we learn the general concept of the input images by converting them to a textual token with a learnable image encoder.
Diffusion Personalization Tuning Free
Text-to-Image Generation
Mixture of personality improved Spiking actor network for efficient multi-agent cooperation
Inspired by this two-step psychology theory, we propose a biologically plausible mixture of personality (MoP) improved spiking actor network (SAN), whereby a determinantal point process is used to simulate the complex formation and integration of different types of personality in MoP, and dynamic and spiking neurons are incorporated into the SAN for the efficient reinforcement learning.
DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment
In this work, we propose a novel and personalized text-to-sound generation approach with visual alignment based on latent diffusion models, namely DiffAVA, that can simply fine-tune lightweight visual-text alignment modules with frozen modality-specific encoders to update visual-aligned text embeddings as the condition.
VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition
In this paper, we first propose ViLaS (Vision and Language into Automatic Speech Recognition), a novel multimodal ASR model based on the continuous integrate-and-fire (CIF) mechanism, which can integrate visual and textual context simultaneously or separately, to facilitate speech recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models
With the emergence of text-to-video (T2V) diffusion models, its temporal counterpart, motion customization, has not yet been well investigated.
VIXEN: Visual Text Comparison Network for Image Difference Captioning
We present VIXEN - a technique that succinctly summarizes in text the visual differences between a pair of images in order to highlight any content manipulation present.
Text-to-Audio Generation Synchronized with Videos
Extensive evaluations on the AudioCaps and T2AV-Bench demonstrate that our T2AV sets a new standard for video-aligned TTA generation in ensuring visual alignment and temporal consistency.
