Search Results for author:
Found 13 papers, 5 papers with code
Building Machine Learning Challenges for Anomaly Detection in Science
We present the different datasets along with a scheme to make machine learning challenges around the three datasets findable, accessible, interoperable, and reusable (FAIR).
Tell What You Hear From What You See -- Video to Audio Generation Through Text
Experiments show that when VATT is compared to existing video-to-audio generation methods in objective metrics, it achieves competitive performance when the audio caption is not provided.
Ranked #8 on
Video-to-Sound Generation
on VGG-Sound
CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulation
We present the results of the "Fast Calorimeter Simulation Challenge 2022" - the CaloChallenge.
From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation
Previous studies of audio-visual modalities primarily focused on either audio-visual representation learning or generative modeling of a modality conditioned on the other, creating a disconnect between these two branches.
Calo-VQ: Vector-Quantized Two-Stage Generative Model in Calorimeter Simulation
We introduce a novel machine learning method developed for the fast simulation of calorimeter detector response, adapting vector-quantized variational autoencoder (VQ-VAE).
Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering
In recent years, there has been a growing emphasis on the intersection of audio, vision, and text modalities, driving forward the advancements in multimodal research.
MuseChat: A Conversational Music Recommendation System for Videos
The reasoning module, equipped with the power of Large Language Model (Vicuna-7B) and extended to multi-modal inputs, is able to provide reasonable explanation for the recommended music.
CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments
Audio-visual navigation of an agent towards locating an audio goal is a challenging task especially when the audio is sporadic or the environment is noisy.
KGTrust: Evaluating Trustworthiness of SIoT via Knowledge Enhanced Graph Neural Networks
Graph neural networks for trust evaluation typically adopt a straightforward way such as one-hot or node2vec to comprehend node characteristics, which ignores the valuable semantic knowledge attached to nodes.
How Does it Sound?
It is often the case that the experience of watching the video can be enhanced by adding a musical soundtrack that is in-sync with the rhythmic features of these activities.
Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
We propose a novel system that takes as an input body movements of a musician playing a musical instrument and generates music in an unsupervised setting.
Audeo: Audio Generation for a Silent Performance Video
We present a novel system that gets as an input video frames of a musician playing the piano and generates the music for that video.
PREDICT & CLUSTER: Unsupervised Skeleton Based Action Recognition
Given inputs of body keypoints sequences obtained during various movements, our system associates the sequences with actions.
