Search Results for author:
Found 27 papers, 13 papers with code
Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark
The dataset includes high-quality and densely captured room impulse response data paired with multi-view images, and precise 6DoF pose tracking data for sound emitters and listeners in the rooms.
ScoreDec: A Phase-preserving High-Fidelity Audio Codec with A Generalized Score-based Diffusion Post-filter
Although recent mainstream waveform-domain end-to-end (E2E) neural audio codecs achieve impressive coded audio quality with a very low bitrate, the quality gap between the coded and natural audio is still significant.
From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
We present a framework for generating full-bodied photorealistic avatars that gesture according to the conversational dynamics of a dyadic interaction.
Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio
While 3D human body modeling has received much attention in computer vision, modeling the acoustic equivalent, i. e. modeling 3D spatial audio produced by body motion and speech, has fallen short in the community.
AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec
A good audio codec for live applications such as telecommunication is characterized by three key properties: (1) compression, i. e.\ the bitrate that is required to transmit the signal should be as low as possible; (2) latency, i. e.\ encoding and decoding the signal needs to be fast enough to enable communication without or with only minimal noticeable delay; and (3) reconstruction quality of the signal.
Novel-View Acoustic Synthesis
We introduce the novel-view acoustic synthesis (NVAS) task: given the sight and sound observed at a source viewpoint, can we synthesize the sound of that scene from an unseen target viewpoint?
Multiface: A Dataset for Neural Face Rendering
Along with the release of the dataset, we conduct ablation studies on the influence of different model architectures toward the model's interpolation capacity of novel viewpoint and expressions.
End-to-End Binaural Speech Synthesis
In this work, we present an end-to-end binaural speech synthesis system that combines a low-bitrate audio codec with a powerful binaural decoder that is capable of accurate speech binauralization while faithfully reconstructing environmental factors like ambient noise or reverb.
Implicit Neural Spatial Filtering for Multichannel Source Separation in the Waveform Domain
We present a single-stage casual waveform-to-waveform multichannel model that can separate moving sound sources based on their broad spatial locations in a dynamic acoustic scene.
Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis
Since facial actions such as lip movements contain significant information about speech content, it is not surprising that audio-visual speech enhancement methods are more accurate than their audio-only counterparts.
LiP-Flow: Learning Inference-time Priors for Codec Avatars via Normalizing Flows in Latent Space
To mitigate this asymmetry, we introduce a prior model that is conditioned on the runtime inputs and tie this prior space to the 3D face model via a normalizing flow in the latent space.
Conditional Diffusion Probabilistic Model for Speech Enhancement
Speech enhancement is a critical component of many user-oriented audio applications, yet current systems still suffer from distorted and unnatural outputs.
Deep Impulse Responses: Estimating and Parameterizing Filters with Deep Networks
Impulse response estimation in high noise and in-the-wild settings, with minimal control of the underlying data distributions, is a challenging problem.
MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
To improve upon existing models, we propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face.
Ranked #2 on
3D Face Animation
on VOCASET
Neural Synthesis of Binaural Audio
We present a neural rendering approach for binaural sound synthesis that can produce realistic and spatially accurate binaural sound in realtime.
Audio- and Gaze-driven Facial Animation of Codec Avatars
Codec Avatars are a recent class of learned, photorealistic face models that accurately represent the geometry and texture of a person in 3D (i. e., for virtual reality), and are almost indistinguishable from video.
On Evaluating Weakly Supervised Action Segmentation Methods
Action segmentation is the task of temporally segmenting every frame of an untrimmed video.
Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data
Action recognition is so far mainly focusing on the problem of classification of hand selected preclipped actions and reaching impressive results in this field.
A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
Action recognition has become a rapidly developing research field within the last decade.
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning
Video learning is an important task in computer vision and has experienced increasing interest over the recent years.
When will you do what? - Anticipating Temporal Occurrences of Activities
Analyzing human actions in videos has gained increased attention recently.
Recurrent Residual Learning for Action Recognition
In this work, we propose a novel recurrent ConvNet architecture called recurrent residual networks to address the task of action recognition.
Action Sets: Weakly Supervised Action Segmentation without Ordering Constraints
Action detection and temporal segmentation of actions in videos are topics of increasing interest.
Weakly Supervised Action Learning with RNN based Fine-to-coarse Modeling
We present an approach for weakly supervised learning of human actions.
A Bag-of-Words Equivalent Recurrent Neural Network for Action Recognition
In this work, we propose a recurrent neural network that is equivalent to the traditional bag-of-words approach but enables for the application of discriminative training.
Weakly supervised learning of actions from transcripts
Our system is based on the idea that, given a sequence of input data and a transcript, i. e. a list of the order the actions occur in the video, it is possible to infer the actions within the video stream, and thus, learn the related action models without the need for any frame-based annotation.
Temporal Action Detection Using a Statistical Language Model
While current approaches to action recognition on pre-segmented video clips already achieve high accuracies, temporal action detection is still far from comparably good results.
