Search Results for author:
Found 35 papers, 13 papers with code
Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control
Many real-world control problems involve conflicting objectives where we desire a dense and high-quality set of control policies that are optimal for different objective preferences (called Pareto-optimal).
Multi-Objective Reinforcement Learning
reinforcement-learning
Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
Ranked #1 on
Audio-Visual Speech Recognition
on LRS3-TED
(using extra training data)
Audio-Visual Speech Recognition
Automatic Speech Recognition
+4
SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments
Existing work has typically been tailored for particular environments or representations.
Learning Cross-lingual Visual Speech Representations
Cross-lingual self-supervised learning has been a growing research topic in the last few years.
Jointly Learning Visual and Auditory Speech Representations from Raw Data
We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained.
Ranked #1 on
Speech Recognition
on LRS2
(using extra training data)
Streaming Audio-Visual Speech Recognition with Alignment Regularization
The audio and the visual encoder neural networks are both based on the conformer architecture, which is made streamable using chunk-wise self-attention (CSA) and causal convolution.
Audio-Visual Speech Recognition
Automatic Speech Recognition
+5
Training Strategies for Improved Lip-reading
In this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators.
Ranked #1 on
Lipreading
on Lip Reading in the Wild
(using extra training data)
XInsight: eXplainable Data Analysis Through The Lens of Causality
XInsight is a three-module, end-to-end pipeline designed to extract causal graphs, translate causal primitives into XDA semantics, and quantify the quantitative contribution of each explanation to a data fact.
RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation
To train this predictor, we formulate a new loss on rendering variances using gradients from differentiable rendering.
Fast Aquatic Swimmer Optimization with Differentiable Projective Dynamics and Neural Network Hydrodynamic Models
For the deformable solid simulation of the swimmer's body, we use state-of-the-art techniques from the field of computer graphics to speed up the finite-element method (FEM).
Self-supervised Video-centralised Transformer for Video Face Clustering
We also investigate face clustering in egocentric videos, a fast-emerging field that has not been studied yet in works related to face clustering.
Ranked #1 on
Face Clustering
on EasyCom
Visual Speech Recognition for Multiple Languages in the Wild
However, these advances are usually due to the larger training sets rather than the model design.
Ranked #1 on
Lipreading
on GRID corpus (mixed-speech)
(using extra training data)
Sim2Real for Soft Robotic Fish via Differentiable Simulation
Accurate simulation of soft mechanisms under dynamic actuation is critical for the design of soft robots.
Improving Deep Metric Learning by Divide and Conquer
We propose to build a more expressive representation by jointly splitting the embedding space and the data hierarchically into smaller sub-parts.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audiovisual self-supervised learning.
End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks
In this work, we propose a new end-to-end video-to-speech model based on Generative Adversarial Networks (GANs) which translates spoken video to waveform end-to-end without using any intermediate representation or separate waveform synthesis algorithm.
DiffAqua: A Differentiable Computational Design Pipeline for Soft Underwater Swimmers with Shape Interpolation
The computational design of soft underwater swimmers is challenging because of the high degrees of freedom in soft-body modeling.
End-to-end Audio-visual Speech Recognition with Conformers
In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
Ranked #3 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+5
DiffPD: Differentiable Projective Dynamics
Inspired by Projective Dynamics (PD), we present Differentiable Projective Dynamics (DiffPD), an efficient differentiable soft-body simulator based on PD with implicit time integration.
MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing
Envisioning the general difficulty for text-to-SQL models to preserve prediction consistency against linguistic and schema variations, we propose MT-Teql, a Metamorphic Testing-based framework for systematically evaluating and augmenting the consistency of TExt-to-SQL models.
Lip-reading with Densely Connected Temporal Convolutional Networks
In this work, we present the Densely Connected Temporal Convolutional Network (DC-TCN) for lip-reading of isolated words.
Towards Practical Lipreading with Distilled and Efficient Models
However, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8. 2x and 3. 9x in terms of computational cost and number of parameters, respectively, which we hope will enable the deployment of lipreading models in practical applications.
Ranked #4 on
Lipreading
on Lip Reading in the Wild
Efficient Continuous Pareto Exploration in Multi-Task Learning
We present a novel, efficient method that generates locally continuous Pareto sets and Pareto fronts, which opens up the possibility of continuous analysis of Pareto optimal solutions in machine learning problems.
A Content Transformation Block For Image Style Transfer
Recent work has significantly improved the representation of color and texture and computational speed and image resolution.
Lipreading using Temporal Convolutional Networks
We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively.
Ranked #7 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Visually Guided Self Supervised Learning of Speech Representations
Self supervised representation learning has recently attracted a lot of research interest for both the audio and visual modalities.
Ranked #6 on
Speech Emotion Recognition
on CREMA-D
Detecting Adversarial Attacks On Audiovisual Speech Recognition
In this work, we propose an efficient and straightforward detection method based on the temporal correlation between audio and video streams.
Learning Efficient Video Representation with Video Shuffle Networks
To this end, we propose the video shuffle, a parameter-free plug-in component that efficiently reallocates the inputs of 2D convolution so that its receptive field can be extended to the temporal dimension.
Towards Pose-invariant Lip-Reading
The proposed model significantly outperforms previous approaches on non-frontal views while retaining the superior performance on frontal and near frontal mouth views.
Video-Driven Speech Reconstruction using Generative Adversarial Networks
Speech is a means of communication which relies on both audio and visual information.
Investigating the Lombard Effect Influence on End-to-End Audio-Visual Speech Recognition
Several audio-visual speech recognition models have been recently proposed which aim to improve the robustness over audio-only models in the presence of noise.
End-to-End Visual Speech Recognition for Small-Scale Datasets
In this work, we present an end-to-end visual speech recognition system based on fully-connected layers and Long-Short Memory (LSTM) networks which is suitable for small-scale datasets.
Artificial Intelligence for Prosthetics - challenge solutions
In the NeurIPS 2018 Artificial Intelligence for Prosthetics challenge, participants were tasked with building a controller for a musculoskeletal model with a goal of matching a given time-varying velocity vector.
Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture
Therefore, we could use a CTC loss in combination with an attention-based model in order to force monotonic alignments and at the same time get rid of the conditional independence assumption.
Ranked #5 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+3
End-to-end Audiovisual Speech Recognition
In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.
Ranked #17 on
Lipreading
on Lip Reading in the Wild
