Search Results for author:
Found 59 papers, 21 papers with code
Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control
Many real-world control problems involve conflicting objectives where we desire a dense and high-quality set of control policies that are optimal for different objective preferences (called Pareto-optimal).
Multi-Objective Reinforcement Learning
reinforcement-learning
ZigMa: A DiT-style Zigzag Mamba Diffusion Model
The diffusion model has long been plagued by scalability and quadratic complexity issues, especially within transformer-based structures.
DepthFM: Fast Monocular Depth Estimation with Flow Matching
Due to the generative nature of our approach, our model reliably predicts the confidence of its depth estimates.
Eliminating Information Leakage in Hard Concept Bottleneck Models with Supervised, Hierarchical Concept Learning
This paper alleviates the information leakage issue by introducing label supervision in concept predication and constructing a hierarchical concept set.
An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarios
In this paper, we present a comprehensive empirical study on the accuracy and robustness of LLMs in the context of the Chinese industrial production area.
Motion Flow Matching for Human Motion Synthesis and Editing
In this paper, we propose \emph{Motion Flow Matching}, a novel generative model designed for human motion generation featuring efficient sampling and effectiveness in motion editing applications.
Boosting Latent Diffusion with Flow Matching
We demonstrate that introducing FM between the Diffusion model and the convolutional decoder offers high-resolution image synthesis with reduced computational cost and model size.
VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
With recent advancements in Large Multimodal Models (LMMs) across various domains, a novel prompting method called visual referring prompting has emerged, showing significant potential in enhancing human-computer interaction within multimodal systems.
InstructTA: Instruction-Tuned Targeted Attack for Large Vision-Language Models
Initially, we utilize a public text-to-image generative model to "reverse" the target response into a target image, and employ GPT-4 to infer a reasonable instruction $\boldsymbol{p}^\prime$ from the target response.
TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorch
TorchAudio is an open-source audio and speech processing library built for PyTorch.
Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach
We illustrate the insights that our framework can provide by studying over 3 popular LLMs with over 12 prompt adjustment strategies.
Split and Merge: Aligning Position Biases in Large Language Model based Evaluators
Specifically, PORTIA splits the answers into multiple segments, aligns similar content across candidate answers, and then merges them back into a single prompt for evaluation by LLMs.
ASAP: Automated Sequence Planning for Complex Robotic Assembly with Physical Feasibility
The automated assembly of complex products requires a system that can automatically plan a physically feasible sequence of actions for assembling many parts together.
Enabling Runtime Verification of Causal Discovery Algorithms with Automated Conditional Independence Reasoning (Extended Version)
Based on the decision procedure to CIR, CICheck includes two variants: ED-CICheck and ED-CICheck, which detect erroneous CI tests (to enhance reliability) and prune excessive CI tests (to enhance privacy), respectively.
How Can Large Language Models Help Humans in Design and Manufacturing?
The advancement of Large Language Models (LLMs), including GPT-4, provides exciting new opportunities for generative design.
SparseVSR: Lightweight and Noise Robust Visual Speech Recognition
We evaluate our 50% sparse model on 7 different visual noise types and achieve an overall absolute improvement of more than 2% WER compared to the dense equivalent.
Towards Practical Federated Causal Structure Learning
Federated learning can solve this problem, but existing solutions for federated causal structure learning make unrealistic assumptions about data and lack convergence guarantees.
Causality-Aided Trade-off Analysis for Machine Learning Fairness
This paper uses causality analysis as a principled method for analyzing trade-offs between fairness parameters and other crucial metrics in ML pipelines.
Explain Any Concept: Segment Anything Meets Concept-Based Explanation
For computer vision tasks, mainstream pixel-based XAI methods explain DNN decisions by identifying important pixels, and emerging concept-based XAI explore forming explanations with concepts (e. g., a head in an image).
Medical records condensation: a roadmap towards healthcare data democratisation
The prevalence of artificial intelligence (AI) has envisioned an era of healthcare democratisation that promises every stakeholder a new and better way of life.
"Oops, Did I Just Say That?" Testing and Repairing Unethical Suggestions of Large Language Models with Suggest-Critique-Reflect Process
Moreover, we propose a novel on-the-fly (OTF) repairing scheme that repairs unethical suggestions made by LLMs in real-time.
Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics
Many NN approaches learn an end-to-end model that implicitly models both the governing PDE and constitutive models (or material models).
Demonstration of InsightPilot: An LLM-Empowered Automated Data Exploration System
In brief, an IQuery is an abstraction and automation of data analysis operations, which mimics the approach of data analysts and simplifies the exploration process for users.
SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision
Furthermore, when combined with large-scale pseudo-labeled audio-visual data SynthVSR yields a new state-of-the-art VSR WER of 16. 9% using publicly available data only, surpassing the recent state-of-the-art approaches trained with 29 times more non-public machine-transcribed video data (90, 000 hours).
Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
Ranked #1 on
Automatic Speech Recognition (ASR)
on LRS3-TED
Audio-Visual Speech Recognition
Automatic Speech Recognition
+4
SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments
Existing work has typically been tailored for particular environments or representations.
Learning Cross-lingual Visual Speech Representations
Cross-lingual self-supervised learning has been a growing research topic in the last few years.
Cross-Image-Attention for Conditional Embeddings in Deep Metric Learning
Experiments on established DML benchmarks show that our cross-attention conditional embedding during training improves the underlying standard DML pipeline significantly so that it outperforms the state-of-the-art.
Jointly Learning Visual and Auditory Speech Representations from Raw Data
We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained.
Ranked #1 on
Speech Recognition
on LRS2
(using extra training data)
Streaming Audio-Visual Speech Recognition with Alignment Regularization
In this work, we propose a streaming AV-ASR system based on a hybrid connectionist temporal classification (CTC)/attention neural network architecture.
Audio-Visual Speech Recognition
Automatic Speech Recognition
+5
Training Strategies for Improved Lip-reading
In this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators.
Ranked #1 on
Lipreading
on Lip Reading in the Wild
(using extra training data)
XInsight: eXplainable Data Analysis Through The Lens of Causality
XInsight is a three-module, end-to-end pipeline designed to extract causal graphs, translate causal primitives into XDA semantics, and quantify the quantitative contribution of each explanation to a data fact.
RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation
To train this predictor, we formulate a new loss on rendering variances using gradients from differentiable rendering.
Fast Aquatic Swimmer Optimization with Differentiable Projective Dynamics and Neural Network Hydrodynamic Models
For the deformable solid simulation of the swimmer's body, we use state-of-the-art techniques from the field of computer graphics to speed up the finite-element method (FEM).
Self-supervised Video-centralised Transformer for Video Face Clustering
We also investigate face clustering in egocentric videos, a fast-emerging field that has not been studied yet in works related to face clustering.
Ranked #1 on
Face Clustering
on EasyCom
Visual Speech Recognition for Multiple Languages in the Wild
However, these advances are usually due to the larger training sets rather than the model design.
Ranked #1 on
Lipreading
on GRID corpus (mixed-speech)
(using extra training data)
Sim2Real for Soft Robotic Fish via Differentiable Simulation
Accurate simulation of soft mechanisms under dynamic actuation is critical for the design of soft robots.
Improving Deep Metric Learning by Divide and Conquer
We propose to build a more expressive representation by jointly splitting the embedding space and the data hierarchically into smaller sub-parts.
LiRA: Learning Visual Speech Representations from Audio through Self-supervision
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audiovisual self-supervised learning.
End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks
In this work, we propose a new end-to-end video-to-speech model based on Generative Adversarial Networks (GANs) which translates spoken video to waveform end-to-end without using any intermediate representation or separate waveform synthesis algorithm.
DiffAqua: A Differentiable Computational Design Pipeline for Soft Underwater Swimmers with Shape Interpolation
The computational design of soft underwater swimmers is challenging because of the high degrees of freedom in soft-body modeling.
End-to-end Audio-visual Speech Recognition with Conformers
In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
Ranked #3 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+6
DiffPD: Differentiable Projective Dynamics
Inspired by Projective Dynamics (PD), we present Differentiable Projective Dynamics (DiffPD), an efficient differentiable soft-body simulator based on PD with implicit time integration.
MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing
Envisioning the general difficulty for text-to-SQL models to preserve prediction consistency against linguistic and schema variations, we propose MT-Teql, a Metamorphic Testing-based framework for systematically evaluating and augmenting the consistency of TExt-to-SQL models.
Lip-reading with Densely Connected Temporal Convolutional Networks
In this work, we present the Densely Connected Temporal Convolutional Network (DC-TCN) for lip-reading of isolated words.
Towards Practical Lipreading with Distilled and Efficient Models
However, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8. 2x and 3. 9x in terms of computational cost and number of parameters, respectively, which we hope will enable the deployment of lipreading models in practical applications.
Ranked #4 on
Lipreading
on Lip Reading in the Wild
Efficient Continuous Pareto Exploration in Multi-Task Learning
We present a novel, efficient method that generates locally continuous Pareto sets and Pareto fronts, which opens up the possibility of continuous analysis of Pareto optimal solutions in machine learning problems.
A Content Transformation Block For Image Style Transfer
Recent work has significantly improved the representation of color and texture and computational speed and image resolution.
Lipreading using Temporal Convolutional Networks
We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively.
Ranked #7 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Visually Guided Self Supervised Learning of Speech Representations
Self supervised representation learning has recently attracted a lot of research interest for both the audio and visual modalities.
Ranked #8 on
Speech Emotion Recognition
on CREMA-D
Detecting Adversarial Attacks On Audiovisual Speech Recognition
In this work, we propose an efficient and straightforward detection method based on the temporal correlation between audio and video streams.
Learning Efficient Video Representation with Video Shuffle Networks
To this end, we propose the video shuffle, a parameter-free plug-in component that efficiently reallocates the inputs of 2D convolution so that its receptive field can be extended to the temporal dimension.
Towards Pose-invariant Lip-Reading
The proposed model significantly outperforms previous approaches on non-frontal views while retaining the superior performance on frontal and near frontal mouth views.
Video-Driven Speech Reconstruction using Generative Adversarial Networks
Speech is a means of communication which relies on both audio and visual information.
Investigating the Lombard Effect Influence on End-to-End Audio-Visual Speech Recognition
Several audio-visual speech recognition models have been recently proposed which aim to improve the robustness over audio-only models in the presence of noise.
End-to-End Visual Speech Recognition for Small-Scale Datasets
In this work, we present an end-to-end visual speech recognition system based on fully-connected layers and Long-Short Memory (LSTM) networks which is suitable for small-scale datasets.
Artificial Intelligence for Prosthetics - challenge solutions
In the NeurIPS 2018 Artificial Intelligence for Prosthetics challenge, participants were tasked with building a controller for a musculoskeletal model with a goal of matching a given time-varying velocity vector.
Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture
Therefore, we could use a CTC loss in combination with an attention-based model in order to force monotonic alignments and at the same time get rid of the conditional independence assumption.
Ranked #5 on
Audio-Visual Speech Recognition
on LRS2
Audio-Visual Speech Recognition
Automatic Speech Recognition (ASR)
+3
End-to-end Audiovisual Speech Recognition
In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.
Ranked #17 on
Lipreading
on Lip Reading in the Wild
