Search Results for author:
Found 63 papers, 29 papers with code
Epistemology of Language Models: Do Language Models Have Holistic Knowledge?
This paper investigates the inherent knowledge in language models from the perspective of epistemological holism.
Accelerating String-Key Learned Index Structures via Memoization-based Incremental Training
Learned indexes use machine learning models to learn the mappings between keys and their corresponding positions in key-value indexes.
Ant Colony Sampling with GFlowNets for Combinatorial Optimization
This paper introduces the Generative Flow Ant Colony Sampler (GFACS), a novel neural-guided meta-heuristic algorithm for combinatorial optimization.
TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages
We propose a novel Tri-Modal Translation (TMT) model that translates between arbitrary modalities spanning speech, image, and text.
Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
In visual speech processing, context modeling capability is one of the most important requirements due to the ambiguous nature of lip movements.
Ranked #4 on
Lipreading
on LRS3-TED
(using extra training data)
Anfinsen Goes Neural: a Graphical Model for Conditional Antibody Design
Antibody design plays a pivotal role in advancing therapeutics.
On diffusion models for amortized inference: Benchmarking and improving stochastic control and sampling
We study the problem of training diffusion models to sample from a distribution with a given unnormalized density or energy function.
Genetic-guided GFlowNets: Advancing in Practical Molecular Optimization Benchmark
This paper proposes a novel variant of GFlowNet, genetic-guided GFlowNet (Genetic GFN), which integrates an iterative genetic search into GFlowNet.
Multilingual Visual Speech Recognition with a Single Model by Learning with Discrete Visual Speech Units
By using the visual speech units as the inputs of our system, we pre-train the model to predict corresponding text outputs on massive multilingual data constructed by merging several VSR databases.
Quilt: Robust Data Segment Selection against Concept Drifts
However, we contend that explicitly utilizing the drifted data together leads to much better model accuracy and propose Quilt, a data-centric framework for identifying and selecting data segments that maximize model accuracy.
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation
To mitigate the problem of the absence of a parallel AV2AV translation dataset, we propose to train our spoken language translation system with the audio-only dataset of A2A.
Learning Energy Decompositions for Partial Inference of GFlowNets
In particular, we focus on improving GFlowNet with partial inference: training flow functions with the evaluation of the intermediate states or transitions.
Local Search GFlowNets
Generative Flow Networks (GFlowNets) are amortized sampling methods that learn a distribution over discrete objects proportional to their rewards.
Learning to Scale Logits for Temperature-Conditional GFlowNets
We find that the challenge is greatly reduced if a learned function of the temperature is used to scale the policy's logits directly.
BroadBEV: Collaborative LiDAR-camera Fusion for Broad-sighted Bird's Eye View Map Construction
The method boosts the learning of depth estimation of the camera branch and induces accurate location of dense camera features in BEV space.
Analysis of the Memorization and Generalization Capabilities of AI Agents: Are Continual Learners Robust?
The generalization and memorization performance of the proposed framework are theoretically analyzed.
Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens
To this end, we start with importing the rich knowledge related to image comprehension and language modeling from a large-scale pre-trained vision-language model into Im2Sp.
Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper
Different from previous methods that tried to improve the VSR performance for the target language by using knowledge learned from other languages, we explore whether we can increase the amount of training data itself for the different languages without human intervention.
Learning Residual Elastic Warps for Image Stitching under Dirichlet Boundary Condition
Trendy suggestions for learning-based elastic warps enable the deep image stitchings to align images exposed to large parallax errors.
Implicit Neural Image Stitching
Existing frameworks for image stitching often provide visually reasonable stitchings.
DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion
We contribute a new large-scale 3D facial mesh dataset, 3D-HDTF to enable the synthesis of variations in identities, poses, and facial motions of 3D face mesh.
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge
In order to mitigate the challenge, we try to learn general speech knowledge, the ability to model lip movements, from a high-resource language through the prediction of speech units.
AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model
Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements.
Many-to-Many Spoken Language Translation via Unified Speech and Text Representation Learning with Unit-to-Unit Translation
A single pre-trained model with UTUT can be employed for diverse multilingual speech- and text-related tasks, such as Speech-to-Speech Translation (STS), multilingual Text-to-Speech Synthesis (TTS), and Text-to-Speech Translation (TTST).
RL4CO: a Unified Reinforcement Learning for Combinatorial Optimization Library
To address these challenges, we introduce RL4CO, a unified Reinforcement Learning (RL) for Combinatorial Optimization (CO) library.
Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models
The visual speaker embedding is derived from a single target face image and enables improved mapping of input text to the learned audio latent space by incorporating the speaker characteristics inherent in the audio.
Meta-SAGE: Scale Meta-Learning Scheduled Adaptation with Guided Exploration for Mitigating Scale Shift on Combinatorial Optimization
First, SML transforms the context embedding for subsequent adaptation of SAGE based on scale information.
Equity-Transformer: Solving NP-hard Min-Max Routing Problems as Sequential Generation with Equity Context
Notably, our method achieves significant reductions of runtime, approximately 335 times, and cost values of about 53\% compared to a competitive heuristic (LKH3) in the case of 100 vehicles with 1, 000 cities of mTSP.
Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences
The subsequent stage involves bootstrapping, which augments the training dataset with self-generated data labeled by a proxy score function.
Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization
Deep reinforcement learning (DRL) has significantly advanced the field of combinatorial optimization (CO).
Intelligible Lip-to-Speech Synthesis with Speech Units
Therefore, the proposed L2S model is trained to generate multiple targets, mel-spectrogram and speech units.
Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generation
The contextualized lip motion unit then guides the latter in synthesizing a target identity with context-aware lip motion.
Multi-Temporal Lip-Audio Memory for Visual Speech Recognition
Visual Speech Recognition (VSR) is a task to predict a sentence or word from lip movements.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
PartMix: Regularization Strategy to Learn Part Discovery for Visible-Infrared Person Re-identification
Modern data augmentation using a mixture-based technique can regularize the models from overfitting to the training data in various computer vision applications, but a proper data augmentation technique tailored for the part-based Visible-Infrared person Re-IDentification (VI-ReID) models remains unexplored.
Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring
Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models.
Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video
The proposed DVFA can align the input transcription (i. e., sentence) with the talking face video without accessing the speech audio.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Lip-to-Speech Synthesis in the Wild with Multi-task Learning
To this end, we design multi-task learning that guides the model using multimodal supervision, i. e., text and audio, to complement the insufficient word representations of acoustic feature reconstruction loss.
Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition
As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements, and this makes the VSR models show degraded performance when they are applied to unseen speakers.
SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory
It stores lip motion features from sequential ground truth images in the value memory and aligns them with corresponding audio features so that they can be retrieved using audio input at inference time.
Meta Input: How to Leverage Off-the-Shelf Deep Neural Networks
The proposed meta input can be optimized with a small number of testing data only by considering the relation between testing input data and its output prediction.
Speaker-adaptive Lip Reading with User-dependent Padding
In this paper, to remedy the performance degradation of lip reading model on unseen speakers, we propose a speaker-adaptive lip reading method, namely user-dependent padding.
Green, Quantized Federated Learning over Wireless Networks: An Energy-Efficient Design
In this paper, a green-quantized FL framework, which represents data with a finite precision level in both local training and uplink transmission, is proposed.
Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition
The enhanced audio features are fused with the visual features and taken to an encoder-decoder model composed of Conformer and Transformer for speech recognition.
CoVA: Exploiting Compressed-Domain Analysis to Accelerate Video Analytics
This paper presents CoVA, a novel cascade architecture that splits the cascade computation between compressed domain and pixel domain to address the decoding bottleneck, supporting both temporal and spatial queries.
VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection
Thus, the proposed framework brings the advantage of synthesizing the speech containing the right content even with the silent talking face video of an unseen subject.
CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation
Usually, expensive text-code paired data is essential for training a code generation model.
Ranked #122 on
Code Generation
on HumanEval
DevFormer: A Symmetric Transformer for Context-Aware Device Placement
In this paper, we present DevFormer, a novel transformer-based architecture for addressing the complex and computationally demanding problem of hardware design optimization.
Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization
Deep reinforcement learning (DRL)-based combinatorial optimization (CO) methods (i. e., DRL-NCO) have shown significant merit over the conventional CO solvers as DRL-NCO is capable of learning CO solvers less relying on problem-specific expert domain knowledge (heuristic method) and supervised labeled data (supervised learning method).
Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading
With the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input lip movement.
Ranked #2 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Lip to Speech Synthesis with Visual Context Attentional GAN
In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis.
Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video
By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and it provides rich information for its downstream tasks.
Ranked #3 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Transformer Network-based Reinforcement Learning Method for Power Distribution Network (PDN) Optimization of High Bandwidth Memory (HBM)
Therefore, without additional training, the trained network can solve new decap optimization problems.
Speech Reconstruction with Reminiscent Sound via Visual Voice Memory
Our key contributions are: (1) proposing the Visual Voice memory that brings rich information of audio that complements the visual features, thus producing high-quality speech from silent video, and (2) enabling multi-speaker and unseen speaker training by memorizing auditory features and the corresponding visual features.
On the Tradeoff between Energy, Precision, and Accuracy in Federated Quantized Neural Networks
In this paper, a quantized FL framework, that represents data with a finite level of precision in both local training and uplink transmission, is proposed.
Learning Collaborative Policies to Solve NP-hard Routing Problems
Recently, deep reinforcement learning (DRL) frameworks have shown potential for solving NP-hard routing problems such as the traveling salesman problem (TSP) without problem-specific expert knowledge.
Learning Canonical 3D Object Representation for Fine-Grained Recognition
By incorporating 3D shape and appearance jointly in a deep representation, our method learns the discriminative representation of the object and achieves competitive performance on fine-grained image recognition and vehicle re-identification.
Precoding Design for Multi-user MIMO Systems with Delay-Constrained and -Tolerant Users
We consider an optimization problem that maximizes the sum spectral efficiency of delay-tolerant users while satisfying the latency constraint of delay-constrained users, and propose a generalized power iteration (GPI) precoding algorithm that finds a principal precoding vector.
Non-Terrestrial Networks for UAVs: Base Station Service Provisioning Schemes with Antenna Tilt
By considering the antenna tilt angle-based channel gain, we derive the network outage probability for both IS-BS and ES-BS schemes, and show the existence of the optimal tilt angle that minimizes the network outage probability after analyzing the conflict impact of the antenna tilt angle.
Securing Communications with Friendly Unmanned Aerial Vehicle Jammers
In this paper, we analyze the impact of a friendly unmanned aerial vehicle (UAV) jammer on UAV communications in the presence of multiple eavesdroppers.
Cross-Domain Grouping and Alignment for Domain Adaptive Semantic Segmentation
Existing techniques to adapt semantic segmentation networks across the source and target domains within deep convolutional neural networks (CNNs) deal with all the samples from the two domains in a global or category-aware manner.
Ensuring Data Freshness for Blockchain-enabled Monitoring Networks
The age of information (AoI) is a recently proposed metric for quantifying data freshness in real-time status monitoring systems where timeliness is of importance.
Age of Information Analysis in Hyperledger Fabric Blockchain-enabled Monitoring Networks
In this paper, we explore the data freshness in the Hyperledger Fabric Blockchain-enabled monitoring network (HeMN) by leveraging the AoI metric.
Cylindrical Convolutional Networks for Joint Object Detection and Viewpoint Estimation
To overcome this limitation, we introduce a learnable module, cylindrical convolutional networks (CCNs), that exploit cylindrical representation of a convolutional kernel defined in the 3D space.
