Search Results for author:
Found 72 papers, 35 papers with code
LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale
Recently, there has been an extensive research effort in building efficient large language model (LLM) inference serving systems.
Hierarchically Structured Neural Bones for Reconstructing Animatable Objects from Casual Videos
We propose a new framework for creating and easily manipulating 3D models of arbitrary objects using casually captured videos.
Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation
In this paper, we introduce a novel Face-to-Face spoken dialogue model.
SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead
To optimize the pruning process itself, only thresholds are communicated between a server and clients instead of parameters, thereby learning how to prune.
Amortizing intractable inference in diffusion models for vision, language, and control
Diffusion models have emerged as effective distribution estimators in vision, language, and reinforcement learning, but their use as priors in downstream tasks poses an intractable posterior inference problem.
Learning diverse attacks on large language models for robust red-teaming and safety tuning
Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of large language models (LLMs).
RC-Mixup: A Data Augmentation Strategy against Noisy Data for Regression Tasks
We study the problem of robust data augmentation for regression tasks in the presence of noisy data.
Addressing Diverging Training Costs using BEVRestore for High-resolution Bird's Eye View Map Construction
Consequently, the problem poses an unavoidable bottleneck in constructing high-resolution (HR) BEV maps, as their large-sized features cause significant increases in costs including GPU memory consumption and computing latency, named diverging training costs issue.
Accounting for the Effects of Probabilistic Uncertainty During Fast Charging of Lithium-ion Batteries
Our results demonstrate that uncertainty in ambient temperature results in violations of constraints on the voltage and temperature.
Epistemology of Language Models: Do Language Models Have Holistic Knowledge?
This paper investigates the inherent knowledge in language models from the perspective of epistemological holism.
Accelerating String-Key Learned Index Structures via Memoization-based Incremental Training
Learned indexes use machine learning models to learn the mappings between keys and their corresponding positions in key-value indexes.
Ant Colony Sampling with GFlowNets for Combinatorial Optimization
This paper introduces the Generative Flow Ant Colony Sampler (GFACS), a neural-guided probabilistic search algorithm for solving combinatorial optimization (CO).
TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages
We propose a novel Tri-Modal Translation (TMT) model that translates between arbitrary modalities spanning speech, image, and text.
Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
In this paper, we propose a novel framework, namely Visual Speech Processing incorporated with LLMs (VSP-LLM), to maximize the context modeling ability by bringing the overwhelming power of LLMs.
Ranked #5 on
Lipreading
on LRS3-TED
(using extra training data)
Decoupled Sequence and Structure Generation for Realistic Antibody Design
Antibody design plays a pivotal role in advancing therapeutics.
Improved off-policy training of diffusion samplers
We study the problem of training diffusion models to sample from a distribution with a given unnormalized density or energy function.
Genetic-guided GFlowNets for Sample Efficient Molecular Optimization
The challenge of discovering new molecules with desired properties is crucial in domains like drug discovery and material design.
Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation
By using the visual speech units as the inputs of our system, we propose to pre-train a VSR model to predict corresponding text outputs on multilingual data constructed by merging several VSR databases.
Quilt: Robust Data Segment Selection against Concept Drifts
However, we contend that explicitly utilizing the drifted data together leads to much better model accuracy and propose Quilt, a data-centric framework for identifying and selecting data segments that maximize model accuracy.
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation
To mitigate the problem of the absence of a parallel AV2AV translation dataset, we propose to train our spoken language translation system with the audio-only dataset of A2A.
Learning Energy Decompositions for Partial Inference of GFlowNets
In particular, we focus on improving GFlowNet with partial inference: training flow functions with the evaluation of the intermediate states or transitions.
Learning to Scale Logits for Temperature-Conditional GFlowNets
We find that the challenge is greatly reduced if a learned function of the temperature is used to scale the policy's logits directly.
Local Search GFlowNets
Generative Flow Networks (GFlowNets) are amortized sampling methods that learn a distribution over discrete objects proportional to their rewards.
BroadBEV: Collaborative LiDAR-camera Fusion for Broad-sighted Bird's Eye View Map Construction
The method boosts the learning of depth estimation of the camera branch and induces accurate location of dense camera features in BEV space.
Analysis of the Memorization and Generalization Capabilities of AI Agents: Are Continual Learners Robust?
The generalization and memorization performance of the proposed framework are theoretically analyzed.
Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper
Different from previous methods that tried to improve the VSR performance for the target language by using knowledge learned from other languages, we explore whether we can increase the amount of training data itself for the different languages without human intervention.
Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens
To this end, we start with importing the rich knowledge related to image comprehension and language modeling from a large-scale pre-trained vision-language model into Im2Sp.
Implicit Neural Image Stitching
Existing frameworks for image stitching often provide visually reasonable stitchings.
Learning Residual Elastic Warps for Image Stitching under Dirichlet Boundary Condition
Trendy suggestions for learning-based elastic warps enable the deep image stitchings to align images exposed to large parallax errors.
DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion
We contribute a new large-scale 3D facial mesh dataset, 3D-HDTF to enable the synthesis of variations in identities, poses, and facial motions of 3D face mesh.
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge
In order to mitigate the challenge, we try to learn general speech knowledge, the ability to model lip movements, from a high-resource language through the prediction of speech units.
AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model
Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements.
Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translation
By setting both the inputs and outputs of our learning problem as speech units, we propose to train an encoder-decoder model in a many-to-many spoken language translation setting, namely Unit-to-Unit Translation (UTUT).
RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark
To fill this gap, we introduce RL4CO, a unified and extensive benchmark with in-depth library coverage of 23 state-of-the-art methods and more than 20 CO problems.
Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models
The visual speaker embedding is derived from a single target face image and enables improved mapping of input text to the learned audio latent space by incorporating the speaker characteristics inherent in the audio.
Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequences
The subsequent stage involves bootstrapping, which augments the training dataset with self-generated data labeled by a proxy score function.
Meta-SAGE: Scale Meta-Learning Scheduled Adaptation with Guided Exploration for Mitigating Scale Shift on Combinatorial Optimization
First, SML transforms the context embedding for subsequent adaptation of SAGE based on scale information.
Equity-Transformer: Solving NP-hard Min-Max Routing Problems as Sequential Generation with Equity Context
Notably, our method achieves significant reductions of runtime, approximately 335 times, and cost values of about 53\% compared to a competitive heuristic (LKH3) in the case of 100 vehicles with 1, 000 cities of mTSP.
Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization
Deep reinforcement learning (DRL) has significantly advanced the field of combinatorial optimization (CO).
Intelligible Lip-to-Speech Synthesis with Speech Units
Therefore, the proposed L2S model is trained to generate multiple targets, mel-spectrogram and speech units.
Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generation
The contextualized lip motion unit then guides the latter in synthesizing a target identity with context-aware lip motion.
Multi-Temporal Lip-Audio Memory for Visual Speech Recognition
Visual Speech Recognition (VSR) is a task to predict a sentence or word from lip movements.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
PartMix: Regularization Strategy to Learn Part Discovery for Visible-Infrared Person Re-identification
Modern data augmentation using a mixture-based technique can regularize the models from overfitting to the training data in various computer vision applications, but a proper data augmentation technique tailored for the part-based Visible-Infrared person Re-IDentification (VI-ReID) models remains unexplored.
Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring
Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models.
Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video
The proposed DVFA can align the input transcription (i. e., sentence) with the talking face video without accessing the speech audio.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Lip-to-Speech Synthesis in the Wild with Multi-task Learning
To this end, we design multi-task learning that guides the model using multimodal supervision, i. e., text and audio, to complement the insufficient word representations of acoustic feature reconstruction loss.
Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition
As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements, and this makes the VSR models show degraded performance when they are applied to unseen speakers.
SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory
It stores lip motion features from sequential ground truth images in the value memory and aligns them with corresponding audio features so that they can be retrieved using audio input at inference time.
Meta Input: How to Leverage Off-the-Shelf Deep Neural Networks
The proposed meta input can be optimized with a small number of testing data only by considering the relation between testing input data and its output prediction.
Speaker-adaptive Lip Reading with User-dependent Padding
In this paper, to remedy the performance degradation of lip reading model on unseen speakers, we propose a speaker-adaptive lip reading method, namely user-dependent padding.
Green, Quantized Federated Learning over Wireless Networks: An Energy-Efficient Design
In this paper, a green-quantized FL framework, which represents data with a finite precision level in both local training and uplink transmission, is proposed.
Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition
The enhanced audio features are fused with the visual features and taken to an encoder-decoder model composed of Conformer and Transformer for speech recognition.
CoVA: Exploiting Compressed-Domain Analysis to Accelerate Video Analytics
This paper presents CoVA, a novel cascade architecture that splits the cascade computation between compressed domain and pixel domain to address the decoding bottleneck, supporting both temporal and spatial queries.
VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection
Thus, the proposed framework brings the advantage of synthesizing the speech containing the right content even with the silent talking face video of an unseen subject.
CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation
Usually, expensive text-code paired data is essential for training a code generation model.
Ranked #130 on
Code Generation
on HumanEval
Sym-NCO: Leveraging Symmetricity for Neural Combinatorial Optimization
Deep reinforcement learning (DRL)-based combinatorial optimization (CO) methods (i. e., DRL-NCO) have shown significant merit over the conventional CO solvers as DRL-NCO is capable of learning CO solvers less relying on problem-specific expert domain knowledge (heuristic method) and supervised labeled data (supervised learning method).
DevFormer: A Symmetric Transformer for Context-Aware Device Placement
In this paper, we present DevFormer, a novel transformer-based architecture for addressing the complex and computationally demanding problem of hardware design optimization.
Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading
With the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input lip movement.
Ranked #3 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Lip to Speech Synthesis with Visual Context Attentional GAN
In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis.
Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video
By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and it provides rich information for its downstream tasks.
Ranked #4 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
Transformer Network-based Reinforcement Learning Method for Power Distribution Network (PDN) Optimization of High Bandwidth Memory (HBM)
Therefore, without additional training, the trained network can solve new decap optimization problems.
Speech Reconstruction with Reminiscent Sound via Visual Voice Memory
Our key contributions are: (1) proposing the Visual Voice memory that brings rich information of audio that complements the visual features, thus producing high-quality speech from silent video, and (2) enabling multi-speaker and unseen speaker training by memorizing auditory features and the corresponding visual features.
On the Tradeoff between Energy, Precision, and Accuracy in Federated Quantized Neural Networks
In this paper, a quantized FL framework, that represents data with a finite level of precision in both local training and uplink transmission, is proposed.
Learning Collaborative Policies to Solve NP-hard Routing Problems
Recently, deep reinforcement learning (DRL) frameworks have shown potential for solving NP-hard routing problems such as the traveling salesman problem (TSP) without problem-specific expert knowledge.
Learning Canonical 3D Object Representation for Fine-Grained Recognition
By incorporating 3D shape and appearance jointly in a deep representation, our method learns the discriminative representation of the object and achieves competitive performance on fine-grained image recognition and vehicle re-identification.
Precoding Design for Multi-user MIMO Systems with Delay-Constrained and -Tolerant Users
We consider an optimization problem that maximizes the sum spectral efficiency of delay-tolerant users while satisfying the latency constraint of delay-constrained users, and propose a generalized power iteration (GPI) precoding algorithm that finds a principal precoding vector.
Non-Terrestrial Networks for UAVs: Base Station Service Provisioning Schemes with Antenna Tilt
By considering the antenna tilt angle-based channel gain, we derive the network outage probability for both IS-BS and ES-BS schemes, and show the existence of the optimal tilt angle that minimizes the network outage probability after analyzing the conflict impact of the antenna tilt angle.
Securing Communications with Friendly Unmanned Aerial Vehicle Jammers
In this paper, we analyze the impact of a friendly unmanned aerial vehicle (UAV) jammer on UAV communications in the presence of multiple eavesdroppers.
Cross-Domain Grouping and Alignment for Domain Adaptive Semantic Segmentation
Existing techniques to adapt semantic segmentation networks across the source and target domains within deep convolutional neural networks (CNNs) deal with all the samples from the two domains in a global or category-aware manner.
Ensuring Data Freshness for Blockchain-enabled Monitoring Networks
The age of information (AoI) is a recently proposed metric for quantifying data freshness in real-time status monitoring systems where timeliness is of importance.
Age of Information Analysis in Hyperledger Fabric Blockchain-enabled Monitoring Networks
In this paper, we explore the data freshness in the Hyperledger Fabric Blockchain-enabled monitoring network (HeMN) by leveraging the AoI metric.
Cylindrical Convolutional Networks for Joint Object Detection and Viewpoint Estimation
To overcome this limitation, we introduce a learnable module, cylindrical convolutional networks (CCNs), that exploit cylindrical representation of a convolutional kernel defined in the 3D space.
