Search Results for author:
Found 105 papers, 50 papers with code
Machine Translation With Weakly Paired Bilingual Documents
Neural machine translation, which achieves near human-level performance in some languages, strongly relies on the availability of large amounts of parallel sentences, which hinders its applicability to low-resource language pairs.
Finding the Dominant Winning Ticket in Pre-Trained Language Models
Empirically, we show that (a) the dominant winning ticket can achieve performance that is comparable with that of the full-parameter model, (b) the dominant winning ticket is transferable across different tasks, (c) and the dominant winning ticket has a natural structure within each parameter matrix.
Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder
Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space.
Knowledge-driven AI-generated data for accurate and interpretable breast ultrasound diagnoses
In the prospective external evaluation, our diagnostic model outperforms the average performance of nine radiologists by 33. 5% in specificity with the same sensitivity, improving their performance by providing predictions with an interpretable decision-making process.
Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks
Despite the efforts of some online training methods, tackling spatial credit assignments by alternatives with comparable performance as spatial BP remains a significant problem.
Let the Code LLM Edit Itself When You Edit the Code
Simply encoding the edited subsequence and integrating it to the original KV cache meets the temporal confusion problem, leading to significantly worse performance.
GeoMFormer: A General Architecture for Geometric Molecular Representation Learning
We argue that there is a strong need for a general and flexible framework for learning both invariant and equivariant features.
Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning
Spiking neural networks (SNNs) are investigated as biologically inspired models of neural computation, distinguished by their computational capability and energy efficiency due to precise spiking times and sparse spikes with event-driven computation.
DPO Meets PPO: Reinforced Token Optimization for RLHF
For its practical implementation, \texttt{RTO} innovatively integrates Direct Preference Optimization (DPO) and PPO.
Boosting Meta-Training with Base Class Information for Few-Shot Learning
However, as a non-end-to-end training method, indicating the meta-training stage can only begin after the completion of pre-training, Meta-Baseline suffers from higher training cost and suboptimal performance due to the inherent conflicts of the two training stages.
Do Efficient Transformers Really Save Computation?
Our results show that while these models are expressive enough to solve general DP tasks, contrary to expectations, they require a model size that scales with the problem size.
Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks
Neuromorphic computing with spiking neural networks is promising for energy-efficient artificial intelligence (AI) applications.
DOF: Accelerating High-order Differential Operators with Forward Propagation
Solving partial differential equations (PDEs) efficiently is essential for analyzing complex physical systems.
Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE).
Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion
We propose an approach for continuous prediction of turn-taking and backchanneling locations in spoken dialogue by fusing a neural acoustic model with a large language model (LLM).
Two-pass Endpoint Detection for Speech Recognition
Endpoint (EP) detection is a key component of far-field speech recognition systems that assist the user through voice commands.
Beyond Weisfeiler-Lehman: A Quantitative Framework for GNN Expressiveness
Specifically, we identify a fundamental expressivity measure termed homomorphism expressivity, which quantifies the ability of GNN models to count graphs under homomorphism.
End-to-End Crystal Structure Prediction from Powder X-Ray Diffraction
XtalNet represents a significant advance in CSP, enabling the prediction of complex structures from PXRD data without the need for external databases or manual intervention.
REST: Retrieval-Based Speculative Decoding
We introduce Retrieval-Based Speculative Decoding (REST), a novel algorithm designed to speed up language model generation.
CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexity
With distributed machine learning being a prominent technique for large-scale machine learning tasks, communication complexity has become a major bottleneck for speeding up training and scaling up machine numbers.
Forward Laplacian: A New Computational Framework for Neural Network-based Variational Monte Carlo
Neural network-based variational Monte Carlo (NN-VMC) has emerged as a promising cutting-edge technique of ab initio quantum chemistry.
Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective
By using circuit complexity theory, we first give impossibility results showing that bounded-depth Transformers are unable to directly produce correct answers for basic arithmetic/equation tasks unless the model size grows super-polynomially with respect to the input length.
Personalized Predictive ASR for Latency Reduction in Voice Assistants
Streaming Automatic Speech Recognition (ASR) in voice assistants can utilize prefetching to partially hide the latency of response generation.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Adaptive Endpointing with Deep Contextual Multi-armed Bandits
In this paper, we aim to provide a solution for adaptive endpointing by proposing an efficient method for choosing an optimal endpointing configuration given utterance-level audio features in an online setting, while avoiding hyperparameter grid-search.
Highly Accurate Quantum Chemical Property Prediction with Uni-Mol+
Uni-Mol+ first generates a raw 3D molecule conformation from inexpensive methods such as RDKit.
A Complete Expressiveness Hierarchy for Subgraph GNNs via Subgraph Weisfeiler-Lehman Tests
Recently, subgraph GNNs have emerged as an important direction for developing expressive graph neural networks (GNNs).
Ranked #1 on
Subgraph Counting - C6
on Synthetic Graph
3D Molecular Generation via Virtual Dynamics
Extensive experiment results on pocket-based molecular generation demonstrate that VD-Gen can generate novel 3D molecules to fill the target pocket cavity with high binding affinities, significantly outperforming previous baselines.
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs).
Rethinking the Expressive Power of GNNs via Graph Biconnectivity
In this paper, we take a fundamentally different perspective to study the expressive power of GNNs beyond the WL test.
DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets
However, due to the sparse characteristics of point clouds, it is non-trivial to apply a standard transformer on sparse points.
Ranked #1 on
3D Object Detection
on waymo cyclist
Matching entropy based disparity estimation from light field
A major challenge for matching-based depth estimation is to prevent mismatches in occlusion and smooth regions.
Denoising Masked AutoEncoders Help Robust Classification
In this paper, we propose a new self-supervised method, which is called Denoising Masked AutoEncoders (DMAE), for learning certified robust classifiers of images.
Online Training Through Time for Spiking Neural Networks
With OTTT, it is the first time that two mainstream supervised SNN training methods, BPTT with SG and spike representation-based training, are connected, and meanwhile in a biologically plausible form.
Ranked #3 on
Event data classification
on CIFAR10-DVS
One Transformer Can Understand Both 2D & 3D Molecular Data
To achieve this goal, in this work, we develop a novel Transformer-based Molecular model called Transformer-M, which can take molecular data of 2D or 3D formats as input and generate meaningful semantic representations.
Ranked #4 on
Graph Regression
on PCQM4Mv2-LSC
Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective
Designing neural networks with bounded Lipschitz constant is a promising way to obtain certifiably robust classifiers against adversarial examples.
Adversarial Noises Are Linearly Separable for (Nearly) Random Neural Networks
Adversarial examples, which are usually generated for specific inputs with a specific model, are ubiquitous for neural networks.
Is $L^2$ Physics-Informed Loss Always Suitable for Training Physics-Informed Neural Network?
In particular, we leverage the concept of stability in the literature of partial differential equation to study the asymptotic behavior of the learned solution as the loss approaches zero.
Your Transformer May Not be as Powerful as You Expect
Extensive experiments covering typical architectures and tasks demonstrate that our model is parameter-efficient and can achieve superior performance to strong baselines in a wide range of applications.
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model.
Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation.
An Empirical Study of Graphormer on Large-Scale Molecular Modeling Datasets
This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation.
VADOI:Voice-Activity-Detection Overlapping Inference For End-to-end Long-form Speech Recognition
Results show that the proposed method can achieve a 20% relative computation cost reduction on Librispeech and Microsoft Speech Language Translation long-form corpus while maintaining the WER performance when comparing to the best performing overlapping inference algorithm.
Learning Physics-Informed Neural Networks without Stacked Back-propagation
In this work, we develop a novel approach that can significantly accelerate the training of Physics-Informed Neural Networks.
HousE: Knowledge Graph Embedding with Householder Parameterization
The effectiveness of knowledge graph embedding (KGE) largely depends on the ability to model intrinsic relation patterns and mapping properties.
Do Transformers Really Perform Badly for Graph Representation?
Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
Exploration of Dark Chemical Genomics Space via Portal Learning: Applied to Targeting the Undruggable Genome and COVID-19 Anti-Infective Polypharmacology
Advances in biomedicine are largely fueled by exploring uncharted territories of human biology.
Can Vision Transformers Perform Convolution?
Several recent studies have demonstrated that attention-based networks, such as Vision Transformer (ViT), can outperform Convolutional Neural Networks (CNNs) on several computer vision tasks without using convolutional layers.
Boosting the Certified Robustness of L-infinity Distance Nets
Recently, Zhang et al. (2021) developed a new neural network architecture based on $\ell_\infty$-distance functions, which naturally possesses certified $\ell_\infty$ robustness by its construction.
Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
Since in many state-of-the-art models, relative positional encoding is used as default, designing efficient Transformers that can incorporate RPE is appealing.
First Place Solution of KDD Cup 2021 & OGB Large-Scale Challenge Graph Prediction Track
In this technical report, we present our solution of KDD Cup 2021 OGB Large-Scale Challenge - PCQM4M-LSC Track.
Do Transformers Really Perform Bad for Graph Representation?
Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model.
Ranked #1 on
Graph Regression
on PCQM4M-LSC
Two Coupled Rejection Metrics Can Tell Adversarial Examples Apart
Along with this routine, we find that confidence and a rectified confidence (R-Con) can form two coupled rejection metrics, which could provably distinguish wrongly classified inputs from correctly classified ones.
How could Neural Networks understand Programs?
Inspired by this, we propose a novel program semantics learning paradigm, that the model should learn from information composed of (1) the representations which align well with the fundamental operations in operational semantics, and (2) the information of environment transition, which is indispensable for program understanding.
Wav2vec-C: A Self-supervised Model for Speech Representation Learning
However, the quantization process is regularized by an additional consistency network that learns to reconstruct the input features to the wav2vec 2. 0 network from the quantized representations in a way similar to a VQ-VAE model.
Transformers with Competitive Ensembles of Independent Mechanisms
In this work we explore a way in which the Transformer architecture is deficient: it represents each position with a large monolithic hidden representation and a single set of parameters which are applied over the entire hidden representation.
LazyFormer: Self Attention with Lazy Update
In each lazy block, the self-attention distribution is only computed once in the first layer and then is reused in all upper layers.
Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder
Dense retrieval requires high-quality text sequence embeddings to support effective search in the representation space.
Revisiting Language Encoding in Learning Multilingual Representations
The language embedding can be either added to the word embedding or attached at the beginning of the sentence.
Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons
This directly provides a rigorous guarantee of the certified robustness based on the margin of prediction outputs.
CODE-AE: A Coherent De-confounding Autoencoder for Predicting Patient-Specific Drug Response From Cell Line Transcriptomics
Thus, CODE-AE provides a useful framework to take advantage of in vitro omics data for developing generalized patient predictive models.
Taking Notes on the Fly Helps Language Pre-Training
In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization.
Pretrain-to-Finetune Adversarial Training via Sample-wise Randomized Smoothing
For certification, we carefully allocate specific robust regions for each test sample.
A Cross-Level Information Transmission Network for Predicting Phenotype from New Genotype: Application to Cancer Precision Medicine
An unsolved fundamental problem in biology and ecology is to predict observable traits (phenotypes) from a new genetic constitution (genotype) of an organism under environmental perturbations (e. g., drug treatment).
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
We provide an explanation by showing that InstanceNorm serves as a preconditioner for GNNs, but such preconditioning effect is weaker with BatchNorm due to the heavy batch noise in graph datasets.
Ranked #25 on
Graph Property Prediction
on ogbg-molhiv
Taking Notes on the Fly Helps BERT Pre-training
In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization.
Transferred Discrepancy: Quantifying the Difference Between Representations
Our experiments show that TD can provide fine-grained information for varied downstream tasks, and for the models trained from different initializations, the learned features are not the same in terms of downstream-task predictions.
Rethinking Positional Encoding in Language Pre-training
In this work, we investigate the positional encoding methods used in language pre-training (e. g., BERT) and identify several problems in the existing formulations.
MC-BERT: Efficient Language Pre-Training via a Meta Controller
Pre-trained contextual representations (e. g., BERT) have become the foundation to achieve state-of-the-art results on many NLP tasks.
Invertible Image Rescaling
High-resolution digital images are usually downscaled to fit various display screens or save the cost of storage and bandwidth, meanwhile the post-upscaling is adpoted to recover the original resolutions or the details in the zoom-in images.
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View.
In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method.
Incorporating BERT into Neural Machine Translation
While BERT is more commonly used as fine-tuning instead of contextual embedding for downstream language understanding tasks, in NMT, our preliminary exploration of using BERT as contextual embedding is better than using for fine-tuning.
On Layer Normalization in the Transformer Architecture
This motivates us to remove the warm-up stage for the training of Pre-LN Transformers.
MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly.
Defective Convolutional Networks
Robustness of convolutional neural networks (CNNs) has gained in importance on account of adversarial examples, i. e., inputs added as well-designed perturbations that are imperceptible to humans but can cause the model to predict incorrectly.
Machine Translation With Weakly Paired Documents
1) We provide a simple approach to mine implicitly bilingual sentence pairs from document pairs which can then be used as supervised training signals.
Fast Structured Decoding for Sequence Models
However, these models assume that the decoding process of each token is conditionally independent of others.
On the Anomalous Generalization of GANs
Motivated by this observation, we discover two specific problems of GANs leading to anomalous generalization behaviour, which we refer to as the sample insufficiency and the pixel-wise combination.
Defective Convolutional Layers Learn Robust CNNs
We first show that predictions made by the defective CNN are less dependent on textural information, but more on shape information, and further find that adversarial examples generated by the defective CNN appear to have semantic shapes.
Hint-Based Training for Non-Autoregressive Machine Translation
Due to the unparallelizable nature of the autoregressive factorization, AutoRegressive Translation (ART) models have to generate tokens sequentially during decoding and thus suffer from high inference latency.
Multilingual Neural Machine Translation with Language Clustering
We study two methods for language clustering: (1) using prior knowledge, where we cluster languages according to language family, and (2) using language embedding, in which we represent each language by an embedding vector and cluster them in the embedding space.
Representation Degeneration Problem in Training Natural Language Generation Models
We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the \emph{representation degeneration problem}.
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system.
Adversarially Robust Generalization Just Requires More Unlabeled Data
Neural network robustness has recently been highlighted by the existence of adversarial examples.
Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks.
Hint-based Training for Non-Autoregressive Translation
To improve the accuracy of NART models, in this paper, we propose to leverage the hints from a well-trained ART model to train the NART model.
Multilingual Neural Machine Translation with Knowledge Distillation
Multilingual machine translation, which translates multiple languages with a single model, has attracted much attention due to its efficiency of offline training and online serving.
Non-Autoregressive Machine Translation with Auxiliary Regularization
However, the high efficiency has come at the cost of not capturing the sequential dependency on the target side of translation, which causes NAT to suffer from two kinds of translation errors: 1) repeated translations (due to indistinguishable adjacent decoder hidden states), and 2) incomplete translations (due to incomplete transfer of source side information via the decoder hidden states).
Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
Non-autoregressive translation (NAT) models, which remove the dependence on previous target tokens from the inputs of the decoder, achieve significantly inference speedup but at the cost of inferior accuracy compared to autoregressive translation (AT) models.
Sentence-wise Smooth Regularization for Sequence to Sequence Learning
Maximum-likelihood estimation (MLE) is widely used in sequence to sequence tasks for model training.
Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation
Neural Machine Translation (NMT) has achieved remarkable progress with the quick evolvement of model structures.
When CTC Training Meets Acoustic Landmarks
In this paper, the convergence properties of CTC are improved by incorporating acoustic landmarks.
Expressiveness in Deep Reinforcement Learning
Based on our observations, we formally define expressiveness of the state extractor as the rank of the matrix composed by representations.
Augmenting Input Method Language Model with user Location Type Information
Geo-tags from micro-blog posts have been shown to be useful in many data mining applications.
Social and Information Networks Computers and Society
FRAGE: Frequency-Agnostic Word Representation
Continuous word representation (aka word embedding) is a basic building block in many neural network-based models used in natural language processing tasks.
Ranked #3 on
Machine Translation
on IWSLT2015 German-English
Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
Many previous works have discussed the relationship between error propagation and the \emph{accuracy drop} (i. e., the left part of the translated sentence is often better than its right part in left-to-right decoding models) problem.
Double Path Networks for Sequence to Sequence Learning
In this work we propose Double Path Networks for Sequence to Sequence learning (DPN-S2S), which leverage the advantages of both models by using double path information fusion.
Towards Binary-Valued Gates for Robust LSTM Training
Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling.
Dense Information Flow for Neural Machine Translation
Recently, neural machine translation has achieved remarkable progress by introducing well-designed deep neural networks into its encoder-decoder framework.
Ranked #68 on
Machine Translation
on WMT2014 English-German
Improved ASR for Under-Resourced Languages Through Multi-Task Learning with Acoustic Landmarks
Furui first demonstrated that the identity of both consonant and vowel can be perceived from the C-V transition; later, Stevens proposed that acoustic landmarks are the primary cues for speech perception, and that steady-state regions are secondary or supplemental.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Decoding with Value Networks for Neural Machine Translation
Inspired by the success and methodology of AlphaGo, in this paper we propose using a prediction network to improve beam search, which takes the source sentence $x$, the currently available decoding output $y_1,\cdots, y_{t-1}$ and a candidate word $w$ at step $t$ as inputs and predicts the long-term value (e. g., BLEU score) of the partial target sentence if it is completed by the NMT model.
Dual Learning for Machine Translation
Based on the feedback signals generated during this process (e. g., the language-model likelihood of the output of a model, and the reconstruction error of the original sentence after the primal and dual translations), we can iteratively update the two models until convergence (e. g., using the policy gradient methods).
Sentence Level Recurrent Topic Model: Letting Topics Speak for Themselves
We propose Sentence Level Recurrent Topic Model (SLRTM), a new topic model that assumes the generation of each word within a sentence to depend on both the topic of the sentence and the whole history of its preceding words in the sentence.
A Game-theoretic Machine Learning Approach for Revenue Maximization in Sponsored Search
Sponsored search is an important monetization channel for search engines, in which an auction mechanism is used to select the ads shown to users and determine the prices charged from advertisers.
A Theoretical Analysis of NDCG Type Ranking Measures
We show that NDCG with logarithmic discount has consistent distinguishability although it converges to the same limit for all ranking functions.
