Search Results for author:
Found 26 papers, 13 papers with code
A Primal-Dual Framework for Transformers and Neural Networks
Self-attention is key to the remarkable success of transformers in sequence modeling tasks including many applications in natural language processing and computer vision.
Low-light phase retrieval with implicit generative priors
Phase retrieval (PR) is fundamentally important in scientific imaging and is crucial for nanoscale techniques like coherent diffractive imaging (CDI).
Wasserstein proximal operators describe score-based generative models and resolve memorization
Via a Cole-Hopf transformation and taking advantage of the fact that the cross-entropy can be related to a linear functional of the density, we show that the HJB equation is an uncontrolled FP equation.
PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Laws
We show the positive evidence to the second question, i. e., ICON can generalize well to some PDEs with new forms without any fine-tuning.
Fine-Tune Language Models as Multi-Modal Differential Equation Solvers
In the growing domain of scientific machine learning, in-context operator learning has shown notable potential in building foundation models, as in this framework the model is trained to learn operators and solve differential equations using prompted data, during the inference stage without weight updates.
In-Context Operator Learning with Data Prompts for Differential Equation Problems
This paper introduces a new neural-network-based approach, namely In-Context Operator Networks (ICON), to simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update.
Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization
Transformers have achieved remarkable success in sequence modeling and beyond but suffer from quadratic computational and memory complexities with respect to the length of the input sequence.
Transformer with Fourier Integral Attentions
Multi-head attention empowers the recent success of transformers, the state-of-the-art models that have achieved remarkable success in sequence modeling and beyond.
Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs
Learning neural ODEs often requires solving very stiff ODE systems, primarily using explicit adaptive step size ODE solvers.
Parameter Inference of Time Series by Delay Embeddings and Learning Differentiable Operators
We provide a method to identify system parameters of dynamical systems, called ID-ODE -- Inference by Differentiation and Observing Delay Embeddings.
Improving Transformers with Probabilistic Attention Keys
Inspired by this observation, we propose Transformer with a Mixture of Gaussian Keys (Transformer-MGK), a novel transformer architecture that replaces redundant heads in transformers with a mixture of keys at each head.
Heavy Ball Neural Ordinary Differential Equations
We propose heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural ODEs (NODEs) training and inference.
FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
For instance, FMMformers achieve an average classification accuracy of $60. 74\%$ over the five Long Range Arena tasks, which is significantly better than the standard transformer's average accuracy of $58. 70\%$.
MomentumRNN: Integrating Momentum into Recurrent Neural Networks
Designing deep neural networks is an art that often involves an expensive search over candidate architectures.
Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets
In this paper, we focus on a co-design of efficient DNN compression algorithms and sparse neural architectures for robust and accurate deep learning.
Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent
Nesterov accelerated gradient (NAG) improves the convergence rate of gradient descent (GD) for convex optimization using a specially designed momentum; however, it accumulates error when an inexact gradient is used (such as in SGD), slowing convergence at best and diverging at worst.
Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games
By phrasing the problem in this manner, solving the MFG can be interpreted as a special case of training a generative adversarial network (GAN).
Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning
The proposed DNN with graph interpolating activation integrates the advantages of both deep learning and manifold learning.
DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM
At the core of DP-LSSGD is the Laplacian smoothing, which smooths out the Gaussian noise used in the Gaussian mechanism.
A Deterministic Gradient-Based Approach to Avoid Saddle Points
First-order methods such as gradient descent are usually the methods of choice for training machine learning models.
ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies
However, both natural and robust accuracies, in classifying clean and adversarial images, respectively, of the trained robust models are far from satisfactory.
Mathematical Analysis of Adversarial Attacks
In this paper, we analyze efficacy of the fast gradient sign method (FGSM) and the Carlini-Wagner's L2 (CW-L2) attack.
Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization
We improve the robustness of Deep Neural Net (DNN) to adversarial attacks by using an interpolating function as the output activation.
Deep Neural Nets with Interpolating Function as Output Activation
We replace the output layer of deep neural nets, typically the softmax function, by a novel interpolating function.
Deep Learning for Real-Time Crime Forecasting and its Ternarization
In this work, we first present a proper representation of crime data.
Optimal Data Collection For Informative Rankings Expose Well-Connected Graphs
Our approach, based on experimental design, is to view data collection as a bi-level optimization problem where the inner problem is the ranking problem and the outer problem is to identify data which maximizes the informativeness of the ranking.
