Search Results for author:
Found 74 papers, 28 papers with code
A Dual-Perspective Approach to Evaluating Feature Attribution Methods
We apply these metrics to mainstream attribution methods, offering a novel lens through which to analyze and compare feature attribution methods.
Tackling the Curse of Dimensionality with Physics-Informed Neural Networks
We experimentally demonstrate that the proposed method allows us to solve many notoriously hard high-dimensional PDEs, including the Hamilton-Jacobi-Bellman (HJB) and the Schr\"{o}dinger equations in thousands of dimensions very fast on a single GPU using the PINNs mesh-free approach.
IF2Net: Innately Forgetting-Free Networks for Continual Learning
Continual learning can incrementally absorb new concepts without interfering with previously learned knowledge.
Multi-View Class Incremental Learning
Multi-view learning (MVL) has gained great success in integrating information from multiple perspectives of a dataset to improve downstream task performance.
Fast Diffusion Model
To mitigate this, we propose a Fast Diffusion Model (FDM) which improves the diffusion process of DMs from a stochastic optimization perspective to speed up both training and sampling.
How Does Information Bottleneck Help Deep Learning?
In this paper, we provide the first rigorous learning theory for justifying the benefit of information bottleneck in deep learning by mathematically relating information bottleneck to generalization errors.
Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
Large Language Models (LLMs) have shown promising performance in knowledge-intensive reasoning tasks that require a compound understanding of knowledge.
Automatic Model Selection with Large Language Models for Reasoning
Chain-of-Thought and Program-Aided Language Models represent two distinct reasoning methods, each with its own strengths and weaknesses.
Ranked #1 on
Math Word Problem Solving
on SVAMP
Decomposition Enhances Reasoning via Self-Evaluation Guided Decoding
With the self-evaluation guided stochastic beam search, we also balance the quality-diversity trade-off in the generation of reasoning chains.
Ranked #3 on
Math Word Problem Solving
on SVAMP
Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networks
As machine learning has been deployed ubiquitously across applications in modern data science, algorithmic fairness has become a great concern.
D4FT: A Deep Learning Approach to Kohn-Sham Density Functional Theory
Kohn-Sham Density Functional Theory (KS-DFT) has been traditionally solved by the Self-Consistent Field (SCF) method.
An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization
In this paper, we provide an information-theoretic perspective on Variance-Invariance-Covariance Regularization (VICReg) for self-supervised learning.
Auxiliary Learning as an Asymmetric Bargaining Game
Auxiliary learning is an effective method for enhancing the generalization capabilities of trained models, particularly when dealing with small datasets.
MixupE: Understanding and Improving Mixup from Directional Derivative Perspective
Mixup is a popular data augmentation technique for training deep neural networks where additional samples are generated by linearly interpolating pairs of inputs and their labels.
Single-Pass Contrastive Learning Can Work for Both Homophilic and Heterophilic Graph
To answer this question, we theoretically study the concentration property of features obtained by neighborhood aggregation on homophilic and heterophilic graphs, introduce the single-pass graph contrastive learning loss based on the property, and provide performance guarantees for the minimizer of the loss on downstream tasks.
Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology
We also show cases where XPINN is already better than PINN, so APINN can still slightly improve XPINN.
Neural Active Learning on Heteroskedastic Distributions
To this end, we demonstrate the catastrophic failure of these active learning algorithms on heteroskedastic distributions and propose a fine-tuning-based approach to mitigate these failures.
Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning
Goal-conditioned reinforcement learning (RL) is a promising direction for training agents that are capable of solving multiple tasks and reach a diverse set of objectives.
TuneUp: A Simple Improved Training Strategy for Graph Neural Networks
Extensive evaluation of TuneUp on five diverse GNN architectures, three types of prediction tasks, and both transductive and inductive settings shows that TuneUp significantly improves the performance of the base GNN on tail nodes, while often even improving the performance on head nodes.
GFlowOut: Dropout with Generative Flow Networks
These methods face two important challenges: (a) the posterior distribution over masks can be highly multi-modal which can be difficult to approximate with standard variational inference and (b) it is not trivial to fully utilize sample-dependent information and correlation among dropout masks to improve posterior estimation.
MGNNI: Multiscale Graph Neural Networks with Implicit Layers
Recently, implicit graph neural networks (GNNs) have been proposed to capture long-range dependencies in underlying graphs.
Self-Distillation for Further Pre-training of Transformers
Pre-training a large transformer model on a massive amount of unlabeled data and fine-tuning it on labeled datasets for diverse downstream tasks has proven to be a successful strategy, for a variety of vision and natural language processing tasks.
Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation
Recent work on mini-batch consistency (MBC) for set functions has brought attention to the need for sequentially processing and aggregating chunks of a partitioned set while guaranteeing the same output for all partitions.
Discrete Key-Value Bottleneck
Deep neural networks perform well on classification tasks where data streams are i. i. d.
Robustness Implies Generalization via Data-Dependent Generalization Bounds
This paper proves that robustness implies generalization via data-dependent generalization bounds.
Set-based Meta-Interpolation for Few-Task Meta-Learning
Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to design augmentation techniques to densify the meta-training task distribution.
Simplicial Embeddings in Self-Supervised Learning and Downstream Classification
Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into $L$ simplices of $V$ dimensions each using a softmax operation.
EIGNN: Efficient Infinite-Depth Graph Neural Networks
Motivated by this limitation, we propose a GNN model with infinite depth, which we call Efficient Infinite-Depth Graph Neural Networks (EIGNN), to efficiently capture very long-range dependencies.
Multi-Task Learning as a Bargaining Game
In this paper, we propose viewing the gradients combination step as a bargaining game, where tasks negotiate to reach an agreement on a joint direction of parameter update.
Ranked #1 on
Multi-Task Learning
on QM9
Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization
Vector Quantization (VQ) is a method for discretizing latent representations and has become a major part of the deep learning toolkit.
ExpertNet: A Symbiosis of Classification and Clustering
We theoretically analyze the effect of clustering on its generalization gap, and empirically show that clustered latent representations from ExpertNet lead to disentangling the intrinsic structure and improvement in classification performance.
Training Free Graph Neural Networks for Graph Matching
We present a framework of Training Free Graph Matching (TFGM) to boost the performance of Graph Neural Networks (GNNs) based graph matching, providing a fast promising solution without training (training-free).
Noether Networks: Meta-Learning Useful Conserved Quantities
Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases.
Understanding End-to-End Model-Based Reinforcement Learning Methods as Implicit Parameterization
Estimating the per-state expected cumulative rewards is a critical aspect of reinforcement learning approaches, however the experience is obtained, but standard deep neural-network function-approximation methods are often inefficient in this setting.
Model-based Reinforcement Learning
reinforcement-learning
+1
Combined Scaling for Zero-shot Transfer Learning
Second, while increasing the dataset size and the model size has been the defacto method to improve the performance of deep learning models like BASIC, the effect of a large contrastive batch size on such contrastive-trained image-text models is not well-understood.
When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
Specifically, for general multi-layer PINNs and XPINNs, we first provide a prior generalization bound via the complexity of the target functions in the PDE problem, and a posterior generalization bound via the posterior matrix norms of the networks after optimization.
Meta-learning PINN loss functions
In the computational examples, the meta-learned losses are employed at test time for addressing regression and PDE task distributions.
Discrete-Valued Neural Communication
Deep learning has advanced from fully connected architectures to structured models organized into components, e. g., the transformer composed of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes.
Understanding Dynamics of Nonlinear Representation Learning and Its Application
Representation learning allows us to automatically discover suitable representations from raw sensory data.
Adversarial Training Helps Transfer Learning via Better Representations
Recent works empirically demonstrate that adversarial training in the source data can improve the ability of models to transfer to new domains.
Sketch-Based Anomaly Detection in Streaming Graphs
This higher-order sketch has the useful property of preserving the dense subgraph structure (dense subgraphs in the input turn into dense submatrices in the data structure).
MemStream: Memory-Based Streaming Anomaly Detection
Given a stream of entries over time in a multi-dimensional data setting where concept drift is present, how can we detect anomalous activities?
Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions
We propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions.
Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth
Our results show that the training of GNNs is implicitly accelerated by skip connections, more depth, and/or a good label distribution.
A Recipe for Global Convergence Guarantee in Deep Neural Networks
Existing global convergence guarantees of (stochastic) gradient descent do not apply to practical deep networks in the practical regime of deep learning beyond the neural tangent kernel (NTK) regime.
Clustering Aware Classification for Risk Prediction and Subtyping in Clinical Data
In data containing heterogeneous subpopulations, classification performance benefits from incorporating the knowledge of cluster structure in the classifier.
On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
In this paper, we analyze the gradient dynamics of deep equilibrium models with nonlinearity only on weight matrices and non-convex objective functions of weights for regression and classification.
When and How Mixup Improves Calibration
In addition, we study how Mixup improves calibration in semi-supervised learning.
Dynamics of Deep Equilibrium Linear Models
A deep equilibrium linear model is implicitly defined through an equilibrium point of an infinite sequence of computation.
Towards Domain-Agnostic Contrastive Learning
Despite recent success, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation.
How Does Mixup Help With Robustness and Generalization?
For robustness, we show that minimizing the Mixup loss corresponds to approximately minimizing an upper bound of the adversarial loss.
Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
Adding auxiliary losses to the main objective function is a general way of encoding biases that can help networks learn better representations.
GraphMix: Improved Training of GNNs for Semi-Supervised Learning
We present GraphMix, a regularization method for Graph Neural Network based semi-supervised object classification, whereby we propose to train a fully-connected network jointly with the graph neural network via parameter sharing and interpolation-based regularization.
Ranked #1 on
Node Classification
on Bitcoin-Alpha
Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks
Furthermore, the proposed methods with the slope recovery are shown to accelerate the training process.
Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes
The theory developed in this paper only requires the practical degrees of over-parameterization unlike previous theories.
Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimization
The traditional approaches, such as (mini-batch) stochastic gradient descent (SGD), utilize an unbiased gradient estimator of the empirical average loss.
Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
Adversarial robustness has become a central goal in deep learning, both in the theory and the practice.
Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learning
Furthermore, as special cases of our general results, this article improves or complements several state-of-the-art theoretical results on deep neural networks, deep residual networks, and overparameterized deep neural networks with a unified proof technique and novel geometric insights.
Interpolation Consistency Training for Semi-Supervised Learning
We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm.
Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit
Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization.
Elimination of All Bad Local Minima in Deep Learning
At every local minimum of any deep neural network with these added neurons, the set of parameters of the original neural network (without added neurons) is guaranteed to be a global minimum of the original neural network.
Effect of Depth and Width on Local Minima in Deep Learning
In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature.
Depth with Nonlinearity Creates No Bad Local Minima in ResNets
In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are guaranteed to further improve via residual representations.
Generalization in Machine Learning via Analytical Learning Theory
This paper introduces a novel measure-theoretic theory for machine learning that does not require statistical assumptions.
Theory of Deep Learning III: explaining the non-overfitting puzzle
In this note, we show that the dynamics associated to gradient descent minimization of nonlinear networks is topologically equivalent, near the asymptotically stable minima of the empirical error, to linear gradient system in a quadratic potential with a degenerate (for square loss) or almost degenerate (for logistic or crossentropy loss) Hessian.
Generalization in Deep Learning
This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature.
Deep Semi-Random Features for Nonlinear Function Approximation
For deep models, with no unrealistic assumptions, we prove universal approximation ability, a lower bound on approximation error, a partial optimization guarantee, and a generalization bound.
Depth Creates No Bad Local Minima
In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces.
Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning.
Global Continuous Optimization with Error Bound and Fast Convergence
This paper proposes a new global optimization algorithm, called Locally Oriented Global Optimization (LOGO), to aim for both fast convergence in practice and finite-time error bound in theory.
Deep Learning without Poor Local Minima
In this paper, we prove a conjecture published in 1989 and also partially address an open problem announced at the Conference on Learning Theory (COLT) 2015.
Bounded Optimal Exploration in MDP
Within the framework of probably approximately correct Markov decision processes (PAC-MDP), much theoretical work has focused on methods to attain near optimality after a relatively long period of learning and exploration.
Bayesian Optimization with Exponential Convergence
This paper presents a Bayesian optimization method with exponential convergence without the need of auxiliary optimization and without the delta-cover sampling.
A Greedy Approximation of Bayesian Reinforcement Learning with Probably Optimistic Transition Model
This is a natural consequence of the fact that the proposed algorithm is greedier than the other algorithms.
