Search Results for author:
Found 101 papers, 38 papers with code
A Greedy Approximation of Bayesian Reinforcement Learning with Probably Optimistic Transition Model
This is a natural consequence of the fact that the proposed algorithm is greedier than the other algorithms.
Bounded Optimal Exploration in MDP
Within the framework of probably approximately correct Markov decision processes (PAC-MDP), much theoretical work has focused on methods to attain near optimality after a relatively long period of learning and exploration.
Bayesian Optimization with Exponential Convergence
This paper presents a Bayesian optimization method with exponential convergence without the need of auxiliary optimization and without the delta-cover sampling.
Deep Learning without Poor Local Minima
In this paper, we prove a conjecture published in 1989 and also partially address an open problem announced at the Conference on Learning Theory (COLT) 2015.
Global Continuous Optimization with Error Bound and Fast Convergence
This paper proposes a new global optimization algorithm, called Locally Oriented Global Optimization (LOGO), to aim for both fast convergence in practice and finite-time error bound in theory.
Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning.
Depth Creates No Bad Local Minima
In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces.
Deep Semi-Random Features for Nonlinear Function Approximation
For deep models, with no unrealistic assumptions, we prove universal approximation ability, a lower bound on approximation error, a partial optimization guarantee, and a generalization bound.
Generalization in Deep Learning
This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature.
Theory of Deep Learning III: explaining the non-overfitting puzzle
In this note, we show that the dynamics associated to gradient descent minimization of nonlinear networks is topologically equivalent, near the asymptotically stable minima of the empirical error, to linear gradient system in a quadratic potential with a degenerate (for square loss) or almost degenerate (for logistic or crossentropy loss) Hessian.
Generalization in Machine Learning via Analytical Learning Theory
This paper introduces a novel measure-theoretic theory for machine learning that does not require statistical assumptions.
Depth with Nonlinearity Creates No Bad Local Minima in ResNets
In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are guaranteed to further improve via residual representations.
Effect of Depth and Width on Local Minima in Deep Learning
In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature.
Elimination of All Bad Local Minima in Deep Learning
At every local minimum of any deep neural network with these added neurons, the set of parameters of the original neural network (without added neurons) is guaranteed to be a global minimum of the original neural network.
Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit
Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization.
Interpolation Consistency Training for Semi-Supervised Learning
We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm.
Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learning
Furthermore, as special cases of our general results, this article improves or complements several state-of-the-art theoretical results on deep neural networks, deep residual networks, and overparameterized deep neural networks with a unified proof technique and novel geometric insights.
Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
Adversarial robustness has become a central goal in deep learning, both in the theory and the practice.
Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimization
The traditional approaches, such as (mini-batch) stochastic gradient descent (SGD), utilize an unbiased gradient estimator of the empirical average loss.
Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes
The theory developed in this paper only requires the practical degrees of over-parameterization unlike previous theories.
GraphMix: Improved Training of GNNs for Semi-Supervised Learning
We present GraphMix, a regularization method for Graph Neural Network based semi-supervised object classification, whereby we propose to train a fully-connected network jointly with the graph neural network via parameter sharing and interpolation-based regularization.
Ranked #1 on
Node Classification
on Pubmed random partition
Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks
Furthermore, the proposed methods with the slope recovery are shown to accelerate the training process.
Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
Adding auxiliary losses to the main objective function is a general way of encoding biases that can help networks learn better representations.
How Does Mixup Help With Robustness and Generalization?
For robustness, we show that minimizing the Mixup loss corresponds to approximately minimizing an upper bound of the adversarial loss.
Towards Domain-Agnostic Contrastive Learning
Despite recent success, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation.
Dynamics of Deep Equilibrium Linear Models
A deep equilibrium linear model is implicitly defined through an equilibrium point of an infinite sequence of computation.
When and How Mixup Improves Calibration
In addition, we study how Mixup improves calibration in semi-supervised learning.
On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
In this paper, we analyze the gradient dynamics of deep equilibrium models with nonlinearity only on weight matrices and non-convex objective functions of weights for regression and classification.
Clustering Aware Classification for Risk Prediction and Subtyping in Clinical Data
In data containing heterogeneous subpopulations, classification performance benefits from incorporating the knowledge of cluster structure in the classifier.
A Recipe for Global Convergence Guarantee in Deep Neural Networks
Existing global convergence guarantees of (stochastic) gradient descent do not apply to practical deep networks in the practical regime of deep learning beyond the neural tangent kernel (NTK) regime.
Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth
Our results show that the training of GNNs is implicitly accelerated by skip connections, more depth, and/or a good label distribution.
Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions
We propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions.
MemStream: Memory-Based Streaming Anomaly Detection
Given a stream of entries over time in a multi-dimensional data setting where concept drift is present, how can we detect anomalous activities?
Sketch-Based Anomaly Detection in Streaming Graphs
This higher-order sketch has the useful property of preserving the dense subgraph structure (dense subgraphs in the input turn into dense submatrices in the data structure).
Adversarial Training Helps Transfer Learning via Better Representations
Recent works empirically demonstrate that adversarial training in the source data can improve the ability of models to transfer to new domains.
Understanding Dynamics of Nonlinear Representation Learning and Its Application
Representation learning allows us to automatically discover suitable representations from raw sensory data.
Discrete-Valued Neural Communication
Deep learning has advanced from fully connected architectures to structured models organized into components, e. g., the transformer composed of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes.
Meta-learning PINN loss functions
In the computational examples, the meta-learned losses are employed at test time for addressing regression and PDE task distributions.
When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
Specifically, for general multi-layer PINNs and XPINNs, we first provide a prior generalization bound via the complexity of the target functions in the PDE problem, and a posterior generalization bound via the posterior matrix norms of the networks after optimization.
Combined Scaling for Zero-shot Transfer Learning
Second, while increasing the dataset size and the model size has been the defacto method to improve the performance of deep learning models like BASIC, the effect of a large contrastive batch size on such contrastive-trained image-text models is not well-understood.
Understanding End-to-End Model-Based Reinforcement Learning Methods as Implicit Parameterization
Estimating the per-state expected cumulative rewards is a critical aspect of reinforcement learning approaches, however the experience is obtained, but standard deep neural-network function-approximation methods are often inefficient in this setting.
Model-based Reinforcement Learning
reinforcement-learning
+1
Noether Networks: Meta-Learning Useful Conserved Quantities
Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases.
Training Free Graph Neural Networks for Graph Matching
We present a framework of Training Free Graph Matching (TFGM) to boost the performance of Graph Neural Networks (GNNs) based graph matching, providing a fast promising solution without training (training-free).
ExpertNet: A Symbiosis of Classification and Clustering
We theoretically analyze the effect of clustering on its generalization gap, and empirically show that clustered latent representations from ExpertNet lead to disentangling the intrinsic structure and improvement in classification performance.
Multi-Task Learning as a Bargaining Game
In this paper, we propose viewing the gradients combination step as a bargaining game, where tasks negotiate to reach an agreement on a joint direction of parameter update.
Ranked #1 on
Multi-Task Learning
on Cityscapes test
Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization
Vector Quantization (VQ) is a method for discretizing latent representations and has become a major part of the deep learning toolkit.
EIGNN: Efficient Infinite-Depth Graph Neural Networks
Motivated by this limitation, we propose a GNN model with infinite depth, which we call Efficient Infinite-Depth Graph Neural Networks (EIGNN), to efficiently capture very long-range dependencies.
Simplicial Embeddings in Self-Supervised Learning and Downstream Classification
Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into $L$ simplices of $V$ dimensions each using a softmax operation.
Set-based Meta-Interpolation for Few-Task Meta-Learning
Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to design augmentation techniques to densify the meta-training task distribution.
Robustness Implies Generalization via Data-Dependent Generalization Bounds
This paper proves that robustness implies generalization via data-dependent generalization bounds.
Discrete Key-Value Bottleneck
Deep neural networks perform well on classification tasks where data streams are i. i. d.
Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation
Recent work on mini-batch consistency (MBC) for set functions has brought attention to the need for sequentially processing and aggregating chunks of a partitioned set while guaranteeing the same output for all partitions.
Self-Distillation for Further Pre-training of Transformers
Pre-training a large transformer model on a massive amount of unlabeled data and fine-tuning it on labeled datasets for diverse downstream tasks has proven to be a successful strategy, for a variety of vision and natural language processing tasks.
MGNNI: Multiscale Graph Neural Networks with Implicit Layers
Recently, implicit graph neural networks (GNNs) have been proposed to capture long-range dependencies in underlying graphs.
GFlowOut: Dropout with Generative Flow Networks
These methods face two important challenges: (a) the posterior distribution over masks can be highly multi-modal which can be difficult to approximate with standard variational inference and (b) it is not trivial to fully utilize sample-dependent information and correlation among dropout masks to improve posterior estimation.
TuneUp: A Simple Improved Training Strategy for Graph Neural Networks
Extensive evaluation of TuneUp on five diverse GNN architectures, three types of prediction tasks, and both transductive and inductive settings shows that TuneUp significantly improves the performance of the base GNN on tail nodes, while often even improving the performance on head nodes.
Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning
Goal-conditioned reinforcement learning (RL) is a promising direction for training agents that are capable of solving multiple tasks and reach a diverse set of objectives.
Neural Active Learning on Heteroskedastic Distributions
To this end, we demonstrate the catastrophic failure of these active learning algorithms on heteroskedastic distributions and propose a fine-tuning-based approach to mitigate these failures.
Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology
We also show cases where XPINN is already better than PINN, so APINN can still slightly improve XPINN.
Single-Pass Contrastive Learning Can Work for Both Homophilic and Heterophilic Graph
To answer this question, we theoretically study the concentration property of features obtained by neighborhood aggregation on homophilic and heterophilic graphs, introduce the single-pass augmentation-free graph contrastive learning loss based on the property, and provide performance guarantees for the minimizer of the loss on downstream tasks.
MixupE: Understanding and Improving Mixup from Directional Derivative Perspective
Mixup is a popular data augmentation technique for training deep neural networks where additional samples are generated by linearly interpolating pairs of inputs and their labels.
Auxiliary Learning as an Asymmetric Bargaining Game
Auxiliary learning is an effective method for enhancing the generalization capabilities of trained models, particularly when dealing with small datasets.
D4FT: A Deep Learning Approach to Kohn-Sham Density Functional Theory
Kohn-Sham Density Functional Theory (KS-DFT) has been traditionally solved by the Self-Consistent Field (SCF) method.
An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization
In this paper, we provide an information-theoretic perspective on Variance-Invariance-Covariance Regularization (VICReg) for self-supervised learning.
Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networks
As machine learning has been deployed ubiquitously across applications in modern data science, algorithmic fairness has become a great concern.
Self-Evaluation Guided Beam Search for Reasoning
Stochastic beam search balances exploitation and exploration of the search space with temperature-controlled randomness.
Boosting Visual-Language Models by Exploiting Hard Samples
In this work, we present HELIP, a cost-effective strategy tailored to enhance the performance of existing CLIP models without the need for training a model from scratch or collecting additional data.
Automatic Model Selection with Large Language Models for Reasoning
Chain-of-Thought (CoT) and Program-Aided Language Models (PAL) represent two distinct reasoning methods, each with its own strengths.
Ranked #1 on
Math Word Problem Solving
on SVAMP
Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
Large Language Models (LLMs) have shown promising performance in knowledge-intensive reasoning tasks that require a compound understanding of knowledge.
How Does Information Bottleneck Help Deep Learning?
In this paper, we provide the first rigorous learning theory for justifying the benefit of information bottleneck in deep learning by mathematically relating information bottleneck to generalization errors.
Fast Diffusion Model
In this paper, we propose a Fast Diffusion Model (FDM) to significantly speed up DMs from a stochastic optimization perspective for both faster training and sampling.
Multi-View Class Incremental Learning
Multi-view learning (MVL) has gained great success in integrating information from multiple perspectives of a dataset to improve downstream task performance.
IF2Net: Innately Forgetting-Free Networks for Continual Learning
Continual learning can incrementally absorb new concepts without interfering with previously learned knowledge.
Tackling the Curse of Dimensionality with Physics-Informed Neural Networks
We demonstrate in various diverse tests that the proposed method can solve many notoriously hard high-dimensional PDEs, including the Hamilton-Jacobi-Bellman (HJB) and the Schr\"{o}dinger equations in tens of thousands of dimensions very fast on a single GPU using the PINNs mesh-free approach.
A Dual-Perspective Approach to Evaluating Feature Attribution Methods
We apply these metrics to mainstream attribution methods, offering a novel lens through which to analyze and compare feature attribution methods.
On Copyright Risks of Text-to-Image Diffusion Models
Specifically, we introduce a data generation pipeline to systematically produce data for studying copyright in diffusion models.
Drug Discovery with Dynamic Goal-aware Fragments
Additionally, the existing fragment-based generative models cannot update the fragment vocabulary with goal-aware fragments newly discovered during the generation.
Self-Supervised Dataset Distillation for Transfer Learning
To achieve this, we also introduce the MSE between representations of the inner model and the self-supervised target model on the original full dataset for outer optimization.
AttributionLab: Faithfulness of Feature Attribution Under Controllable Environments
Feature attribution explains neural network outputs by identifying relevant input features.
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
MolCA enables an LM (e. g., Galactica) to understand both text- and graph-based molecular contents via the cross-modal projector.
Ranked #4 on
Molecule Captioning
on ChEBI-20
Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules
Our results show that a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding have a large impact on the encoder's representation learning.
ChOiRe: Characterizing and Predicting Human Opinions with Chain of Opinion Reasoning
Aligning language models (LMs) with human opinion is challenging yet vital to enhance their grasp of human values, preferences, and beliefs.
Bias-Variance Trade-off in Physics-Informed Neural Networks with Randomized Smoothing for High-Dimensional PDEs
To optimize the bias-variance trade-off, we combine the two approaches in a hybrid method that balances the rapid convergence of the biased version with the high accuracy of the unbiased version.
VA3: Virtually Assured Amplification Attack on Probabilistic Copyright Protection for Text-to-Image Generative Models
The booming use of text-to-image generative models has raised concerns about their high risk of producing copyright-infringing content.
Learning Unorthogonalized Matrices for Rotation Estimation
Estimating 3D rotations is a common procedure for 3D computer vision.
Prompt Optimization via Adversarial In-Context Learning
We propose a new method, Adversarial In-Context Learning (adv-ICL), to optimize prompt for in-context learning (ICL) by employing one LLM as a generator, another as a discriminator, and a third as a prompt modifier.
Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks
We further showcase HTE's convergence to the original PINN loss and its unbiased behavior under specific conditions.
Can AI Be as Creative as Humans?
With the rise of advanced generative AI models capable of tasks once reserved for human creativity, the study of AI's creative potential becomes imperative for its responsible development and application.
Simple Hierarchical Planning with Diffusion
Diffusion-based generative methods have proven effective in modeling trajectories with offline datasets.
The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipeline
This study explores the vulnerabilities associated with copyright protection in DMs by introducing a backdoor data poisoning attack (SilentBadDiffusion) against text-to-image diffusion models.
Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding
Deep neural networks are susceptible to catastrophic forgetting when trained on sequential tasks.
Towards 3D Molecule-Text Interpretation in Language Models
Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM.
Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck Equations
The score function, defined as the gradient of the LL, plays a fundamental role in inferring LL and PDF and enables fast SDE sampling.
Unsupervised Concept Discovery Mitigates Spurious Correlations
Models prone to spurious correlations in training data often produce brittle predictions and introduce unintended biases.
The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling
Surprisingly, the improvement persists when we increase the number of sampling steps and can even surpass the best result from EDM-2 (1. 58) with only 39 NFEs (1. 57).
Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion
In this work, we aim to address this alignment challenge for conditional generation tasks.
AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging
In this paper, we acknowledge the mutual reliance between task ability and language ability and direct our attention toward the gap between the target language and the source language on tasks.
How do Large Language Models Handle Multilingualism?
We introduce a framework that depicts LLMs' processing of multilingual inputs: In the first several layers, LLMs understand the question, converting multilingual inputs into English to facilitate the task-solving phase.
Accelerating Greedy Coordinate Gradient via Probe Sampling
Safety of Large Language Models (LLMs) has become a central issue given their rapid progress and wide applications.
Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
Recent advancements in diffusion models have notably improved the perceptual quality of generated images in text-to-image synthesis tasks.
Towards Robust Out-of-Distribution Generalization Bounds via Sharpness
To bridge this gap between optimization and OOD generalization, we study the effect of sharpness on how a model tolerates data change in domain shift which is usually captured by "robustness" in generalization.
