Search Results for author:
Found 121 papers, 53 papers with code
Effortless Efficiency: Low-Cost Pruning of Diffusion Models
Diffusion models have achieved impressive advancements in various vision tasks.
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
To address this, we develop AnchorAttention, a plug-and-play attention method that alleviates numerical issues caused by BFloat16, improves long-context capabilities, and speeds up training.
Reasoning Robustness of LLMs to Adversarial Typographical Errors
It shows that LLMs are sensitive to minimal adversarial typographical changes.
Multi-expert Prompting Improves Reliability, Safety, and Usefulness of Large Language Models
We present Multi-expert Prompting, a novel enhancement of ExpertPrompting (Xu et al., 2023), designed to improve the large language model (LLM) generation.
Investigating Layer Importance in Large Language Models
In this study, we advance the understanding of LLM by investigating the significance of individual layers in LLMs.
State-space models are accurate and efficient neural operators for dynamical systems
We demonstrate Mamba's superior performance in both interpolation and challenging extrapolation tasks.
LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs
We present the first systematic evaluation examining format bias in performance of large language models (LLMs).
Single Character Perturbations Break LLM Alignment
When LLMs are deployed in sensitive, human-facing settings, it is crucial that they do not output unsafe, biased, or privacy-violating outputs.
Self-Evaluation as a Defense Against Adversarial Attacks on LLMs
We introduce a defense against adversarial attacks on LLMs utilizing self-evaluation.
Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization
Bi-level optimization (BO) has become a fundamental mathematical framework for addressing hierarchical machine learning problems.
Score-fPINN: Fractional Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck-Levy Equations
We introduce an innovative approach for solving high-dimensional Fokker-Planck-L\'evy (FPL) equations in modeling non-Brownian processes across disciplines such as physics, finance, and ecology.
Tackling the Curse of Dimensionality in Fractional and Tempered Fractional PDEs with Physics-Informed Neural Networks
We validate our methods on various forward and inverse problems of fractional and tempered fractional PDEs, scaling up to 100, 000 dimensions.
PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer
At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals.
Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers
We further suggest how our method can be adapted to achieve cheap approximate conversion of ICL tokens, even in regular transformer networks that are not linearized.
Learning diverse attacks on large language models for robust red-teaming and safety tuning
Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of large language models (LLMs).
FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models
Overparametrized transformer networks are the state-of-the-art architecture for Large Language Models (LLMs).
ReactXT: Understanding Molecular "Reaction-ship" via Reaction-Contextualized Molecule-Text Pretraining
To resolve the challenges above, we propose a new pretraining method, ReactXT, for reaction-text modeling, and a new dataset, OpenExp, for experimental procedure prediction.
ProtT3: Protein-to-Text Generation for Text-based Protein Understanding
ProtT3 empowers an LM to understand protein sequences of amino acids by incorporating a PLM as its protein understanding module, enabling effective protein-to-text generation.
Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process inspired by the successful strategy employed by AlphaZero.
Deep Regression Representation Learning with Topology
Based on these two connections, we introduce PH-Reg, a regularizer specific to regression that matches the intrinsic dimension and topology of the feature space with the target space.
Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models
Recent advancements in diffusion models have notably improved the perceptual quality of generated images in text-to-image synthesis tasks.
Towards Robust Out-of-Distribution Generalization Bounds via Sharpness
To bridge this gap between optimization and OOD generalization, we study the effect of sharpness on how a model tolerates data change in domain shift which is usually captured by "robustness" in generalization.
Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling
Safety of Large Language Models (LLMs) has become a critical issue given their rapid progresses.
How do Large Language Models Handle Multilingualism?
Based on observed language ratio shifts among layers and the relationships between network structures and certain capabilities, we hypothesize the LLM's multilingual workflow ($\texttt{MWork}$): LLMs initially understand the query, converting multilingual inputs into English for task-solving.
AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging
In this paper, we acknowledge the mutual reliance between task ability and language ability and direct our attention toward the gap between the target language and the source language on tasks.
Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion
In this work, we aim to address this alignment challenge for conditional generation tasks.
The Surprising Effectiveness of Skip-Tuning in Diffusion Sampling
Surprisingly, the improvement persists when we increase the number of sampling steps and can even surpass the best result from EDM-2 (1. 58) with only 39 NFEs (1. 57).
Unsupervised Concept Discovery Mitigates Spurious Correlations
Models prone to spurious correlations in training data often produce brittle predictions and introduce unintended biases.
Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck Equations
The score function, defined as the gradient of the LL, plays a fundamental role in inferring LL and PDF and enables fast SDE sampling.
Towards 3D Molecule-Text Interpretation in Language Models
Through 3D molecule-text alignment and 3D molecule-centric instruction tuning, 3D-MoLM establishes an integration of 3D molecular encoder and LM.
Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding
Deep neural networks are susceptible to catastrophic forgetting when trained on sequential tasks.
The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright Breaches Without Adjusting Finetuning Pipeline
Our method strategically embeds connections between pieces of copyrighted information and text references in poisoning data while carefully dispersing that information, making the poisoning data inconspicuous when integrated into a clean dataset.
Simple Hierarchical Planning with Diffusion
Diffusion-based generative methods have proven effective in modeling trajectories with offline datasets.
Can AI Be as Creative as Humans?
With the rise of advanced generative AI models capable of tasks once reserved for human creativity, the study of AI's creative potential becomes imperative for its responsible development and application.
Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks
We further showcase HTE's convergence to the original PINN loss and its unbiased behavior under specific conditions.
Prompt Optimization via Adversarial In-Context Learning
We propose a new method, Adversarial In-Context Learning (adv-ICL), to optimize prompt for in-context learning (ICL) by employing one LLM as a generator, another as a discriminator, and a third as a prompt modifier.
Learning Unorthogonalized Matrices for Rotation Estimation
Estimating 3D rotations is a common procedure for 3D computer vision.
Ranked #15 on
3D Human Pose Estimation
on 3DPW
VA3: Virtually Assured Amplification Attack on Probabilistic Copyright Protection for Text-to-Image Generative Models
The booming use of text-to-image generative models has raised concerns about their high risk of producing copyright-infringing content.
Bias-Variance Trade-off in Physics-Informed Neural Networks with Randomized Smoothing for High-Dimensional PDEs
To optimize the bias-variance trade-off, we combine the two approaches in a hybrid method that balances the rapid convergence of the biased version with the high accuracy of the unbiased version.
Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoning
Reasoning and predicting human opinions with large language models (LLMs) is essential yet challenging.
Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules
Our results show that a subgraph-level tokenizer and a sufficiently expressive decoder with remask decoding have a large impact on the encoder's representation learning.
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
MolCA enables an LM (e. g., Galactica) to understand both text- and graph-based molecular contents via the cross-modal projector.
Ranked #5 on
Molecule Captioning
on ChEBI-20
AttributionLab: Faithfulness of Feature Attribution Under Controllable Environments
Feature attribution explains neural network outputs by identifying relevant input features.
Self-Supervised Dataset Distillation for Transfer Learning
To achieve this, we also introduce the MSE between representations of the inner model and the self-supervised target model on the original full dataset for outer optimization.
Drug Discovery with Dynamic Goal-aware Fragments
Additionally, the existing fragment-based generative models cannot update the fragment vocabulary with goal-aware fragments newly discovered during the generation.
On Copyright Risks of Text-to-Image Diffusion Models
Specifically, we introduce a data generation pipeline to systematically produce data for studying copyright in diffusion models.
A Dual-Perspective Approach to Evaluating Feature Attribution Methods
We apply these metrics to mainstream attribution methods, offering a novel lens through which to analyze and compare feature attribution methods.
Tackling the Curse of Dimensionality with Physics-Informed Neural Networks
We demonstrate in various diverse tests that the proposed method can solve many notoriously hard high-dimensional PDEs, including the Hamilton-Jacobi-Bellman (HJB) and the Schr\"{o}dinger equations in tens of thousands of dimensions very fast on a single GPU using the PINNs mesh-free approach.
IF2Net: Innately Forgetting-Free Networks for Continual Learning
Continual learning can incrementally absorb new concepts without interfering with previously learned knowledge.
Multi-View Class Incremental Learning
Multi-view learning (MVL) has gained great success in integrating information from multiple perspectives of a dataset to improve downstream task performance.
Fast Diffusion Model
In this paper, we propose a Fast Diffusion Model (FDM) to significantly speed up DMs from a stochastic optimization perspective for both faster training and sampling.
How Does Information Bottleneck Help Deep Learning?
In this paper, we provide the first rigorous learning theory for justifying the benefit of information bottleneck in deep learning by mathematically relating information bottleneck to generalization errors.
Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
Large Language Models (LLMs) have shown promising performance in knowledge-intensive reasoning tasks that require a compound understanding of knowledge.
Automatic Model Selection with Large Language Models for Reasoning
Chain-of-Thought (CoT) and Program-Aided Language Models (PAL) represent two distinct reasoning methods, each with its own strengths.
Ranked #2 on
Math Word Problem Solving
on SVAMP
Boosting Visual-Language Models by Exploiting Hard Samples
In this work, we present HELIP, a cost-effective strategy tailored to enhance the performance of existing CLIP models without the need for training a model from scratch or collecting additional data.
Self-Evaluation Guided Beam Search for Reasoning
Stochastic beam search balances exploitation and exploration of the search space with temperature-controlled randomness.
Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networks
As machine learning has been deployed ubiquitously across applications in modern data science, algorithmic fairness has become a great concern.
An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization
Variance-Invariance-Covariance Regularization (VICReg) is a self-supervised learning (SSL) method that has shown promising results on a variety of tasks.
D4FT: A Deep Learning Approach to Kohn-Sham Density Functional Theory
Kohn-Sham Density Functional Theory (KS-DFT) has been traditionally solved by the Self-Consistent Field (SCF) method.
Auxiliary Learning as an Asymmetric Bargaining Game
Auxiliary learning is an effective method for enhancing the generalization capabilities of trained models, particularly when dealing with small datasets.
MixupE: Understanding and Improving Mixup from Directional Derivative Perspective
Mixup is a popular data augmentation technique for training deep neural networks where additional samples are generated by linearly interpolating pairs of inputs and their labels.
Single-Pass Contrastive Learning Can Work for Both Homophilic and Heterophilic Graph
To answer this question, we theoretically study the concentration property of features obtained by neighborhood aggregation on homophilic and heterophilic graphs, introduce the single-pass augmentation-free graph contrastive learning loss based on the property, and provide performance guarantees for the minimizer of the loss on downstream tasks.
Augmented Physics-Informed Neural Networks (APINNs): A gating network-based soft domain decomposition methodology
We also show cases where XPINN is already better than PINN, so APINN can still slightly improve XPINN.
Neural Active Learning on Heteroskedastic Distributions
To this end, we demonstrate the catastrophic failure of these active learning algorithms on heteroskedastic distributions and propose a fine-tuning-based approach to mitigate these failures.
Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning
Goal-conditioned reinforcement learning (RL) is a promising direction for training agents that are capable of solving multiple tasks and reach a diverse set of objectives.
TuneUp: A Simple Improved Training Strategy for Graph Neural Networks
Extensive evaluation of TuneUp on five diverse GNN architectures, three types of prediction tasks, and both transductive and inductive settings shows that TuneUp significantly improves the performance of the base GNN on tail nodes, while often even improving the performance on head nodes.
GFlowOut: Dropout with Generative Flow Networks
These methods face two important challenges: (a) the posterior distribution over masks can be highly multi-modal which can be difficult to approximate with standard variational inference and (b) it is not trivial to fully utilize sample-dependent information and correlation among dropout masks to improve posterior estimation.
MGNNI: Multiscale Graph Neural Networks with Implicit Layers
Recently, implicit graph neural networks (GNNs) have been proposed to capture long-range dependencies in underlying graphs.
Self-Distillation for Further Pre-training of Transformers
Pre-training a large transformer model on a massive amount of unlabeled data and fine-tuning it on labeled datasets for diverse downstream tasks has proven to be a successful strategy, for a variety of vision and natural language processing tasks.
Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation
Recent work on mini-batch consistency (MBC) for set functions has brought attention to the need for sequentially processing and aggregating chunks of a partitioned set while guaranteeing the same output for all partitions.
Discrete Key-Value Bottleneck
Deep neural networks perform well on classification tasks where data streams are i. i. d.
Robustness Implies Generalization via Data-Dependent Generalization Bounds
This paper proves that robustness implies generalization via data-dependent generalization bounds.
Set-based Meta-Interpolation for Few-Task Meta-Learning
Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to design augmentation techniques to densify the meta-training task distribution.
Simplicial Embeddings in Self-Supervised Learning and Downstream Classification
Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into $L$ simplices of $V$ dimensions each using a softmax operation.
EIGNN: Efficient Infinite-Depth Graph Neural Networks
Motivated by this limitation, we propose a GNN model with infinite depth, which we call Efficient Infinite-Depth Graph Neural Networks (EIGNN), to efficiently capture very long-range dependencies.
Multi-Task Learning as a Bargaining Game
In this paper, we propose viewing the gradients combination step as a bargaining game, where tasks negotiate to reach an agreement on a joint direction of parameter update.
Ranked #2 on
Multi-Task Learning
on Cityscapes test
Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization
Vector Quantization (VQ) is a method for discretizing latent representations and has become a major part of the deep learning toolkit.
ExpertNet: A Symbiosis of Classification and Clustering
We theoretically analyze the effect of clustering on its generalization gap, and empirically show that clustered latent representations from ExpertNet lead to disentangling the intrinsic structure and improvement in classification performance.
Training Free Graph Neural Networks for Graph Matching
We present a framework of Training Free Graph Matching (TFGM) to boost the performance of Graph Neural Networks (GNNs) based graph matching, providing a fast promising solution without training (training-free).
Noether Networks: Meta-Learning Useful Conserved Quantities
Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases.
Understanding End-to-End Model-Based Reinforcement Learning Methods as Implicit Parameterization
Estimating the per-state expected cumulative rewards is a critical aspect of reinforcement learning approaches, however the experience is obtained, but standard deep neural-network function-approximation methods are often inefficient in this setting.
Model-based Reinforcement Learning
reinforcement-learning
+1
Combined Scaling for Zero-shot Transfer Learning
Second, while increasing the dataset size and the model size has been the defacto method to improve the performance of deep learning models like BASIC, the effect of a large contrastive batch size on such contrastive-trained image-text models is not well-understood.
When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
Specifically, for general multi-layer PINNs and XPINNs, we first provide a prior generalization bound via the complexity of the target functions in the PDE problem, and a posterior generalization bound via the posterior matrix norms of the networks after optimization.
Meta-learning PINN loss functions
In the computational examples, the meta-learned losses are employed at test time for addressing regression and PDE task distributions.
Discrete-Valued Neural Communication
Deep learning has advanced from fully connected architectures to structured models organized into components, e. g., the transformer composed of positional elements, modular architectures divided into slots, and graph neural nets made up of nodes.
Understanding Dynamics of Nonlinear Representation Learning and Its Application
Representation learning allows us to automatically discover suitable representations from raw sensory data.
Adversarial Training Helps Transfer Learning via Better Representations
Recent works empirically demonstrate that adversarial training in the source data can improve the ability of models to transfer to new domains.
Sketch-Based Anomaly Detection in Streaming Graphs
This higher-order sketch has the useful property of preserving the dense subgraph structure (dense subgraphs in the input turn into dense submatrices in the data structure).
MemStream: Memory-Based Streaming Anomaly Detection
Given a stream of entries over time in a multi-dimensional data setting where concept drift is present, how can we detect anomalous activities?
Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions
We propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions.
Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth
Our results show that the training of GNNs is implicitly accelerated by skip connections, more depth, and/or a good label distribution.
A Recipe for Global Convergence Guarantee in Deep Neural Networks
Existing global convergence guarantees of (stochastic) gradient descent do not apply to practical deep networks in the practical regime of deep learning beyond the neural tangent kernel (NTK) regime.
Clustering Aware Classification for Risk Prediction and Subtyping in Clinical Data
In data containing heterogeneous subpopulations, classification performance benefits from incorporating the knowledge of cluster structure in the classifier.
On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
In this paper, we analyze the gradient dynamics of deep equilibrium models with nonlinearity only on weight matrices and non-convex objective functions of weights for regression and classification.
When and How Mixup Improves Calibration
In addition, we study how Mixup improves calibration in semi-supervised learning.
Dynamics of Deep Equilibrium Linear Models
A deep equilibrium linear model is implicitly defined through an equilibrium point of an infinite sequence of computation.
Towards Domain-Agnostic Contrastive Learning
Despite recent success, most contrastive self-supervised learning methods are domain-specific, relying heavily on data augmentation techniques that require knowledge about a particular domain, such as image cropping and rotation.
How Does Mixup Help With Robustness and Generalization?
For robustness, we show that minimizing the Mixup loss corresponds to approximately minimizing an upper bound of the adversarial loss.
Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
Adding auxiliary losses to the main objective function is a general way of encoding biases that can help networks learn better representations.
Locally adaptive activation functions with slope recovery term for deep and physics-informed neural networks
Furthermore, the proposed methods with the slope recovery are shown to accelerate the training process.
GraphMix: Improved Training of GNNs for Semi-Supervised Learning
We present GraphMix, a regularization method for Graph Neural Network based semi-supervised object classification, whereby we propose to train a fully-connected network jointly with the graph neural network via parameter sharing and interpolation-based regularization.
Ranked #1 on
Node Classification
on Pubmed random partition
Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes
The theory developed in this paper only requires the practical degrees of over-parameterization unlike previous theories.
Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimization
The traditional approaches, such as (mini-batch) stochastic gradient descent (SGD), utilize an unbiased gradient estimator of the empirical average loss.
Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
Adversarial robustness has become a central goal in deep learning, both in the theory and the practice.
Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learning
Furthermore, as special cases of our general results, this article improves or complements several state-of-the-art theoretical results on deep neural networks, deep residual networks, and overparameterized deep neural networks with a unified proof technique and novel geometric insights.
Interpolation Consistency Training for Semi-Supervised Learning
We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for training Deep Neural Networks in the semi-supervised learning paradigm.
Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit
Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization.
Elimination of All Bad Local Minima in Deep Learning
At every local minimum of any deep neural network with these added neurons, the set of parameters of the original neural network (without added neurons) is guaranteed to be a global minimum of the original neural network.
Effect of Depth and Width on Local Minima in Deep Learning
In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature.
Depth with Nonlinearity Creates No Bad Local Minima in ResNets
In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are guaranteed to further improve via residual representations.
Generalization in Machine Learning via Analytical Learning Theory
This paper introduces a novel measure-theoretic theory for machine learning that does not require statistical assumptions.
Theory of Deep Learning III: explaining the non-overfitting puzzle
In this note, we show that the dynamics associated to gradient descent minimization of nonlinear networks is topologically equivalent, near the asymptotically stable minima of the empirical error, to linear gradient system in a quadratic potential with a degenerate (for square loss) or almost degenerate (for logistic or crossentropy loss) Hessian.
Generalization in Deep Learning
This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature.
Deep Semi-Random Features for Nonlinear Function Approximation
For deep models, with no unrealistic assumptions, we prove universal approximation ability, a lower bound on approximation error, a partial optimization guarantee, and a generalization bound.
Depth Creates No Bad Local Minima
In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces.
Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning.
Global Continuous Optimization with Error Bound and Fast Convergence
This paper proposes a new global optimization algorithm, called Locally Oriented Global Optimization (LOGO), to aim for both fast convergence in practice and finite-time error bound in theory.
Deep Learning without Poor Local Minima
In this paper, we prove a conjecture published in 1989 and also partially address an open problem announced at the Conference on Learning Theory (COLT) 2015.
Bounded Optimal Exploration in MDP
Within the framework of probably approximately correct Markov decision processes (PAC-MDP), much theoretical work has focused on methods to attain near optimality after a relatively long period of learning and exploration.
Bayesian Optimization with Exponential Convergence
This paper presents a Bayesian optimization method with exponential convergence without the need of auxiliary optimization and without the delta-cover sampling.
A Greedy Approximation of Bayesian Reinforcement Learning with Probably Optimistic Transition Model
This is a natural consequence of the fact that the proposed algorithm is greedier than the other algorithms.
