Search Results for author:
Found 85 papers, 33 papers with code
Mix-n-Match : Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning
This paper studies the problem of post-hoc calibration of machine learning classifiers.
Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis
In Phase 2, the attention matrices and the MLP evolve jointly to enlarge the classification margin and reduce the loss to a near minimum value.
Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion
Speculative decoding has emerged as a widely adopted method to accelerate large language model inference without sacrificing the quality of the model outputs.
ELFS: Enhancing Label-Free Coreset Selection via Clustering-based Pseudo-Labeling
For instance, at a 90% pruning rate, ELFS surpasses the best-performing baseline by 5. 3% on CIFAR10 and 7. 1% on CIFAR100.
Low-rank finetuning for LLMs: A fairness perspective
Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models (LLMs) due to their reduced computational and memory requirements.
Transformers Can Do Arithmetic with the Right Embeddings
The poor performance of transformers on arithmetic tasks seems to stem in large part from their inability to keep track of the exact position of each digit inside of a large span of digits.
SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning
In this work, we shed light on the significance of optimizer selection in LLM unlearning for the first time, establishing a clear connection between second-order optimization and influence unlearning (a classical approach using influence functions to update the model for data influence removal).
Introducing v0.5 of the AI Safety Benchmark from MLCommons
We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0. 5 benchmark.
End-to-End Mesh Optimization of a Hybrid Deep Learning Black-Box PDE Solver
In this study, we explore the feasibility of end-to-end training of a hybrid model with a black-box PDE solver and a deep learning model for fluid flow prediction.
Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies
However, our scaling laws also predict robustness slowly grows then plateaus at $90$%: dwarfing our new SOTA by scaling is impractical, and perfect robustness is impossible.
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
While state-of-the-art (SoTA) compression methods boast impressive advancements in preserving benign task performance, the potential risks of compression in terms of safety and trustworthiness have been largely neglected.
GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
We further characterize the game-theoretic properties of LLMs, such as equilibrium and Pareto Efficiency in repeated games.
TrustLLM: Trustworthiness in Large Language Models
This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions.
Scaling Compute Is Not All You Need for Adversarial Robustness
Finally, we make our benchmarking framework (built on top of \texttt{timm}~\citep{rw2019timm}) publicly available to facilitate future analysis in efficient robust deep learning.
When Bio-Inspired Computing meets Deep Learning: Low-Latency, Accurate, & Energy-Efficient Spiking Neural Networks from Artificial Neural Networks
However, advanced ANN-to-SNN conversion approaches demonstrate that for lossless conversion, the number of SNN time steps must equal the number of quantization steps in the ANN activation function.
Pursing the Sparse Limitation of Spiking Deep Learning Structures
It posits that within dense neural networks, there exist winning tickets or subnetworks that are sparser but do not compromise performance.
Leveraging Hierarchical Feature Sharing for Efficient Dataset Condensation
To better align DC with this hierarchical nature and encourage more efficient information sharing inside data containers, we propose a novel data parameterization architecture, Hierarchical Memory Network (HMN).
NEFTune: Noisy Embeddings Improve Instruction Finetuning
We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation.
DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training
Our extensive experiments show that DeepZero achieves state-of-the-art (SOTA) accuracy on ResNet-20 trained on CIFAR-10, approaching FO training performance for the first time.
Neural Image Compression: Generalization, Robustness, and Spectral Biases
Our benchmarks, spectral inspection tools, and findings provide a crucial bridge to the real-world adoption of NIC.
On the Fly Neural Style Smoothing for Risk-Averse Domain Generalization
To enable risk-averse predictions from a DG classifier, we propose a novel inference procedure, Test-Time Neural Style Smoothing (TT-NSS), that uses a "style-smoothed" version of the DG classifier for prediction at test time.
Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models
Large Language Models (LLMs) show promising results in language generation and instruction following but frequently "hallucinate", making their outputs less reliable.
Less is More: Data Pruning for Faster Adversarial Training
Deep neural networks (DNNs) are sensitive to adversarial examples, resulting in fragile and unreliable performance in the real world.
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities
To address this problem, there has been a great deal of research on *algorithmically-efficient deep learning*, which seeks to reduce training costs not at the hardware or implementation level, but through changes in the semantics of the training program.
Efficient Multi-Prize Lottery Tickets: Enhanced Accuracy, Training, and Inference Speed
Recently, Diffenderfer and Kailkhura proposed a new paradigm for learning compact yet highly accurate binary neural networks simply by pruning and quantizing randomly weighted full precision neural networks.
Models Out of Line: A Fourier Lens on Distribution Shift Robustness
However, there still is no clear understanding of the conditions on OOD data and model properties that are required to observe effective robustness.
On Certifying and Improving Generalization to Unseen Domains
This highlights that the performance of DG methods on a few benchmark datasets may not be representative of their performance on unseen domains in the wild.
Improving Diversity with Adversarially Learned Transformations for Domain Generalization
To be successful in single source domain generalization, maximizing diversity of synthesized domains has emerged as one of the most effective strategies.
Zeroth-Order SciML: Non-intrusive Integration of Scientific Software with Deep Learning
Existing knowledge integration approaches are limited to using differentiable knowledge source to be compatible with first-order DL training paradigm.
Representing Polymers as Periodic Graphs with Learned Descriptors for Accurate Polymer Property Predictions
Our results illustrate how the incorporation of chemical intuition through directly encoding periodicity into our polymer graph representation leads to a considerable improvement in the accuracy and reliability of polymer property predictions.
A Fast and Convergent Proximal Algorithm for Regularized Nonconvex and Nonsmooth Bi-level Optimization
In this work, we study a proximal gradient-type algorithm that adopts the approximate implicit differentiation (AID) scheme for nonconvex bi-level optimization with possibly nonconvex and nonsmooth regularizers.
Benchmarking Test-Time Unsupervised Deep Neural Network Adaptation on Edge Devices
To improve robustness of DNNs, they must be able to update themselves to enhance their prediction accuracy.
COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks
We leverage COPA to certify three RL environments trained with different algorithms and conclude: (1) The proposed robust aggregation protocols such as temporal aggregation can significantly improve the certifications; (2) Our certification for both per-state action stability and cumulative reward bound are efficient and tight; (3) The certification for different training algorithms and environments are different, implying their intrinsic robustness properties.
Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions
Deep neural networks on 3D point cloud data have been widely used in the real world, especially in safety-critical applications.
Ranked #1 on
3D Point Cloud Data Augmentation
on ModelNet40-C
3D Point Cloud Classification
3D Point Cloud Data Augmentation
+2
Certified Adversarial Defenses Meet Out-of-Distribution Corruptions: Benchmarking Robustness and Simple Baselines
To alleviate this issue, we propose a novel data augmentation scheme, FourierMix, that produces augmentations to improve the spectral coverage of the training data.
I-PGD-AT: Efficient Adversarial Training via Imitating Iterative PGD Attack
Finally, to demonstrate the generality of I-PGD-AT, we integrate it into PGD adversarial training and show that it can even further improve the robustness.
On the Certified Robustness for Ensemble Models and Beyond
Thus, to explore the conditions that guarantee to provide certifiably robust ensemble ML models, we first prove that diversified gradient and large confidence margin are sufficient and necessary conditions for certifiably robust ensemble models under the model-smoothness assumption.
Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning
Unsupervised domain adaptation (UDA) enables cross-domain learning without target domain labels by transferring knowledge from a labeled source domain whose distribution differs from that of the target.
Reliable Graph Neural Network Explanations Through Adversarial Training
Graph neural network (GNN) explanations have largely been facilitated through post-hoc introspection.
On the Effectiveness of Poisoning against Unsupervised Domain Adaptation
However, the limited effect of poisoning is restricted to the setting where training and test data are from the same distribution.
A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
Successful adoption of deep learning (DL) in the wild requires models to be: (1) compact, (2) accurate, and (3) robust to distributional shifts.
Mixture of Robust Experts (MoRE):A Robust Denoising Method towards multiple perturbations
To tackle the susceptibility of deep neural networks to examples, the adversarial training has been proposed which provides a notion of robust through an inner maximization problem presenting the first-order embedded within the outer minimization of the training loss.
Certifiably-Robust Federated Adversarial Learning via Randomized Smoothing
Federated learning is an emerging data-private distributed learning framework, which, however, is vulnerable to adversarial attacks.
Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network
In this paper, we propose (and prove) a stronger Multi-Prize Lottery Ticket Hypothesis: A sufficiently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i. e., binary weights and/or activation) (prize 3).
Classification with Binary Neural Network
Classification with Binary Weight Network
+1
Robusta: Robust AutoML for Feature Selection via Reinforcement Learning
However, these AutoML pipelines only focus on improving the learning accuracy of benign samples while ignoring the ML model robustness under adversarial attacks.
Multi-Prize Lottery Ticket Hypothesis: Finding Generalizable and Efficient Binary Subnetworks in a Randomly Weighted Neural Network
A sufficiently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i. e., binary weights and/or activation) (prize 3).
Can Shape Structure Features Improve Model Robustness Under Diverse Adversarial Settings?
Specifically, EdgeNetRob and EdgeGANRob first explicitly extract shape structure features from a given image via an edge detection algorithm.
Attribute-Guided Adversarial Training for Robustness to Natural Perturbations
While this deviation may not be exactly known, its broad characterization is specified a priori, in terms of attributes.
Leveraging Uncertainty from Deep Learning for Trustworthy Materials Discovery Workflows
In this paper, we leverage predictive uncertainty of deep neural networks to answer challenging questions material scientists usually encounter in machine learning based materials applications workflows.
How Robust are Randomized Smoothing based Defenses to Data Poisoning?
Moreover, our attack is effective even when the victim trains the models from scratch using state-of-the-art robust training methods such as Gaussian data augmentation\cite{cohen2019certified}, MACER\cite{zhai2020macer}, and SmoothAdv\cite{salman2019provably} that achieve high certified adversarial robustness.
A Statistical Mechanics Framework for Task-Agnostic Sample Design in Machine Learning
Using this framework, we show that space-filling sample designs, such as blue noise and Poisson disk sampling, which optimize spectral properties, outperform random designs in terms of the generalization gap and characterize this gain in a closed-form.
FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling
We develop FedCluster--a novel federated learning framework with improved optimization efficiency, and investigate its theoretical convergence properties.
Probabilistic Neighbourhood Component Analysis: Sample Efficient Uncertainty Estimation in Deep Learning
In this work, we show that the uncertainty estimation capability of state-of-the-art BNNs and Deep Ensemble models degrades significantly when the amount of training data is small.
Explainable Deep Learning for Uncovering Actionable Scientific Insights for Materials Discovery and Design
The scientific community has been increasingly interested in harnessing the power of deep learning to solve various domain challenges.
Adversarial Mutual Information for Text Generation
Recent advances in maximizing mutual information (MI) between the source and target have demonstrated its effectiveness in text generation.
Actionable Attribution Maps for Scientific Machine Learning
The scientific community has been increasingly interested in harnessing the power of deep learning to solve various domain challenges.
A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning
Zeroth-order (ZO) optimization is a subset of gradient-free optimization that emerges in many signal processing and machine learning applications.
Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning
We show that none of the existing methods satisfy all three requirements, and demonstrate how Mix-n-Match calibration strategies (i. e., ensemble and composition) can help achieve remarkably better data-efficiency and expressive power while provably maintaining the classification accuracy of the original classifier.
Anomalous Example Detection in Deep Learning: A Survey
This survey tries to provide a structured and comprehensive overview of the research on anomaly detection for DL based applications.
Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond
Linear relaxation based perturbation analysis (LiRPA) for neural networks, which computes provable linear bounds of output neurons given a certain amount of input perturbation, has become a core component in robustness verification and certified defense.
TSS: Transformation-Specific Smoothing for Robustness Certification
Moreover, to the best of our knowledge, TSS is the first approach that achieves nontrivial certified robustness on the large-scale ImageNet dataset.
Towards an Efficient and General Framework of Robust Training for Graph Neural Networks
To overcome these limitations, we propose a general framework which leverages the greedy search algorithms and zeroth-order methods to obtain robust GNNs in a generic and an efficient manner.
MimicGAN: Robust Projection onto Image Manifolds with Corruption Mimicking
However, PGD is a brittle optimization technique that fails to identify the right projection (or latent vector) when the observation is corrupted, or perturbed even by a small amount.
Enabling Machine Learning-Ready HPC Ensembles with Merlin
With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data.
Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification?
Quantifying the importance of each training point to a learning task is a fundamental problem in machine learning and the estimated importance scores have been leveraged to guide a range of data workflows such as data summarization and domain adaption.
Deep Kernels with Probabilistic Embeddings for Small-Data Learning
Experiments on a variety of datasets show that our approach outperforms the state-of-the-art in GP kernel learning in both supervised and semi-supervised settings.
On the Design of Black-box Adversarial Examples by Leveraging Gradient-free Optimization and Operator Splitting Method
Robust machine learning is currently one of the most prominent topics which could potentially help shaping a future of advanced AI platforms that not only perform well in average cases but also in worst cases or adverse situations.
Generative Counterfactual Introspection for Explainable Deep Learning
In this work, we propose an introspection technique for deep neural networks that relies on a generative model to instigate salient editing of the input image for model interpretation.
G-PATE: Scalable Differentially Private Data Generator via Private Aggregation of Teacher Discriminators
In particular, we train a student data generator with an ensemble of teacher discriminators and propose a novel private gradient aggregation mechanism to ensure differential privacy on all information that flows from teacher discriminators to the student generator.
A Look at the Effect of Sample Design on Generalization through the Lens of Spectral Analysis
This paper provides a general framework to study the effect of sampling properties of training data on the generalization error of the learned machine learning (ML) models.
Reliable and Explainable Machine Learning Methods for Accelerated Material Discovery
We also propose a transfer learning technique and show that the performance loss due to models' simplicity can be overcome by exploiting correlations among different material properties.
MR-GAN: Manifold Regularized Generative Adversarial Networks
Despite the growing interest in generative adversarial networks (GANs), training GANs remains a challenging problem, both from a theoretical and a practical standpoint.
MimicGAN: Corruption-Mimicking for Blind Image Recovery & Adversarial Defense
Solving inverse problems continues to be a central challenge in computer vision.
Universal Decision-Based Black-Box Perturbations: Breaking Security-Through-Obscurity Defenses
We study the problem of finding a universal (image-agnostic) perturbation to fool machine learning (ML) classifiers (e. g., neural nets, decision tress) in the hard-label black-box setting.
Coverage-Based Designs Improve Sample Mining and Hyper-Parameter Optimization
Sampling one or more effective solutions from large search spaces is a recurring idea in machine learning, and sequential optimization has become a popular solution.
Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization
As application demands for zeroth-order (gradient-free) optimization accelerate, the need for variance reduced and faster converging approaches is also intensifying.
An Unsupervised Approach to Solving Inverse Problems using Generative Adversarial Networks
We solve this by making successive estimates on the model and the solution in an iterative fashion.
Human-Machine Inference Networks For Smart Decision Making: Opportunities and Challenges
The emerging paradigm of Human-Machine Inference Networks (HuMaINs) combines complementary cognitive strengths of humans and machines in an intelligent manner to tackle various inference tasks and achieves higher performance than either humans or machines by themselves.
A Spectral Approach for the Design of Experiments: Design, Analysis and Algorithms
Third, we propose an efficient estimator to evaluate the space-filling properties of sample designs in arbitrary dimensions and use it to develop an optimization framework to generate high quality space-filling designs.
Robust Decentralized Learning Using ADMM with Unreliable Agents
We also provide conditions on the erroneous updates for exact convergence to the optimal solution.
Robust Local Scaling using Conditional Quantiles of Graph Similarities
In this paper, we propose the use of quantile analysis to obtain local scale estimates for neighborhood graph construction.
Influential Node Detection in Implicit Social Networks using Multi-task Gaussian Copula Models
Influential node detection is a central research topic in social network analysis.
TreeView: Peeking into Deep Neural Networks Via Feature-Space Partitioning
In this paper, we take a step in the direction of tackling the problem of interpretability without compromising the model accuracy.
Universal Collaboration Strategies for Signal Detection: A Sparse Learning Approach
This paper considers the problem of high dimensional signal detection in a large distributed network whose nodes can collaborate with their one-hop neighboring nodes (spatial collaboration).
Consensus based Detection in the Presence of Data Falsification Attacks
This paper considers the problem of detection in distributed networks in the presence of data falsification (Byzantine) attacks.
