Search Results for author:
Found 214 papers, 102 papers with code
A Divide-and-Conquer Method for Sparse Inverse Covariance Estimation
We derive a bound on the distance of the approximate solution to the true solution.
Sparse Inverse Covariance Matrix Estimation Using Quadratic Approximation
The L1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix, or alternatively the underlying graph structure of a Gaussian Markov Random Field, from very limited samples.
A Divide-and-Conquer Solver for Kernel Support Vector Machines
We show theoretically that the support vectors identified by the subproblem solution are likely to be support vectors of the entire kernel SVM problem, provided that the problem is partitioned appropriately by kernel clustering.
Large Scale Distributed Sparse Precision Estimation
We consider the problem of sparse precision matrix estimation in high dimensions using the CLIME estimator, which has several desirable theoretical properties.
NOMAD: Non-locking, stOchastic Multi-machine algorithm for Asynchronous and Decentralized matrix completion
One of the key features of NOMAD is that the ownership of a variable is asynchronously transferred between processors in a decentralized fashion.
Distributed, Parallel, and Cluster Computing
BIG & QUIC: Sparse Inverse Covariance Estimation for a Million Variables
The l1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix even under high-dimensional settings.
PU Learning for Matrix Completion
For the first case, we propose a "shifted matrix completion" method that recovers M using only a subset of indices corresponding to ones, while for the second case, we propose a "biased matrix completion" method that recovers the (thresholded) binary matrix.
QUIC & DIRTY: A Quadratic Approximation Approach for Dirty Statistical Models
In this paper, we develop a family of algorithms for optimizing superposition-structured” or “dirty” statistical estimators for high-dimensional problems involving the minimization of the sum of a smooth loss function with a hybrid regularization.
Fast Prediction for Large-Scale Kernel Machines
Second, we provide a new theoretical analysis on bounding the error of the solution computed by using Nystr¨om kernel approximation method, and show that the error is related to the weighted kmeans objective function where the weights are given by the model computed from the original kernel.
Constant Nullspace Strong Convexity and Fast Convergence of Proximal Methods under High-Dimensional Settings
State of the art statistical estimators for high-dimensional problems take the form of regularized, and hence non-smooth, convex programs.
A Scalable Asynchronous Distributed Algorithm for Topic Modeling
Learning meaningful topic models with massive document collections which contain millions of documents and billions of tokens is challenging because of two reasons: First, one needs to deal with a large number of topics (typically in the order of thousands).
PASSCoDe: Parallel ASynchronous Stochastic dual Co-ordinate Descent
In this paper, we parallelize the SDCD algorithms in LIBLINEAR.
Matrix Completion with Noisy Side Information
Moreover, we study the effectof general features in theory, and show that by using our model, the sample complexity can still be lower than matrix completion as long as features are sufficiently informative.
Sparse Linear Programming via Primal and Dual Augmented Coordinate Descent
Over the past decades, Linear Programming (LP) has been widely used in different areas and considered as one of the mature technologies in numerical optimization.
Communication-Efficient Parallel Block Minimization for Kernel Machines
Kernel machines often yield superior predictive performance on various tasks; however, they suffer from severe computational challenges.
A Greedy Approach for Budgeted Maximum Inner Product Search
Maximum Inner Product Search (MIPS) is an important task in many machine learning applications such as the prediction phase of a low-rank matrix factorization model for a recommender system.
Asynchronous Parallel Greedy Coordinate Descent
n this paper, we propose and study an Asynchronous parallel Greedy Coordinate Descent (Asy-GCD) algorithm for minimizing a smooth function with bounded constraints.
Scalable Demand-Aware Recommendation
In particular for durable goods, time utility is a function of inter-purchase duration within product category because consumers are unlikely to purchase two items in the same category in close temporal succession.
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
On network configurations with low bandwidth or high latency, D-PSGD can be up to one order of magnitude faster than its well-optimized centralized counterparts.
GPU-acceleration for Large-scale Tree Boosting
In this paper, we present a novel massively parallel algorithm for accelerating the decision tree building procedure on GPUs (Graphics Processing Units), which is a crucial step in Gradient Boosted Decision Tree (GBDT) and random forests training.
Gradient Boosted Decision Trees for High Dimensional Sparse Output
In this paper, we study the gradient boosted decision trees (GBDT) when the output space is high dimensional and sparse.
ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
However, different from leveraging attack transferability from substitute models, we propose zeroth order optimization (ZOO) based attacks to directly estimate the gradients of the targeted DNN for generating adversarial examples.
An inexact subsampled proximal Newton-type method for large-scale machine learning
We propose a fast proximal Newton-type algorithm for minimizing regularized finite sums that returns an $\epsilon$-suboptimal point in $\tilde{\mathcal{O}}(d(n + \sqrt{\kappa d})\log(\frac{1}{\epsilon}))$ FLOPS, where $n$ is number of samples, $d$ is feature dimension, and $\kappa$ is the condition number.
EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples
Recent studies have highlighted the vulnerability of deep neural networks (DNNs) to adversarial examples - a visually indistinguishable adversarial image can easily be crafted to cause a well-trained model to misclassify.
ImageNet Training in Minutes
If we can make full use of the supercomputer for DNN training, we should be able to finish the 90-epoch ResNet-50 training in one minute.
Towards Robust Neural Networks via Random Self-ensemble
In this paper, we propose a new defense algorithm called Random Self-Ensemble (RSE) by combining two important concepts: {\bf randomness} and {\bf ensemble}.
Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
Our extensive experiments show that our algorithm can successfully craft visually-similar adversarial examples with randomly targeted captions or keywords, and the adversarial examples can be made highly transferable to other image captioning systems.
Better Generalization by Efficient Trust Region Method
Solving this subproblem is non-trivial---existing methods have only sub-linear convergence rate.
A comparison of second-order methods for deep convolutional neural networks
Despite many second-order methods have been proposed to train neural networks, most of the results were done on smaller single layer fully connected networks, so we still cannot conclude whether it's useful in training deep convolutional networks.
Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach
Our analysis yields a novel robustness metric called CLEVER, which is short for Cross Lipschitz Extreme Value for nEtwork Robustness.
History PCA: A New Algorithm for Streaming PCA
In this paper we propose a new algorithm for streaming principal component analysis.
SQL-Rank: A Listwise Approach to Collaborative Ranking
In this paper, we propose a listwise approach for constructing user-specific rankings in recommendation systems in a collaborative fashion.
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples
In this paper, we study the much more challenging problem of crafting adversarial examples for sequence-to-sequence (seq2seq) models, whose inputs are discrete text strings and outputs have an almost infinite number of possibilities.
Towards Fast Computation of Certified Robustness for ReLU Networks
Verifying the robustness property of a general Rectified Linear Unit (ReLU) network is an NP-complete problem [Katz, Barrett, Dill, Julian and Kochenderfer CAV17].
LearningWord Embeddings for Low-resource Languages by PU Learning
In such a situation, the co-occurrence matrix is sparse as the co-occurrences of many word pairs are unobserved.
GenAttack: Practical Black-box Attacks with Gradient-Free Optimization
Our experiments on different datasets (MNIST, CIFAR-10, and ImageNet) show that GenAttack can successfully generate visually imperceptible adversarial examples against state-of-the-art image recognition models with orders of magnitude fewer queries than previous approaches.
AutoZOOM: Autoencoder-based Zeroth Order Optimization Method for Attacking Black-box Neural Networks
Recent studies have shown that adversarial examples in state-of-the-art image classifiers trained by deep neural networks (DNN) can be easily generated when the target model is transparent to an attacker, known as the white-box setting.
Stochastic Zeroth-order Optimization via Variance Reduction method
However, due to the variance in the search direction, the convergence rates and query complexities of existing methods suffer from a factor of $d$, where $d$ is the problem dimension.
Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data
We present a probabilistic framework for studying adversarial attacks on discrete data.
Learning Word Embeddings for Low-Resource Languages by PU Learning
In such a situation, the co-occurrence matrix is sparse as the co-occurrences of many word pairs are unobserved.
GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking
Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses.
Extreme Learning to Rank via Low Rank Assumption
We consider the setting where we wish to perform ranking for hundreds of thousands of users which is common in recommender systems and web search ranking.
Query-Efficient Hard-label Black-box Attack:An Optimization-based Approach
We study the problem of attacking a machine learning model in the hard-label black-box setting, where no model information is revealed except that the attacker can make queries to probe the corresponding hard-label decisions.
Rob-GAN: Generator, Discriminator, and Adversarial Attacker
Adversarial training is the technique used to improve the robustness of discriminator by combining adversarial attacker and discriminator in the training phase.
Fast Variance Reduction Method with Stochastic Batch Size
In this paper we study a family of variance reduction methods with randomized batch size---at each step, the algorithm first randomly chooses the batch size and then selects a batch of samples to conduct a variance-reduced stochastic update.
RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
Data parallelism has become a dominant method to scale Deep Neural Network (DNN) training across multiple nodes.
Stochastically Controlled Stochastic Gradient for the Convex and Non-convex Composition problem
In this paper, we consider the convex and non-convex composition problem with the structure $\frac{1}{n}\sum\nolimits_{i = 1}^n {{F_i}( {G( x )} )}$, where $G( x )=\frac{1}{n}\sum\nolimits_{j = 1}^n {{G_j}( x )} $ is the inner function, and $F_i(\cdot)$ is the outer function.
Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient
Trust region and cubic regularization methods have demonstrated good performance in small scale non-convex optimization, showing the ability to escape from saddle points.
Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Network
Instead, we model randomness under the framework of Bayesian Neural Network (BNN) to formally learn the posterior distribution of models in a scalable way.
On Extensions of CLEVER: A Neural Network Robustness Evaluation Algorithm
We apply extreme value theory on the new formal robustness guarantee and the estimated robustness is called second-order CLEVER score.
Attack Graph Convolutional Networks by Adding Fake Nodes
In this paper, we propose a new type of "fake node attacks" to attack GCNs by adding malicious fake nodes.
RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications
The Jacobian matrix (or the gradient for single-output networks) is directly related to many important properties of neural networks, such as the function landscape, stationary points, (local) Lipschitz constants and robustness to adversarial attacks.
Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks
The algorithm achieves an order of magnitude faster inference than the original softmax layer for predicting top-$k$ words in various tasks such as beam search in machine translation or next words prediction.
Efficient Neural Network Robustness Certification with General Activation Functions
Finding minimum distortion of adversarial examples and thus certifying robustness in neural network classifiers for given data points is known to be a challenging problem.
Block-wise Partitioning for Extreme Multi-label Classification
Extreme multi-label classification aims to learn a classifier that annotates an instance with a relevant subset of labels from an extremely large label set.
Optimal Transport Classifier: Defending Against Adversarial Attacks by Regularized Deep Embedding
Recent studies have demonstrated the vulnerability of deep convolutional neural networks against adversarial examples.
Learning from Group Comparisons: Exploiting Higher Order Interactions
We study the problem of learning from group comparisons, with applications in predicting outcomes of sports and online games.
The Limitations of Adversarial Training and the Blind-Spot Attack
In our paper, we shed some lights on the practicality and the hardness of adversarial training by showing that the effectiveness (robustness on test set) of adversarial training has a strong correlation with the distance between a test point and the manifold of training data embedded by the network.
Large-Batch Training for LSTM and Beyond
LEGW enables Sqrt Scaling scheme to be useful in practice and as a result we achieve much better results than the Linear Scaling learning rate scheme.
A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks
This framework works for neural networks with diverse architectures and nonlinearities and covers both primal and dual views of robustness verification.
Robust Decision Trees Against Adversarial Examples
Although adversarial examples and model robustness have been extensively studied in the context of linear models and neural networks, research on this issue in tree-based models and how to make tree-based models robust against adversarial examples is still limited.
Efficient Contextual Representation Learning Without Softmax Layer
Our framework reduces the time spent on the output layer to a negligible level, eliminates almost all the trainable parameters of the softmax layer and performs language modeling without truncating the vocabulary.
Efficient Contextual Representation Learning With Continuous Outputs
Contextual representation models have achieved great success in improving various downstream natural language processing tasks.
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.
Ranked #11 on
Question Answering
on SQuAD1.1 dev
(F1 metric)
Evaluating Robustness of Deep Image Super-Resolution against Adversarial Attacks
Single-image super-resolution aims to generate a high-resolution version of a low-resolution image, which serves as an essential component in many computer vision applications.
Query-Efficient Hard-label Black-box Attack: An Optimization-based Approach
We study the problem of attacking machine learning models in the hard-label black-box setting, where no model information is revealed except that the attacker can make queries to probe the corresponding hard-label decisions.
Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
Furthermore, Cluster-GCN allows us to train much deeper GCN without much time and memory overhead, which leads to improved prediction accuracy---using a 5-layer Cluster-GCN, we achieve state-of-the-art test F1 score 99. 36 on the PPI dataset, while the previous best result was 98. 71 by [16].
Ranked #1 on
Node Classification
on Amazon2M
Stochastic Shared Embeddings: Data-driven Regularization of Embedding Layers
We find that when used along with widely-used regularization methods such as weight decay and dropout, our proposed SSE can further reduce overfitting, which often leads to more favorable generalization results.
Graph DNA: Deep Neighborhood Aware Graph Encoding for Collaborative Filtering
In this paper, we propose using Graph DNA, a novel Deep Neighborhood Aware graph encoding algorithm, for exploiting deeper neighborhood information.
Evaluating and Enhancing the Robustness of Dialogue Systems: A Case Study on a Negotiation Agent
Moreover, we show that with the adversarial training, we are able to improve the robustness of negotiation agents by 1. 5 points on average against all our attacks.
Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection.
ML-LOO: Detecting Adversarial Examples with Feature Attribution
Furthermore, we extend our method to include multi-layer feature attributions in order to tackle the attacks with mixed confidence levels.
Robustness Verification of Tree-based Models
We show that there is a simple linear time algorithm for verifying a single tree, and for tree ensembles, the verification problem can be cast as a max-clique problem on a multi-partite graph with bounded boxicity.
Evaluating the Robustness of Nearest Neighbor Classifiers: A Primal-Dual Perspective
Furthermore, we show that dual solutions for these QP problems could give us a valid lower bound of the adversarial perturbation that can be used for formal robustness verification, giving us a nice view of attack/verification for NN models.
Towards Stable and Efficient Training of Verifiably Robust Neural Networks
In this paper, we propose a new certified adversarial training method, CROWN-IBP, by combining the fast IBP bounds in a forward bounding pass and a tight linear relaxation based bound, CROWN, in a backward bounding pass.
Convergence of Adversarial Training in Overparametrized Neural Networks
Neural networks are vulnerable to adversarial examples, i. e. inputs that are imperceptibly perturbed from natural data and yet incorrectly classified by the network.
Efficient Neural Interaction Function Search for Collaborative Filtering
Motivated by the recent success of automated machine learning (AutoML), we propose in this paper the search for simple neural interaction functions (SIF) in CF.
On the Robustness of Self-Attentive Models
This work examines the robustness of self-attentive neural networks against adversarial input perturbations.
VisualBERT: A Simple and Performant Baseline for Vision and Language
We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks.
Ranked #1 on
Visual Reasoning
on NLVR
Temporal Collaborative Ranking Via Personalized Transformer
Recent advances in deep learning, especially the discovery of various attention mechanisms and newer architectures in addition to widely used RNN and CNN in natural language processing, have allowed us to make better use of the temporal ordering of items that each user has engaged with.
Natural Adversarial Sentence Generation with Gradient-based Perturbation
This work proposes a novel algorithm to generate natural language adversarial input for text classification models, in order to investigate the robustness of these models.
BOSH: An Efficient Meta Algorithm for Decision-based Attacks
Adversarial example generation becomes a viable method for evaluating the robustness of a machine learning model.
Sign-OPT: A Query-Efficient Hard-label Adversarial Attack
We study the most practical problem setup for evaluating adversarial robustness of a machine learning system with limited access: the hard-label black-box attack setting for generating adversarial examples, where limited model queries are allowed and only the decision is provided to a queried data input.
LEARNING TO LEARN WITH BETTER CONVERGENCE
We consider the learning to learn problem, where the goal is to leverage deeplearning models to automatically learn (iterative) optimization algorithms for training machine learning models.
Defending Against Adversarial Examples by Regularized Deep Embedding
Recent studies have demonstrated the vulnerability of deep convolutional neural networks against adversarial examples.
SPROUT: Self-Progressing Robust Training
Enhancing model robustness under new and even adversarial environments is a crucial milestone toward building trustworthy and reliable machine learning systems.
SSE-PT: Sequential Recommendation Via Personalized Transformer
Recent advances in deep learning, especially the discovery of various attention mechanisms and newer architectures in addition to widely used RNN and CNN in natural language processing, have allowed for better use of the temporal ordering of items that each user has engaged with.
Ranked #1 on
Recommendation Systems
on MovieLens 1M
(nDCG@10 metric)
Stochastically Controlled Compositional Gradient for the Composition problem
The strategy is also accompanied by a mini-batch version of the proposed method that improves query complexity with respect to the size of the mini-batch.
Elastic-InfoGAN: Unsupervised Disentangled Representation Learning in Class-Imbalanced Data
We propose a novel unsupervised generative model that learns to disentangle object identity from other low-level aspects in class-imbalanced data.
Learning to Learn by Zeroth-Order Oracle
In the learning to learn (L2L) framework, we cast the design of optimization algorithms as a machine learning problem and use deep neural networks to learn the update rules.
A Unified Framework for Data Poisoning Attack to Graph-based Semi-supervised Learning
In this paper, we proposed a general framework for data poisoning attacks to graph-based semi-supervised learning (G-SSL).
Enhancing Certifiable Robustness via a Deep Model Ensemble
We propose an algorithm to enhance certified robustness of a deep model ensemble by optimally weighting each base model.
MulCode: A Multiplicative Multi-way Model for Compressing Neural Language Model
For example, input embedding and Softmax matrices in IWSLT-2014 German-to-English data set account for more than 80{\%} of the total model parameters.
GraphDefense: Towards Robust Graph Convolutional Networks
Inspired by the previous works on adversarial defense for deep neural networks, and especially adversarial training algorithm, we propose a method called GraphDefense to defend against the adversarial perturbations.
Overcoming Catastrophic Forgetting by Generative Regularization
In this paper, we propose a new method to overcome catastrophic forgetting by adding generative regularization to Bayesian inference framework.
MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly.
Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.
Multiscale Non-stationary Stochastic Bandits
Classic contextual bandit algorithms for linear models, such as LinUCB, assume that the reward distribution for an arm is modeled by a stationary linear regression.
Robustness Verification for Transformers
Robustness verification that aims to formally certify the prediction behavior of neural networks has become an important tool for understanding model behavior and obtaining safety guarantees.
CAT: Customized Adversarial Training for Improved Robustness
Adversarial training has become one of the most effective methods for improving robustness of neural networks.
Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond
Linear relaxation based perturbation analysis (LiRPA) for neural networks, which computes provable linear bounds of output neurons given a certain amount of input perturbation, has become a core component in robustness verification and certified defense.
Learning to Encode Position for Transformer with Continuous Dynamical Model
The main reason is that position information among input units is not inherently encoded, i. e., the models are permutation equivalent; this problem justifies why all of the existing models are accompanied by a sinusoidal encoding/embedding layer at the input.
Ranked #5 on
Semantic Textual Similarity
on MRPC
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
Several works have shown this vulnerability via adversarial attacks, but existing approaches on improving the robustness of DRL under this setting have limited success and lack for theoretical principles.
Improved Adversarial Training via Learned Optimizer
Adversarial attack has recently become a tremendous threat to deep learning models.
Spanning Attack: Reinforce Black-box Attacks with Unlabeled Data
By constraining adversarial perturbations in a low-dimensional subspace via spanning an auxiliary unlabeled dataset, the spanning attack significantly improves the query efficiency of a wide variety of existing black-box attacks.
Evaluations and Methods for Explanation through Robustness Analysis
In this paper, we establish a novel set of evaluation criteria for such feature based explanations by robustness analysis.
An Efficient Algorithm For Generalized Linear Bandit: Online Stochastic Gradient Descent and Thompson Sampling
A natural way to resolve this problem is to apply online stochastic gradient descent (SGD) so that the per-step time and memory complexity can be reduced to constant with respect to $t$, but a contextual bandit policy based on online SGD updates that balances exploration and exploitation has remained elusive.
Provably Robust Metric Learning
Metric learning is an important family of algorithms for classification and similarity search, but the robustness of learned metrics against small adversarial perturbations is less studied.
The Limit of the Batch Size
For the first time we scale the batch size on ImageNet to at least a magnitude larger than all previous work, and provide detailed studies on the performance of many state-of-the-art optimization schemes under this setting.
DrNAS: Dirichlet Neural Architecture Search
This paper proposes a novel differentiable architecture search method by formulating it into a distribution learning problem.
Defense against Adversarial Attacks in NLP via Dirichlet Neighborhood Ensemble
Despite neural networks have achieved prominent performance on many natural language processing (NLP) tasks, they are vulnerable to adversarial examples.
What Does BERT with Vision Look At?
Pre-trained visually grounded language models such as ViLBERT, LXMERT, and UNITER have achieved significant performance improvement on vision-and-language tasks but what they learn during pre-training remains unclear.
Evaluating and Enhancing the Robustness of Neural Network-based Dependency Parsing Models with Adversarial Examples
Despite achieving prominent performance on many important tasks, it has been reported that neural networks are vulnerable to adversarial examples.
Multi-Stage Influence Function
With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task.
Improving the Speed and Quality of GAN by Adversarial Training
Generative adversarial networks (GAN) have shown remarkable results in image generation tasks.
On $\ell_p$-norm Robustness of Ensemble Stumps and Trees
In this paper, we study the problem of robustness verification and certified defense with respect to general $\ell_p$ norm perturbations for ensemble decision stumps and trees.
MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation
Knowledge Distillation (KD) has been one of the most popu-lar methods to learn a compact model.
How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers
For end-to-end efficiency, unlike previous work that assumes random hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy.
An Efficient Adversarial Attack for Tree Ensembles
We study the problem of efficient adversarial attacks on tree based ensembles such as gradient boosting decision trees (GBDTs) and random forests (RFs).
On the Transferability of Adversarial Attacksagainst Neural Text Classifier
Based on these studies, we propose a genetic algorithm to find an ensemble of models that can be used to induce adversarial examples to fool almost all existing models.
Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers
Formal verification of neural networks (NNs) is a challenging and important problem.
Voting based ensemble improves robustness of defensive models
Several previous attempts tackled this problem by ensembling the soft-label prediction and have been proved vulnerable based on the latest attack methods.
Learning to Stop: Dynamic Simulation Monte-Carlo Tree Search
This implies that a significant amount of resources can be saved if we are able to stop the searching earlier when we are confident with the current searching result.
Self-Progressing Robust Training
Enhancing model robustness under new and even adversarial environments is a crucial milestone toward building trustworthy machine learning systems.
Learning to Learn with Smooth Regularization
Recent decades have witnessed great prosperity of deep learning in tackling various problems such as classification and decision making.
Towards Robustness of Deep Neural Networks via Regularization
Recent studies have demonstrated the vulnerability of deep neural networks against adversarial examples.
Learning Contextual Perturbation Budgets for Training Robust Neural Networks
We also demonstrate that the perturbation budget generator can produce semantically-meaningful budgets, which implies that the generator can capture contextual information and the sensitivity of different features in a given image.
Data-aware Low-Rank Compression for Large NLP Models
In this paper, we observe that the learned representation of each layer lies in a low-dimensional space.
Adversarial Masking: Towards Understanding Robustness Trade-off for Generalization
By incorporating different feature maps after the masking, we can distill better features to help model generalization.
Robust Text CAPTCHAs Using Adversarial Examples
At the second stage, we design and apply a highly transferable adversarial attack for text CAPTCHAs to better obstruct CAPTCHA solvers.
Emotional EEG Classification using Connectivity Features and Convolutional Neural Networks
Convolutional neural networks (CNNs) are widely used to recognize the user's state through electroencephalography (EEG) signals.
Robust Reinforcement Learning on State Observations with Learned Optimal Adversary
We study the robustness of reinforcement learning (RL) with adversarially perturbed state observations, which aligns with the setting of many adversarial attacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise.
Local Critic Training for Model-Parallel Learning of Deep Neural Networks
We show that the proposed approach successfully decouples the update process of the layer groups for both convolutional neural networks (CNNs) and recurrent neural networks (RNNs).
Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Robustness Verification
Compared to the typically tightest but very costly semidefinite programming (SDP) based incomplete verifiers, we obtain higher verified accuracy with three orders of magnitudes less verification time.
Robust and Accurate Object Detection via Adversarial Learning
Data augmentation has become a de facto component for training high-performance deep image classifiers, but its potential is under-explored for object detection.
Ranked #17 on
Object Detection
on COCO-O
On the Adversarial Robustness of Vision Transformers
Following the success in advancing natural language processing and understanding, transformers are expected to bring revolutionary changes to computer vision.
Fast Certified Robust Training with Short Warmup
Despite that state-of-the-art (SOTA) methods including interval bound propagation (IBP) and CROWN-IBP have per-batch training complexity similar to standard neural network training, they usually use a long warmup schedule with hundreds or thousands epochs to reach SOTA performance and are thus still costly.
Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation
Our method is able to reveal the hidden model biases not directly shown in the test dataset.
On the Sensitivity and Stability of Model Interpretations in NLP
We propose two new criteria, sensitivity and stability, that provide complementary notions of faithfulness to the existed removal-based criteria.
2.5D Visual Relationship Detection
To enable progress on this task, we create a new dataset consisting of 220k human-annotated 2. 5D relationships among 512K objects from 11K images.
Deep Image Destruction: Vulnerability of Deep Image-to-Image Models against Adversarial Attacks
Recently, the vulnerability of deep image classification models to adversarial attacks has been investigated.
Adversarial Examples Detection with Bayesian Neural Network
In this paper, we propose a new framework to detect adversarial examples motivated by the observations that random components can improve the smoothness of predictors and make it easier to simulate the output distribution of a deep neural network.
Balancing Robustness and Sensitivity using Feature Contrastive Learning
It is generally believed that robust training of extremely large networks is critical to their success in real-world applications.
Concurrent Adversarial Learning for Large-Batch Training
Current methods usually use extensive data augmentation to increase the batch size, but we found the performance gain with data augmentation decreases as batch size increases, and data augmentation will become insufficient after certain point.
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input.
Ranked #3 on
Efficient ViTs
on ImageNet-1K (With LV-ViT-S)
When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
Vision Transformers (ViTs) and MLPs signal further efforts on replacing hand-wired features or inductive biases with general-purpose neural architectures.
Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding
Attentional mechanisms are order-invariant.
Robust Stochastic Linear Contextual Bandits Under Adversarial Attacks
We provide theoretical guarantees for our proposed algorithm and show by experiments that our proposed algorithm improves the robustness against various kinds of popular attacks.
Syndicated Bandits: A Framework for Auto Tuning Hyper-parameters in Contextual Bandit Algorithms
To tackle this problem, we first propose a two-layer bandit structure for auto tuning the exploration parameter and further generalize it to the Syndicated Bandits framework which can learn multiple hyper-parameters dynamically in contextual bandit environment.
Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Neural Network Robustness Verification
We develop $\beta$-CROWN, a new bound propagation based method that can fully encode neuron split constraints in branch-and-bound (BaB) based complete verification via optimizable parameters $\beta$.
Label Disentanglement in Partition-based Extreme Multilabel Classification
Partition-based methods are increasingly-used in extreme multi-label classification (XMC) problems due to their scalability to large output spaces (e. g., millions or more).
Defense against Synonym Substitution-based Adversarial Attacks via Dirichlet Neighborhood Ensemble
Although deep neural networks have achieved prominent performance on many NLP tasks, they are vulnerable to adversarial examples.
Rethinking Architecture Selection in Differentiable NAS
Differentiable Neural Architecture Search is one of the most popular Neural Architecture Search (NAS) methods for its search efficiency and simplicity, accomplished by jointly optimizing the model weight and architecture parameters in a weight-sharing supernet via gradient-based algorithms.
RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection
In particular, we propose to generate random layouts of a scene by making use of the objects in the synthetic CAD dataset and learn the 3D scene representation by applying object-level contrastive learning on two random scenes generated from the same set of synthetic objects.
RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving
Predictor-based algorithms have achieved remarkable performance in the Neural Architecture Search (NAS) tasks.
Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution
Recent studies have shown that deep neural networks are vulnerable to intentionally crafted adversarial examples, and various methods have been proposed to defend against adversarial word-substitution attacks for neural NLP models.
Training Meta-Surrogate Model for Transferable Adversarial Attack
In this paper, we tackle this problem from a novel angle -- instead of using the original surrogate models, can we obtain a Meta-Surrogate Model (MSM) such that attacks to this model can be easier transferred to other models?
A Branch and Bound Framework for Stronger Adversarial Attacks of ReLU Networks
In this work, we formulate an adversarial attack using a branch-and-bound (BaB) procedure on ReLU neural networks and search adversarial examples in the activation space corresponding to binary variables in a mixed integer programming (MIP) formulation.
FastEnsemble: Benchmarking and Accelerating Ensemble-based Uncertainty Estimation for Image-to-Image Translation
In addition, we introduce FastEnsemble, a fast ensemble method which only requires less than $8\%$ of the full-ensemble training time to generate a new ensemble member.
Learning to Schedule Learning rate with Graph Neural Networks
By constructing a directed graph for the underlying neural network of the target problem, GNS encodes current dynamics with a graph message passing network and trains an agent to control the learning rate accordingly via reinforcement learning.
Sharpness-Aware Minimization in Large-Batch Training: Training Vision Transformer In Minutes
Large-batch training is an important direction for distributed machine learning, which can improve the utilization of large-scale clusters and therefore accelerate the training process.
Adversarial Attack across Datasets
In this paper, we define a Generalized Transferable Attack (GTA) problem where the attacker doesn't know this information and is acquired to attack any randomly encountered images that may come from unknown datasets.
How and When Adversarial Robustness Transfers in Knowledge Distillation?
Our comprehensive analysis shows several novel insights that (1) With KDIGA, students can preserve or even exceed the adversarial robustness of the teacher model, even when their models have fundamentally different architectures; (2) KDIGA enables robustness to transfer to pre-trained students, such as KD from an adversarially trained ResNet to a pre-trained ViT, without loss of clean accuracy; and (3) Our derived local linearity bounds for characterizing adversarial robustness in KD are consistent with the empirical results.
Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction
We also provide a theoretical analysis that justifies the use of XMC over link prediction and motivates integrating XR-Transformers, a powerful method for solving XMC problems, into the GIANT framework.
Ranked #2 on
Node Property Prediction
on ogbn-papers100M
Can Vision Transformers Perform Convolution?
Several recent studies have demonstrated that attention-based networks, such as Vision Transformer (ViT), can outperform Convolutional Neural Networks (CNNs) on several computer vision tasks without using convolutional layers.
A Review of Adversarial Attack and Defense for Classification Methods
Despite the efficiency and scalability of machine learning systems, recent studies have demonstrated that many classification methods, especially deep neural networks (DNNs), are vulnerable to adversarial examples; i. e., examples that are carefully crafted to fool a well-trained classification model while being indistinguishable from natural data to human.
DRONE: Data-aware Low-rank Compression for Large NLP Models
In addition to compressing standard models, out method can also be used on distilled BERT models to further improve compression rate.
Temporal Shuffling for Defending Deep Action Recognition Models against Adversarial Attacks
Recently, video-based action recognition methods using convolutional neural networks (CNNs) achieve remarkable recognition performance.
Extreme Zero-Shot Learning for Extreme Text Classification
To learn the semantic embeddings of instances and labels with raw text, we propose to pre-train Transformer-based encoders with self-supervised contrastive losses.
Multi Label Text Classification
Multi-Label Text Classification
+2
Towards Efficient and Scalable Sharpness-Aware Minimization
Recently, Sharpness-Aware Minimization (SAM), which connects the geometry of the loss landscape and generalization, has demonstrated significant performance boosts on training large-scale models such as vision transformers.
On the Convergence of Certified Robust Training with Interval Bound Propagation
Interval Bound Propagation (IBP) is so far the base of state-of-the-art methods for training neural networks with certifiable robustness guarantees when potential adversarial perturbations present, while the convergence of IBP training remains unknown in existing literature.
Generalizing Few-Shot NAS with Gradient Matching
Efficient performance estimation of architectures drawn from large search spaces is essential to Neural Architecture Search.
Improving the Adversarial Robustness of NLP Models by Information Bottleneck
Existing studies have demonstrated that adversarial examples can be directly attributed to the presence of non-robust features, which are highly predictive, but can be easily manipulated by adversaries to fool NLP models.
FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning
Federated learning~(FL) has recently attracted increasing attention from academia and industry, with the ultimate goal of achieving collaborative training under privacy and communication constraints.
DC-BENCH: Dataset Condensation Benchmark
Dataset Condensation is a newly emerging technique aiming at learning a tiny dataset that captures the rich information encoded in the original dataset.
General Cutting Planes for Bound-Propagation-Based Neural Network Verification
Our generalized bound propagation method, GCP-CROWN, opens up the opportunity to apply general cutting plane methods for neural network verification while benefiting from the efficiency and GPU acceleration of bound propagation methods.
Concept Gradient: Concept-based Interpretation Without Linear Assumption
We showed that for a general (potentially non-linear) concept, we can mathematically evaluate how a small change of concept affecting the model's prediction, which leads to an extension of gradient-based interpretation to the concept space.
Efficient Non-Parametric Optimizer Search for Diverse Tasks
Efficient and automated design of optimizers plays a crucial role in full-stack AutoML systems.
Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagation
In this paper, we develop an efficient framework for computing the $\ell_\infty$ local Lipschitz constant of a neural network by tightly upper bounding the norm of Clarke Jacobian via linear bound propagation.
Watermarking Pre-trained Language Models with Backdooring
Large pre-trained language models (PLMs) have proven to be a crucial component of modern natural language processing systems.
ELIAS: End-to-End Learning to Index and Search in Large Output Spaces
A popular approach for dealing with the large label space is to arrange the labels into a shallow tree-based index and then learn an ML model to efficiently search this index via beam search.
Uncertainty in Extreme Multi-label Classification
Uncertainty quantification is one of the most crucial tasks to obtain trustworthy and reliable machine learning models for decision making.
Reducing Training Sample Memorization in GANs by Training with Memorization Rejection
In this paper, we propose memorization rejection, a training scheme that rejects generated samples that are near-duplicates of training samples during training.
ADDMU: Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation
Finally, our analysis shows that the two types of uncertainty provided by \textbf{ADDMU} can be leveraged to characterize adversarial examples and identify the ones that contribute most to model's robustness in adversarial training.
Two-stage LLM Fine-tuning with Less Specialization and More Generalization
Pretrained large language models (LLMs) are general purpose problem solvers applicable to a diverse set of tasks with prompts.
Improving Adversarial Robustness to Sensitivity and Invariance Attacks with Deep Metric Learning
Intentionally crafted adversarial samples have effectively exploited weaknesses in deep neural networks.
Are AlphaZero-like Agents Robust to Adversarial Perturbations?
Given that the state space of Go is extremely large and a human player can play the game from any legal state, we ask whether adversarial states exist for Go AIs that may lead them to play surprisingly wrong actions.
Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory
The resulting algorithm sets new SOTA on ImageNet-1K: we can scale up to 50 IPCs (Image Per Class) on ImageNet-1K on a single GPU (all previous methods can only scale to 2 IPCs on ImageNet-1K), leading to the best accuracy (only 5. 9% accuracy drop against full dataset training) while utilizing only 4. 2% of the number of data points - an 18. 2% absolute gain over prior SOTA.
Symbolic Discovery of Optimization Algorithms
On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2. 3x.
Online Continuous Hyperparameter Optimization for Generalized Linear Contextual Bandits
In stochastic contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience to minimize the cumulative regret.
Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding
Existing video-language pre-training methods primarily focus on instance-level alignment between video clips and captions via global contrastive learning but neglect rich fine-grained local information in both videos and text, which is of importance to downstream tasks requiring temporal localization and semantic reasoning.
Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectories
With extensive experimental evaluation, we show the prevalence of \emph{generalization failure} on controllable states from stranger agents.
PINA: Leveraging Side Information in eXtreme Multi-label Classification via Predicted Instance Neighborhood Aggregation
Unlike most existing XMC frameworks that treat labels and input instances as featureless indicators and independent entries, PINA extracts information from the label metadata and the correlations among training instances.
Red Teaming Language Model Detectors with Language Models
The prevalence and strong capability of large language models (LLMs) present significant safety and ethical risks if exploited by malicious users.
Representer Point Selection for Explaining Regularized High-dimensional Models
We introduce a novel class of sample-based explanations we term high-dimensional representers, that can be used to explain the predictions of a regularized high-dimensional model in terms of importance weights for each of the training samples.
MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering
Few-shot question answering (QA) aims at achieving satisfactory results on machine question answering when only a few training samples are available.
Randomized Benchmarking of Local Zeroth-Order Optimizers for Variational Quantum Systems
To address this concern, we compare the performance of classical optimizers across a series of partially-randomized tasks to more broadly sample the space of quantum optimization problems.
Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function
In this paper, we explore a family of stochastic TR and ARC methods that can simultaneously provide inexact computations of the Hessian matrix, gradient, and function values.
Automatic Engineering of Long Prompts
Large language models (LLMs) have demonstrated remarkable capabilities in solving complex open-domain tasks, guided by comprehensive instructions and demonstrations provided in the form of prompts.
Improving the Generation Quality of Watermarked Large Language Models via Word Importance Scoring
In this work, we propose to improve the quality of texts generated by a watermarked language model by Watermarking with Importance Scoring (WIS).
