Search Results for author:
Found 95 papers, 30 papers with code
On Graphical Models via Univariate Exponential Family Distributions
Undirected graphical models, or Markov networks, are a popular class of statistical models, used in a wide variety of applications.
Sparse Inverse Covariance Matrix Estimation Using Quadratic Approximation
The L1-regularized Gaussian maximum likelihood estimator (MLE) has been shown to have strong statistical guarantees in recovering a sparse inverse covariance matrix, or alternatively the underlying graph structure of a Gaussian Markov Random Field, from very limited samples.
Proximal Quasi-Newton for Computationally Intensive L1-regularized M-estimators
We consider the class of optimization problems arising from computationally intensive L1-regularized M-estimators, where the function or gradient values are very expensive to compute.
A General Framework for Mixed Graphical Models
"Mixed Data" comprising a large number of heterogeneous variables (e. g. count, binary, continuous, skewed continuous, among other data types) are prevalent in varied areas such as genomics and proteomics, imaging genetics, national security, social networking, and Internet advertising.
On the Information Theoretic Limits of Learning Ising Models
We provide a general framework for computing lower-bounds on the sample complexity of recovering the underlying graphs of Ising models, given i. i. d samples.
Optimal Decision-Theoretic Classification Using Non-Decomposable Performance Metrics
We provide a general theoretical analysis of expected out-of-sample utility, also referred to as decision-theoretic classification, for non-decomposable binary classification metrics such as F-measure and Jaccard coefficient.
Vector-Space Markov Random Fields via Exponential Families
Specifically, VS-MRFs are the joint graphical model distributions where the node-conditional distributions belong to generic exponential families with general vector space domains.
Exponential Family Matrix Completion under Structural Constraints
We consider the matrix completion problem of recovering a structured matrix from noisy and partial measurements.
Square Root Graphical Models: Multivariate Generalizations of Univariate Exponential Families that Permit Positive Dependencies
With this motivation, we give an example of our model class derived from the univariate exponential distribution that allows for almost arbitrary positive and negative dependencies with only a mild condition on the parameter matrix---a condition akin to the positive definiteness of the Gaussian covariance matrix.
PD-Sparse : A Primal and Dual Sparse Approach to Extreme Multiclass and Multilabel Classification
In this work, we show that a margin-maximizing loss with l1 penalty, in case of Extreme Classification, yields extremely sparse solution both in primal and in dual without sacrificing the expressive power of predictor.
Generalized Root Models: Beyond Pairwise Graphical Models for Univariate Exponential Families
As in the recent work with square root graphical (SQR) models [Inouye et al. 2016]---which was restricted to pairwise dependencies---we give the conditions of the parameters that are needed for normalization using the radial conditionals similar to the pairwise case [Inouye et al. 2016].
Kernel Ridge Regression via Partitioning
In this paper, we investigate a divide and conquer approach to Kernel Ridge Regression (KRR).
A Review of Multivariate Distributions for Count Data Derived from the Poisson Distribution
The Poisson distribution has been widely studied and used for modeling univariate count-valued data.
Online Classification with Complex Metrics
The proposed framework is general, as it applies to both batch and online learning, and to both linear and non-linear models.
Doubly Greedy Primal-Dual Coordinate Descent for Sparse Empirical Risk Minimization
We consider the popular problem of sparse empirical risk minimization with linear predictors and a large number of both features and observations.
Latent Feature Lasso
The latent feature model (LFM), proposed in (Griffiths \& Ghahramani, 2005), but possibly with earlier origins, is a generalization of a mixture model, where each instance is generated not from a single latent class but from a combination of latent features.
Ordinal Graphical Models: A Tale of Two Approaches
While there have been some work on tractable approximations, these do not come with strong statistical guarantees, and moreover are relatively computationally expensive.
A Voting-Based System for Ethical Decision Making
We present a general approach to automating ethical decisions, drawing on machine learning and computational social choice.
Identifiability of Nonparametric Mixture Models and Bayes Optimal Clustering
Motivated by problems in data clustering, we establish general conditions under which families of nonparametric mixture models are identifiable, by introducing a novel framework involving clustering overfitted \emph{parametric} (i. e. misspecified) mixture models.
D2KE: From Distance to Kernel and Embedding
For many machine learning problem settings, particularly with structured inputs such as sequences or sets of objects, a distance measure between inputs can be specified more naturally than a feature representation.
Robust Estimation via Robust Gradient Estimation
We provide a new computationally-efficient class of estimators for risk minimization.
DAGs with NO TEARS: Continuous Optimization for Structure Learning
This is achieved by a novel characterization of acyclicity that is not only smooth but also exact.
Robust Nonparametric Regression under Huber's $ε$-contamination Model
We first show that a simple local binning median step can effectively remove the adversary noise and this median estimator is minimax optimal up to absolute constants over the H\"{o}lder function class with smoothness parameters smaller than or equal to 1.
Binary Classification with Karmic, Threshold-Quasi-Concave Metrics
Complex performance measures, beyond the popular measure of accuracy, are increasingly being used in the context of binary classification.
Revisiting Adversarial Risk
Based on the modified definition, we show that there is no trade-off between adversarial and standard accuracies; there exist classifiers that are robust and achieve high standard accuracy.
Deep Density Destructors
Unlike Gaussianization, our destructive transformation has the elegant property that the density function is equal to the absolute value of the Jacobian determinant.
Loss Decomposition for Fast Learning in Large Output Spaces
For problems with large output spaces, evaluation of the loss function and its gradient are expensive, typically taking linear time in the size of the output space.
Sample Complexity of Nonparametric Semi-Supervised Learning
We study the sample complexity of semi-supervised learning (SSL) and introduce new assumptions based on the mismatch between a mixture model learned from unlabeled data and the true mixture model induced by the (unknown) class conditional distributions.
Efficient Tensor Decomposition with Boolean Factors
Tensor decomposition has been extensively used as a tool for exploratory analysis.
Word Mover's Embedding: From Word2Vec to Document Embedding
While the celebrated Word2Vec technique yields semantically rich representations for individual words, there has been relatively less success in extending to generate unsupervised sentences or documents embeddings.
Representer Point Selection for Explaining Deep Neural Networks
We propose to explain the predictions of a deep neural network, by pointing to the set of what we call representer points in the training set, for a given test point prediction.
Towards Aggregating Weighted Feature Attributions
Current approaches for explaining machine learning models fall into two distinct classes: antecedent event influence and value attribution.
On the (In)fidelity and Sensitivity for Explanations
We analyze optimal explanations with respect to both these measures, and while the optimal explanation for sensitivity is a vacuous constant explanation, the optimal explanation for infidelity is a novel combination of two popular explanation methods.
Adaptive Hard Thresholding for Near-optimal Consistent Robust Regression
We provide a nearly linear time estimator which consistently estimates the true regression vector, even with $1-o(1)$ fraction of corruptions.
DEEP-TRIM: REVISITING L1 REGULARIZATION FOR CONNECTION PRUNING OF DEEP NETWORK
State-of-the-art deep neural networks (DNNs) typically have tens of millions of parameters, which might not fit into the upper levels of the memory hierarchy, thus increasing the inference time and energy consumption significantly, and prohibiting their use on edge devices such as mobile phones.
A Unified Approach to Robust Mean Estimation
Building on this connection, we provide a simple variant of recent computationally-efficient algorithms for mean estimation in Huber's model, which given our connection entails that the same efficient sample-pruning based estimators is simultaneously robust to heavy-tailed noise and Huber contamination.
Certified Robustness to Adversarial Label-Flipping Attacks via Randomized Smoothing
This paper considers label-flipping attacks, a type of data poisoning attack where an adversary relabels a small number of examples in a training set in order to degrade the performance of the resulting classifier.
On Concept-Based Explanations in Deep Neural Networks
Next, we propose a concept discovery method that considers two additional constraints to encourage the interpretability of the discovered concepts.
Learning Sparse Nonparametric DAGs
We develop a framework for learning sparse nonparametric directed acyclic graphs (DAGs) from data.
On Completeness-aware Concept-Based Explanations in Deep Neural Networks
Next, we propose a concept discovery method that aims to infer a complete set of concepts that are additionally encouraged to be interpretable, which addresses the limitations of existing methods on concept explanations.
Optimal Analysis of Subset-Selection Based L_p Low Rank Approximation
We show that this algorithm has an approximation ratio of $O((k+1)^{1/p})$ for $1\le p\le 2$ and $O((k+1)^{1-1/p})$ for $p\ge 2$.
Automated Dependence Plots
To address these drawbacks, we formalize a method for automating the selection of interesting PDPs and extend PDPs beyond showing single features to show the model response along arbitrary directions, for example in raw feature space or a latent space arising from some generative model.
Game Design for Eliciting Distinguishable Behavior
The ability to inferring latent psychological traits from human behavior is key to developing personalized human-interacting machine learning systems.
MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
Adversarial training is one of the most popular ways to learn robust models but is usually attack-dependent and time costly.
Certified Robustness to Label-Flipping Attacks via Randomized Smoothing
Machine learning algorithms are known to be susceptible to data poisoning attacks, where an adversary manipulates the training data to degrade performance of the resulting classifier.
Minimizing FLOPs to Learn Efficient Sparse Representations
Deep representation learning has become one of the most widely adopted approaches for visual search, recommendation, and identification.
Class-Weighted Classification: Trade-offs and Robust Approaches
We define a robust risk that minimizes risk over a set of weightings and show excess risk bounds for this problem.
Evaluations and Methods for Explanation through Robustness Analysis
In this paper, we establish a novel set of evaluation criteria for such feature based explanations by robustness analysis.
Sub-Seasonal Climate Forecasting via Machine Learning: Challenges, Analysis, and Advances
Sub-seasonal climate forecasting (SSF) focuses on predicting key climate variables such as temperature and precipitation in the 2-week to 2-month time scales.
Learning Minimax Estimators via Online Learning
We view the problem of designing minimax estimators as finding a mixed strategy Nash equilibrium of a zero-sum game.
Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification
In this paper, we provide the first result of the optimal minimax guarantees for the excess risk for adversarially robust classification, under Gaussian mixture model proposed by \cite{schmidt2018adversarially}.
The Risks of Invariant Risk Minimization
We furthermore present the very first results in the non-linear regime: we demonstrate that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution--this is precisely the issue that it was intended to solve.
Generalized Boosting
Using thorough empirical evaluation, we show that our learning algorithms have superior performance over traditional additive boosting algorithms, as well as existing greedy learning techniques for DNNs.
On Learning Ising Models under Huber's Contamination Model
We study the problem of learning Ising models in a setting where some of the samples from the underlying distribution can be arbitrarily corrupted.
Fundamental Limits and Tradeoffs in Invariant Representation Learning
A wide range of machine learning applications such as privacy-preserving learning, algorithmic fairness, and domain adaptation/generalization among others, involve learning invariant representations of the data that aim to achieve two competing goals: (a) maximize information or accuracy with respect to a target response, and (b) maximize invariance or independence with respect to a set of protected features (e. g., for fairness, privacy, etc).
When Is Generalizable Reinforcement Learning Tractable?
Agents trained by reinforcement learning (RL) often fail to generalize beyond the environment they were trained in, even when presented with new scenarios that seem similar to the training environment.
On Proximal Policy Optimization's Heavy-tailed Gradients
In this paper, we present a detailed empirical study to characterize the heavy-tailed nature of the gradients of the PPO surrogate reward function.
An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization
A popular assumption for out-of-distribution generalization is that the training data comprises sub-datasets, each drawn from a distinct distribution; the goal is then to "interpolate" these distributions and "extrapolate" beyond them -- this objective is broadly known as domain generalization.
Contrastive learning of strong-mixing continuous-time stochastic processes
Contrastive learning is a family of self-supervised methods where a model is trained to solve a classification task constructed from unlabeled data.
Iterative Alignment Flows
Existing flow-based approaches estimate multiple flows independently, which is equivalent to learning multiple full generative models.
DORO: Distributional and Outlier Robust Optimization
Many machine learning tasks involve subpopulation shift where the testing data distribution is a subpopulation of the training distribution.
Improving Compositional Generalization in Classification Tasks via Structure Annotations
Compositional generalization is the ability to generalize systematically to a new data distribution by combining known components.
Learning latent causal graphs via mixture oracles
We study the problem of reconstructing a causal graphical model from data in the presence of latent variables.
Heavy-tailed Streaming Statistical Estimation
We consider the task of heavy-tailed statistical estimation given streaming $p$-dimensional samples.
FILM: Following Instructions in Language with Modular Methods
In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal.
Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation
Noise-contrastive estimation (NCE) is a statistically consistent method for learning unnormalized probabilistic models.
Boosted CVaR Classification
To learn such randomized classifiers, we propose the Boosted CVaR Classification framework which is motivated by a direct relationship between CVaR and a classical boosting algorithm called LPBoost.
Understanding Why Generalized Reweighting Does Not Improve Over ERM
Together, our results show that a broad category of what we term GRW approaches are not able to achieve distributionally robust generalization.
Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization
Towards this end, we introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift.
Masked prediction tasks: a parameter identifiability view
This lens is undoubtedly very interesting, but suffers from the problem that there isn't a "canonical" set of downstream tasks to focus on -- in practice, this problem is usually resolved by competing on the benchmark dataset du jour.
Threading the Needle of On and Off-Manifold Value Functions for Shapley Explanations
We formalize the desiderata of value functions that respect both the model and the data manifold in a set of axioms and are robust to perturbation on off-manifold regions, and show that there exists a unique value function that satisfies these axioms, which we term the Joint Baseline value function, and the resulting Shapley value the Joint Baseline Shapley (JBshap), and validate the effectiveness of JBshap in experiments.
Explainable Artificial Intelligence (XAI)
Feature Importance
First is Better Than Last for Language Data Influence
However, we observe that since the activation connected to the last layer of weights contains "shared logic", the data influenced calculated via the last layer weights prone to a ``cancellation effect'', where the data influence of different examples have large magnitude that contradicts each other.
Human-Centered Concept Explanations for Neural Networks
We start by introducing concept explanations including the class of Concept Activation Vectors (CAV) which characterize concepts using vectors in appropriate spaces of neural activations, and discuss different properties of useful concepts, and approaches to measure the usefulness of concept vectors.
Faith-Shap: The Faithful Shapley Interaction Index
We show that by additionally requiring the faithful interaction indices to satisfy interaction-extensions of the standard individual Shapley axioms (dummy, symmetry, linearity, and efficiency), we obtain a unique Faithful Shapley Interaction index, which we denote Faith-Shap, as a natural generalization of the Shapley value to interactions.
Building Robust Ensembles via Margin Boosting
Consequently, an emerging line of work has focused on learning an ensemble of neural networks to defend against adversarial attacks.
Identifiability of deep generative models without auxiliary information
We prove identifiability of a broad class of deep latent variable models that (a) have universal approximation capabilities and (b) are the decoders of variational autoencoders that are commonly used in practice.
Concept Gradient: Concept-based Interpretation Without Linear Assumption
We showed that for a general (potentially non-linear) concept, we can mathematically evaluate how a small change of concept affecting the model's prediction, which leads to an extension of gradient-based interpretation to the concept space.
DAGMA: Learning DAGs via M-matrices and a Log-Determinant Acyclicity Characterization
From the optimization side, we drop the typically used augmented Lagrangian scheme and propose DAGMA ($\textit{DAGs via M-matrices for Acyclicity}$), a method that resembles the central path for barrier methods.
Label Propagation with Weak Supervision
Semi-supervised learning and weakly supervised learning are important paradigms that aim to reduce the growing demand for labeled data in current machine learning applications.
Nash Equilibria and Pitfalls of Adversarial Training in Adversarial Robustness Games
Adversarial training is a standard technique for training adversarially robust models.
Individual Fairness under Uncertainty
Algorithmic fairness, the research field of making machine learning (ML) algorithms fair, is an established area in ML.
Learning with Explanation Constraints
In this paper, we formalize this notion as learning from explanation constraints and provide a learning theoretic framework to analyze how such explanations can improve the learning of our models.
Optimizing NOTEARS Objectives via Topological Swaps
In this work, we delve into the optimization challenges associated with this class of non-convex programs.
Representer Point Selection for Explaining Regularized High-dimensional Models
We introduce a novel class of sample-based explanations we term high-dimensional representers, that can be used to explain the predictions of a regularized high-dimensional model in terms of importance weights for each of the training samples.
Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression
Recent work has built the connection between self-supervised learning and the approximation of the top eigenspace of a graph Laplacian operator, suggesting that learning a linear probe atop such representation can be connected to RKHS regression.
Learning Linear Causal Representations from Interventions under General Nonlinear Mixing
We study the problem of learning causal representations from unknown, latent interventions in a general setting, where the latent distribution is Gaussian but the mixing function is completely general.
iSCAN: Identifying Causal Mechanism Shifts among Nonlinear Additive Noise Models
Structural causal models (SCMs) are widely used in various disciplines to represent causal relationships among variables in complex systems.
Identifying Representations for Intervention Extrapolation
In this paper, we consider the task of intervention extrapolation: predicting how interventions affect an outcome, even when those interventions are not observed at training time, and show that identifiable representations can provide an effective solution to this task even if the interventions affect the outcome non-linearly.
An Interventional Perspective on Identifiability in Gaussian LTI Systems with Independent Component Analysis
We investigate the relationship between system identification and intervention design in dynamical systems.
Spectrally Transformed Kernel Regression
Unlabeled data is a key component of modern machine learning.
Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models
In this work, we relate these two approaches and study how to learn human-interpretable concepts from data.
On the Origins of Linear Representations in Large Language Models
To that end, we introduce a simple latent variable model to abstract and formalize the concept dynamics of the next token prediction.
Optimal Statistical Guaratees for Adversarially Robust Gaussian Classification
In this paper, we provide the first result of the \emph{optimal} minimax guarantees for the excess risk for adversarially robust classification, under Gaussian mixture model proposed by \cite{schmidt2018adversarially}.
Uniform Convergence of Rank-weighted Learning
In this work, we study a novel notion of L-Risk based on the classical idea of rank-weighted learning.
Uniform Convergence of Rank-weighted Learning
In this work, we study a novel notion of L-Risk based on the classical idea of rank-weighted learning.
