Search Results for author:
Found 82 papers, 41 papers with code
All in the (Exponential) Family: Information Geometry and Thermodynamic Variational Inference
While the Evidence Lower Bound (ELBO) has become a ubiquitous objective for variational inference, the recently proposed Thermodynamic Variational Objective (TVO) leverages thermodynamic integration to provide a tighter and more general family of bounds.
Temporal Generalization for Spoken Language Understanding
Spoken Language Understanding (SLU) models in industry applications are usually trained offline on historic data, but have to perform well on incoming user requests after deployment.
StATIK: Structure and Text for Inductive Knowledge Graph Completion
Knowledge graphs (KGs) often represent knowledge bases that are incomplete.
Your Diffusion Model is Secretly a Noise Classifier and Benefits from Contrastive Training
This can become problematic for all sampling methods, especially when we move to parallel sampling which requires us to initialize and update the entire sample trajectory of dynamics in parallel, leading to many OOD evaluations.
Prompt Perturbation Consistency Learning for Robust Language Models
However, their performance on sequence labeling tasks such as intent classification and slot filling (IC-SF), which is a central component in personal assistant systems, lags significantly behind discriminative models.
Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encoding
Quantifying the degree of similarity between images is a key copyright issue for image-based machine learning.
Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks
The widespread use of Text-to-Image (T2I) models in content generation requires careful examination of their safety, including their robustness to adversarial attacks.
Interpretable Diffusion via Information Decomposition
For diffusion models, we show that a natural non-negative decomposition of mutual information emerges, allowing us to quantify informative relationships between words and pixels in an image.
Ensembled Prediction Intervals for Causal Outcomes Under Hidden Confounding
We present a simple approach to partial identification using existing causal sensitivity models and show empirically that Caus-Modens gives tighter outcome intervals, as measured by the necessary interval size to achieve sufficient coverage.
Multi-Task Knowledge Enhancement for Zero-Shot and Multi-Domain Recommendation in an AI Assistant Application
We propose in this work to unify these approaches: Using information from interactions in other domains as well as external knowledge graphs to make predictions in a new domain that would be impossible with either information source alone.
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning
This can be a drawback for low-resource applications and training with differential-privacy constraints, where excessive noise may be introduced during finetuning.
Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models
Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pre-trained network.
Measuring and Mitigating Local Instability in Deep Neural Networks
We formulate the model stability problem by studying how the predictions of a model change, even when it is retrained on the same data, as a consequence of stochasticity in the training process.
Improving Mutual Information Estimation with Annealed and Energy-Based Bounds
Since accurate estimation of MI without density information requires a sample size exponential in the true MI, we assume either a single marginal or the full joint density information is known.
Transferring Models Trained on Natural Images to 3D MRI via Position Encoded Slice Models
Transfer learning has remarkably improved computer vision.
Information-Theoretic Diffusion
Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation.
Functional Connectome of the Human Brain with Total Correlation
In this work we build on this idea to infer a large scale (whole brain) connectivity network based on Total Correlation and show the possibility of using this kind of networks as biomarkers of brain alterations.
Towards Sparsified Federated Neuroimaging Models via Weight Pruning
In our experiments in centralized and federated settings on the brain age prediction task (estimating a person's age from their brain MRI), we demonstrate that models can be pruned up to 95% sparsity without affecting performance even in challenging federated learning environments with highly heterogeneous data distributions.
Functional Connectivity via Total Correlation: Analytical results in Visual Areas
As opposed to previous empirical approaches, in this work we present analytical results to prove the advantages of Total Correlation over Mutual Information to describe the functional connectivity.
Formal limitations of sample-wise information-theoretic generalization bounds
Remarkably, PAC-Bayes, single-draw and expected squared generalization gap bounds that depend on information in pairs of examples exist.
Secure & Private Federated Neuroimaging
Each site trains the neural network over its private data for some time, then shares the neural network parameters (i. e., weights, gradients) with a Federation Controller, which in turn aggregates the local models, sends the resulting community model back to each site, and the process repeats.
Federated Progressive Sparsification (Purge, Merge, Tune)+
Second, the models are incrementally constrained to a smaller set of parameters, which facilitates alignment/merging of the local models and improved learning performance at high sparsification rates.
Partial Identification of Dose Responses with Hidden Confounders
Sensitivity analyses provide principled ways to give bounds on causal estimates when confounding variables are hidden.
Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal
Language models excel at generating coherent text, and model compression techniques such as knowledge distillation have enabled their use in resource-constrained settings.
Inferring topological transitions in pattern-forming processes with self-supervised learning
The identification and classification of transitions in topological and microstructural regimes in pattern-forming processes are critical for understanding and fabricating microstructurally precise novel materials in many application domains.
Failure Modes of Domain Generalization Algorithms
On Camelyon-17, domain-invariance degrades the quality of representations on unseen domains.
Implicit SVD for Graph Representation Learning
Recent improvements in the performance of state-of-the-art (SOTA) methods for Graph Representational Learning (GRL) have come at the cost of significant computational resource requirements for training, e. g., for calculating gradients via backprop over many data epochs.
Hamiltonian Dynamics with Non-Newtonian Momentum for Rapid Sampling
Auxiliary neural models can learn to speed up MCMC, but the overhead for training the extra model can be prohibitive.
Information-theoretic generalization bounds for black-box learning algorithms
We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm.
Attributing Fair Decisions with Attention Interventions
The widespread use of Artificial Intelligence (AI) in consequential domains, such as healthcare and parole decision-making systems, has drawn intense scrutiny on the fairness of these methods.
Secure Neuroimaging Analysis using Federated Learning with Homomorphic Encryption
Federated learning (FL) enables distributed computation of machine learning models over various disparate, remote data sources, without requiring to transfer any individual data to a centralized location.
q-Paths: Generalizing the Geometric Annealing Path using Power Means
Many common machine learning methods involve the geometric annealing path, a sequence of intermediate densities between two distributions of interest constructed using the geometric average.
Membership Inference Attacks on Deep Regression Models for Neuroimaging
In particular, we show that it is possible to infer if a sample was used to train the model given only access to the model prediction (black-box) or access to the model itself (white-box) and some leaked samples from the training data distribution.
Fast Graph Learning with Unique Optimal Solutions
We consider two popular Graph Representation Learning (GRL) methods: message passing for node classification and network embedding for link prediction.
Improved Brain Age Estimation with Slice-based Set Networks
Deep Learning for neuroimaging data is a promising but challenging direction.
Graph Traversal with Tensor Functionals: A Meta-Algorithm for Scalable Learning
We propose Graph Traversal via Tensor Functionals(GTTF), a unifying meta-algorithm framework for easing the implementation of diverse graph algorithms and enabling transparent and efficient scaling to large graphs.
Controllable Guarantees for Fair Outcomes via Contrastive Information Estimation
Controlling bias in training datasets is vital for ensuring equal treatment, or parity, between different groups in downstream applications.
Likelihood Ratio Exponential Families
The exponential family is well known in machine learning and statistical physics as the maximum entropy distribution subject to a set of observed constraints, while the geometric mixture path is common in MCMC methods such as annealed importance sampling.
Annealed Importance Sampling with q-Paths
Annealed importance sampling (AIS) is the gold standard for estimating partition functions or marginal likelihoods, corresponding to importance sampling over a path of distributions between a tractable base and an unnormalized target.
Compressing Deep Neural Networks via Layer Fusion
This paper proposes \textit{layer fusion} - a model compression technique that discovers which weights to combine and then fuses weights of similar fully-connected, convolutional and attention layers.
Robust Classification under Class-Dependent Domain Shift
Investigation of machine learning algorithms robust to changes between the training and test distributions is an active area of research.
All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
We propose to choose intermediate distributions using equal spacing in the moment parameters of our exponential family, which matches grid search performance and allows the schedule to adaptively update over the course of training.
Overview of Scanner Invariant Representations
Pooled imaging data from multiple sources is subject to bias from each source.
A Metric Space for Point Process Excitations
A multivariate Hawkes process enables self- and cross-excitations through a triggering matrix that behaves like an asymmetrical covariance structure, characterizing pairwise interactions between the event types.
Improving Generalization by Controlling Label-Noise Information in Neural Network Weights
In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise.
Discovery and Separation of Features for Invariant Representation Learning
Supervised machine learning models often associate irrelevant nuisance factors with the prediction target, which hurts generalization.
Invariant Representations through Adversarial Forgetting
We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism.
Nearly-Unsupervised Hashcode Representations for Biomedical Relation Extraction
Recently, kernelized locality sensitive hashcodes have been successfully employed as representations of natural language text, especially showing high relevance to biomedical relation extraction tasks.
Nearly-Unsupervised Hashcode Representations for Relation Extraction
Recently, kernelized locality sensitive hashcodes have been successfully employed as representations of natural language text, especially showing high relevance to biomedical relation extraction tasks.
Analysis | OPEN | Published: 17 June 2019 Multitask learning and benchmarking with clinical time series data
Health care is one of the most exciting frontiers in data mining and machine learning.
Efficient Covariance Estimation from Temporal Data
Estimating the covariance structure of multivariate time series is a fundamental problem with a wide-range of real-world applications -- from financial modeling to fMRI analysis.
MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing
Existing popular methods for semi-supervised learning with Graph Neural Networks (such as the Graph Convolutional Network) provably cannot learn a general class of neighborhood mixing relationships.
Exact Rate-Distortion in Autoencoders via Echo Noise
The noise is constructed in a data-driven fashion that does not require restrictive distributional assumptions.
Scanner Invariant Representations for Diffusion MRI Harmonization
Purpose: In the present work we describe the correction of diffusion-weighted MRI for site and scanner biases using a novel method based on invariant representation.
Identifying and Analyzing Cryptocurrency Manipulations in Social Media
Because of the speed and relative anonymity offered by social platforms such as Twitter and Telegram, social media has become a preferred platform for scammers who wish to spread false hype about the cryptocurrency they are trying to pump.
Measures of Tractography Convergence
In the present work, we use information theory to understand the empirical convergence rate of tractography, a widely-used approach to reconstruct anatomical fiber pathways in the living brain.
Invariant Representations without Adversarial Training
Representations of data that are invariant to changes in specified factors are useful for a wide range of problems: removing potential biases in prediction problems, controlling the effects of covariates, and disentangling meaningful factors of variation.
A Forest Mixture Bound for Block-Free Parallel Inference
Coordinate ascent variational inference is an important algorithm for inference in probabilistic models, but it is slow because it updates only a single variable at a time.
Modeling Psychotherapy Dialogues with Kernelized Hashcode Representations: A Nonparametric Information-Theoretic Approach
We also derive a novel lower bound on mutual information, used as a model-selection criterion favoring representations with better alignment between the utterances of participants in a collaborative dialogue setting, as well as higher predictability of the generated responses.
Auto-Encoding Total Correlation Explanation
Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned.
Stochastic Learning of Nonstationary Kernels for Natural Language Modeling
Natural language processing often involves computations with semantic or syntactic graphs to facilitate sophisticated reasoning based on structural relationships.
Kernelized Hashcode Representations for Relation Extraction
Here we propose to use random subspaces of KLSH codes for efficiently constructing an explicit representation of NLP structures suitable for general classification methods.
Disentangled Representations via Synergy Minimization
Scientists often seek simplified representations of complex systems to facilitate prediction and understanding.
Unifying Local and Global Change Detection in Dynamic Networks
Many real-world networks are complex dynamical systems, where both local (e. g., changing node attributes) and global (e. g., changing network topology) processes unfold over time.
Unsupervised Learning via Total Correlation Explanation
Learning by children and animals occurs effortlessly and largely without obvious supervision.
Fast structure learning with modular regularization
We also use our approach for estimating covariance structure for a number of real-world datasets and show that it consistently outperforms state-of-the-art estimators at a fraction of the computational cost.
Multitask learning and benchmarking with clinical time series data
Health care is one of the most exciting frontiers in data mining and machine learning.
Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge
While generative models such as Latent Dirichlet Allocation (LDA) have proven fruitful in topic modeling, they often require detailed assumptions and careful specification of hyperparameters.
Toward Interpretable Topic Discovery via Anchored Correlation Explanation
Many predictive tasks, such as diagnosing a patient based on their medical chart, are ultimately defined by the decisions of human experts.
Variational Information Maximization for Feature Selection
We demonstrate that approximations made by existing methods are based on unrealistic assumptions.
Sifting Common Information from Many Variables
Measuring the relationship between any pair of variables is a rich and active area of research that is central to scientific practice.
Latent Space Model for Multi-Modal Social Data
We validate the proposed framework on two problems: prediction of social interactions from user attributes and behaviors, and behavior prediction exploiting network information.
The Information Sieve
Each layer of the sieve recovers a single latent factor that is maximally informative about multivariate dependence in the data.
Understanding confounding effects in linguistic coordination: an information-theoretic approach
We revisit some of the previous studies that reported strong signatures of stylistic accommodation, and find that a significant part of the observed coordination can be attributed to a simple confounding effect - length coordination.
Scalable Link Prediction in Dynamic Networks via Non-Negative Matrix Factorization
We propose a scalable temporal latent space model for link prediction in dynamic social networks, where the goal is to predict links over time based on a sequence of previous graph snapshots.
Efficient Estimation of Mutual Information for Strongly Dependent Variables
We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI.
Maximally Informative Hierarchical Representations of High-Dimensional Data
We present bounds on how informative a representation is about input data.
Discovering Structure in High-Dimensional Data Through Correlation Explanation
We introduce a method to learn a hierarchy of successively more abstract representations of complex data based on optimizing an information-theoretic objective.
Phase Transitions in Community Detection: A Solvable Toy Model
It predicts a first-order detectability transition whenever $q > 2$, while the finite-temperature cavity method shows that this is the case only when $q > 4$.
Demystifying Information-Theoretic Clustering
We propose a novel method for clustering data which is grounded in information-theoretic principles and requires no parametric assumptions.
Information-Theoretic Measures of Influence Based on Content Dynamics
The fundamental building block of social influence is for one person to elicit a response in another.
Social and Information Networks Physics and Society Applications
Information Transfer in Social Media
Recent research has explored the increasingly important role of social media by examining the dynamics of individual and group behavior, characterizing patterns of information diffusion, and identifying influential individuals.
Social and Information Networks Physics and Society Applications
