Search Results for author: Greg Ver Steeg

Found 77 papers, 38 papers with code

All in the (Exponential) Family: Information Geometry and Thermodynamic Variational Inference

no code implementations ICML 2020 Rob Brekelmans, Vaden Masrani, Frank Wood, Greg Ver Steeg, Aram Galstyan

While the Evidence Lower Bound (ELBO) has become a ubiquitous objective for variational inference, the recently proposed Thermodynamic Variational Objective (TVO) leverages thermodynamic integration to provide a tighter and more general family of bounds.

Scheduling Variational Inference

Temporal Generalization for Spoken Language Understanding

no code implementations NAACL (ACL) 2022 Judith Gaspers, Anoop Kumar, Greg Ver Steeg, Aram Galstyan

Spoken Language Understanding (SLU) models in industry applications are usually trained offline on historic data, but have to perform well on incoming user requests after deployment.

Domain Generalization Spoken Language Understanding

Multi-Task Knowledge Enhancement for Zero-Shot and Multi-Domain Recommendation in an AI Assistant Application

no code implementations9 Jun 2023 Elan Markowitz, Ziyan Jiang, Fan Yang, Xing Fan, Tony Chen, Greg Ver Steeg, Aram Galstyan

We propose in this work to unify these approaches: Using information from interactions in other domains as well as external knowledge graphs to make predictions in a new domain that would be impossible with either information source alone.

Knowledge Graphs Recommendation Systems

Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning

1 code implementation30 May 2023 Umang Gupta, Aram Galstyan, Greg Ver Steeg

This can be a drawback for low-resource applications and training with differential-privacy constraints, where excessive noise may be introduced during finetuning.

Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models

no code implementations26 May 2023 Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, Greg Ver Steeg

Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pre-trained network.

Neural Architecture Search

Measuring and Mitigating Local Instability in Deep Neural Networks

no code implementations18 May 2023 Arghya Datta, Subhrangshu Nandi, Jingcheng Xu, Greg Ver Steeg, He Xie, Anoop Kumar, Aram Galstyan

We formulate the model stability problem by studying how the predictions of a model change, even when it is retrained on the same data, as a consequence of stochasticity in the training process.

Natural Language Understanding

Improving Mutual Information Estimation with Annealed and Energy-Based Bounds

1 code implementation ICLR 2022 Rob Brekelmans, Sicong Huang, Marzyeh Ghassemi, Greg Ver Steeg, Roger Grosse, Alireza Makhzani

Since accurate estimation of MI without density information requires a sample size exponential in the true MI, we assume either a single marginal or the full joint density information is known.

Mutual Information Estimation

Information-Theoretic Diffusion

1 code implementation7 Feb 2023 Xianghao Kong, Rob Brekelmans, Greg Ver Steeg

Denoising diffusion models have spurred significant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation.

Denoising Image Generation +1

Functional Connectome of the Human Brain with Total Correlation

no code implementations6 Oct 2022 Qiang Li, Greg Ver Steeg, Shujian Yu, Jesus Malo

In this work we build on this idea to infer a large scale (whole brain) connectivity network based on Total Correlation and show the possibility of using this kind of networks as biomarkers of brain alterations.

Towards Sparsified Federated Neuroimaging Models via Weight Pruning

no code implementations24 Aug 2022 Dimitris Stripelis, Umang Gupta, Nikhil Dhinagar, Greg Ver Steeg, Paul Thompson, José Luis Ambite

In our experiments in centralized and federated settings on the brain age prediction task (estimating a person's age from their brain MRI), we demonstrate that models can be pruned up to 95% sparsity without affecting performance even in challenging federated learning environments with highly heterogeneous data distributions.

Federated Learning

Functional Connectivity in Visual Areas from Total Correlation

no code implementations11 Aug 2022 Qiang Li, Greg Ver Steeg, Jesus Malo

In this multivariate setting (three nodes with multidimensional signals), we derive analytical results for the three-way Total Correlation and for all possible pairwise Mutual Information measures.

Formal limitations of sample-wise information-theoretic generalization bounds

no code implementations13 May 2022 Hrayr Harutyunyan, Greg Ver Steeg, Aram Galstyan

Remarkably, PAC-Bayes, single-draw and expected squared generalization gap bounds that depend on information in pairs of examples exist.

Generalization Bounds

Secure & Private Federated Neuroimaging

no code implementations11 May 2022 Dimitris Stripelis, Umang Gupta, Hamza Saleem, Nikhil Dhinagar, Tanmay Ghai, Rafael Chrysovalantis Anastasiou, Armaghan Asghar, Greg Ver Steeg, Srivatsan Ravi, Muhammad Naveed, Paul M. Thompson, Jose Luis Ambite

Each site trains the neural network over its private data for some time, then shares the neural network parameters (i. e., weights, gradients) with a Federation Controller, which in turn aggregates the local models, sends the resulting community model back to each site, and the process repeats.

Federated Learning

Federated Progressive Sparsification (Purge, Merge, Tune)+

no code implementations26 Apr 2022 Dimitris Stripelis, Umang Gupta, Greg Ver Steeg, Jose Luis Ambite

Second, the models are incrementally constrained to a smaller set of parameters, which facilitates alignment/merging of the local models and improved learning performance at high sparsification rates.

Inferring topological transitions in pattern-forming processes with self-supervised learning

1 code implementation19 Mar 2022 Marcin Abram, Keith Burghardt, Greg Ver Steeg, Aram Galstyan, Remi Dingreville

The identification and classification of transitions in topological and microstructural regimes in pattern-forming processes are critical for understanding and fabricating microstructurally precise novel materials in many application domains.

Self-Supervised Learning

Implicit SVD for Graph Representation Learning

1 code implementation NeurIPS 2021 Sami Abu-El-Haija, Hesham Mostafa, Marcel Nassar, Valentino Crespi, Greg Ver Steeg, Aram Galstyan

Recent improvements in the performance of state-of-the-art (SOTA) methods for Graph Representational Learning (GRL) have come at the cost of significant computational resource requirements for training, e. g., for calculating gradients via backprop over many data epochs.

Graph Representation Learning

Hamiltonian Dynamics with Non-Newtonian Momentum for Rapid Sampling

1 code implementation NeurIPS 2021 Greg Ver Steeg, Aram Galstyan

Auxiliary neural models can learn to speed up MCMC, but the overhead for training the extra model can be prohibitive.

Information-theoretic generalization bounds for black-box learning algorithms

1 code implementation NeurIPS 2021 Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, Aram Galstyan

We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm.

Generalization Bounds

Attributing Fair Decisions with Attention Interventions

1 code implementation NAACL (TrustNLP) 2022 Ninareh Mehrabi, Umang Gupta, Fred Morstatter, Greg Ver Steeg, Aram Galstyan

The widespread use of Artificial Intelligence (AI) in consequential domains, such as healthcare and parole decision-making systems, has drawn intense scrutiny on the fairness of these methods.

Decision Making Fairness

Secure Neuroimaging Analysis using Federated Learning with Homomorphic Encryption

no code implementations7 Aug 2021 Dimitris Stripelis, Hamza Saleem, Tanmay Ghai, Nikhil Dhinagar, Umang Gupta, Chrysovalantis Anastasiou, Greg Ver Steeg, Srivatsan Ravi, Muhammad Naveed, Paul M. Thompson, Jose Luis Ambite

Federated learning (FL) enables distributed computation of machine learning models over various disparate, remote data sources, without requiring to transfer any individual data to a centralized location.

Benchmarking Federated Learning

q-Paths: Generalizing the Geometric Annealing Path using Power Means

1 code implementation1 Jul 2021 Vaden Masrani, Rob Brekelmans, Thang Bui, Frank Nielsen, Aram Galstyan, Greg Ver Steeg, Frank Wood

Many common machine learning methods involve the geometric annealing path, a sequence of intermediate densities between two distributions of interest constructed using the geometric average.

Bayesian Inference

Membership Inference Attacks on Deep Regression Models for Neuroimaging

no code implementations6 May 2021 Umang Gupta, Dimitris Stripelis, Pradeep K. Lam, Paul M. Thompson, José Luis Ambite, Greg Ver Steeg

In particular, we show that it is possible to infer if a sample was used to train the model given only access to the model prediction (black-box) or access to the model itself (white-box) and some leaked samples from the training data distribution.

Federated Learning regression

Fast Graph Learning with Unique Optimal Solutions

1 code implementation ICLR Workshop GTRL 2021 Sami Abu-El-Haija, Valentino Crespi, Greg Ver Steeg, Aram Galstyan

We consider two popular Graph Representation Learning (GRL) methods: message passing for node classification and network embedding for link prediction.

Graph Learning Graph Representation Learning +3

Graph Traversal with Tensor Functionals: A Meta-Algorithm for Scalable Learning

1 code implementation ICLR 2021 Elan Markowitz, Keshav Balasubramanian, Mehrnoosh Mirtaheri, Sami Abu-El-Haija, Bryan Perozzi, Greg Ver Steeg, Aram Galstyan

We propose Graph Traversal via Tensor Functionals(GTTF), a unifying meta-algorithm framework for easing the implementation of diverse graph algorithms and enabling transparent and efficient scaling to large graphs.

Graph Representation Learning

Controllable Guarantees for Fair Outcomes via Contrastive Information Estimation

2 code implementations11 Jan 2021 Umang Gupta, Aaron M Ferber, Bistra Dilkina, Greg Ver Steeg

Controlling bias in training datasets is vital for ensuring equal treatment, or parity, between different groups in downstream applications.


Likelihood Ratio Exponential Families

no code implementations NeurIPS Workshop DL-IG 2020 Rob Brekelmans, Frank Nielsen, Alireza Makhzani, Aram Galstyan, Greg Ver Steeg

The exponential family is well known in machine learning and statistical physics as the maximum entropy distribution subject to a set of observed constraints, while the geometric mixture path is common in MCMC methods such as annealed importance sampling.


Annealed Importance Sampling with q-Paths

2 code implementations NeurIPS Workshop DL-IG 2020 Rob Brekelmans, Vaden Masrani, Thang Bui, Frank Wood, Aram Galstyan, Greg Ver Steeg, Frank Nielsen

Annealed importance sampling (AIS) is the gold standard for estimating partition functions or marginal likelihoods, corresponding to importance sampling over a path of distributions between a tractable base and an unnormalized target.

Compressing Deep Neural Networks via Layer Fusion

no code implementations29 Jul 2020 James O' Neill, Greg Ver Steeg, Aram Galstyan

This paper proposes \textit{layer fusion} - a model compression technique that discovers which weights to combine and then fuses weights of similar fully-connected, convolutional and attention layers.

Exponential degradation Language Modelling +1

Robust Classification under Class-Dependent Domain Shift

no code implementations10 Jul 2020 Tigran Galstyan, Hrant Khachatrian, Greg Ver Steeg, Aram Galstyan

Investigation of machine learning algorithms robust to changes between the training and test distributions is an active area of research.

Classification General Classification +1

All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference

1 code implementation1 Jul 2020 Rob Brekelmans, Vaden Masrani, Frank Wood, Greg Ver Steeg, Aram Galstyan

We propose to choose intermediate distributions using equal spacing in the moment parameters of our exponential family, which matches grid search performance and allows the schedule to adaptively update over the course of training.

Variational Inference

A Metric Space for Point Process Excitations

no code implementations5 May 2020 Myrl G. Marmarelis, Greg Ver Steeg, Aram Galstyan

A multivariate Hawkes process enables self- and cross-excitations through a triggering matrix that behaves like an asymmetrical covariance structure, characterizing pairwise interactions between the event types.

Point Processes

Invariant Representations through Adversarial Forgetting

no code implementations11 Nov 2019 Ayush Jaiswal, Daniel Moyer, Greg Ver Steeg, Wael Abd-Almageed, Premkumar Natarajan

We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism.

Nearly-Unsupervised Hashcode Representations for Biomedical Relation Extraction

no code implementations IJCNLP 2019 Sahil Garg, Aram Galstyan, Greg Ver Steeg, Guillermo Cecchi

Recently, kernelized locality sensitive hashcodes have been successfully employed as representations of natural language text, especially showing high relevance to biomedical relation extraction tasks.

Relation Extraction

Nearly-Unsupervised Hashcode Representations for Relation Extraction

no code implementations9 Sep 2019 Sahil Garg, Aram Galstyan, Greg Ver Steeg, Guillermo Cecchi

Recently, kernelized locality sensitive hashcodes have been successfully employed as representations of natural language text, especially showing high relevance to biomedical relation extraction tasks.

Relation Extraction

Efficient Covariance Estimation from Temporal Data

2 code implementations30 May 2019 Hrayr Harutyunyan, Daniel Moyer, Hrant Khachatrian, Greg Ver Steeg, Aram Galstyan

Estimating the covariance structure of multivariate time series is a fundamental problem with a wide-range of real-world applications -- from financial modeling to fMRI analysis.

Time Series Time Series Analysis

Exact Rate-Distortion in Autoencoders via Echo Noise

1 code implementation NeurIPS 2019 Rob Brekelmans, Daniel Moyer, Aram Galstyan, Greg Ver Steeg

The noise is constructed in a data-driven fashion that does not require restrictive distributional assumptions.

Representation Learning

Scanner Invariant Representations for Diffusion MRI Harmonization

no code implementations10 Apr 2019 Daniel Moyer, Greg Ver Steeg, Chantal M. W. Tax, Paul M. Thompson

Purpose: In the present work we describe the correction of diffusion-weighted MRI for site and scanner biases using a novel method based on invariant representation.

Fairness Image Reconstruction

Identifying and Analyzing Cryptocurrency Manipulations in Social Media

1 code implementation4 Feb 2019 Mehrnoosh Mirtaheri, Sami Abu-El-Haija, Fred Morstatter, Greg Ver Steeg, Aram Galstyan

Because of the speed and relative anonymity offered by social platforms such as Twitter and Telegram, social media has become a preferred platform for scammers who wish to spread false hype about the cryptocurrency they are trying to pump.

Measures of Tractography Convergence

no code implementations12 Jun 2018 Daniel Moyer, Paul M. Thompson, Greg Ver Steeg

In the present work, we use information theory to understand the empirical convergence rate of tractography, a widely-used approach to reconstruct anatomical fiber pathways in the living brain.

Invariant Representations without Adversarial Training

1 code implementation NeurIPS 2018 Daniel Moyer, Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, Aram Galstyan

Representations of data that are invariant to changes in specified factors are useful for a wide range of problems: removing potential biases in prediction problems, controlling the effects of covariates, and disentangling meaningful factors of variation.

Representation Learning

A Forest Mixture Bound for Block-Free Parallel Inference

no code implementations17 May 2018 Neal Lawton, Aram Galstyan, Greg Ver Steeg

Coordinate ascent variational inference is an important algorithm for inference in probabilistic models, but it is slow because it updates only a single variable at a time.

Variational Inference

Modeling Psychotherapy Dialogues with Kernelized Hashcode Representations: A Nonparametric Information-Theoretic Approach

no code implementations26 Apr 2018 Sahil Garg, Irina Rish, Guillermo Cecchi, Palash Goyal, Sarik Ghazarian, Shuyang Gao, Greg Ver Steeg, Aram Galstyan

We also derive a novel lower bound on mutual information, used as a model-selection criterion favoring representations with better alignment between the utterances of participants in a collaborative dialogue setting, as well as higher predictability of the generated responses.

Dialogue Generation Model Selection

Auto-Encoding Total Correlation Explanation

no code implementations16 Feb 2018 Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, Aram Galstyan

Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned.


Stochastic Learning of Nonstationary Kernels for Natural Language Modeling

no code implementations11 Jan 2018 Sahil Garg, Greg Ver Steeg, Aram Galstyan

Natural language processing often involves computations with semantic or syntactic graphs to facilitate sophisticated reasoning based on structural relationships.

Language Modelling

Kernelized Hashcode Representations for Relation Extraction

1 code implementation10 Nov 2017 Sahil Garg, Aram Galstyan, Greg Ver Steeg, Irina Rish, Guillermo Cecchi, Shuyang Gao

Here we propose to use random subspaces of KLSH codes for efficiently constructing an explicit representation of NLP structures suitable for general classification methods.

General Classification Relation Extraction

Disentangled Representations via Synergy Minimization

1 code implementation10 Oct 2017 Greg Ver Steeg, Rob Brekelmans, Hrayr Harutyunyan, Aram Galstyan

Scientists often seek simplified representations of complex systems to facilitate prediction and understanding.

Unifying Local and Global Change Detection in Dynamic Networks

no code implementations9 Oct 2017 Wenzhe Li, Dong Guo, Greg Ver Steeg, Aram Galstyan

Many real-world networks are complex dynamical systems, where both local (e. g., changing node attributes) and global (e. g., changing network topology) processes unfold over time.

Change Detection

Unsupervised Learning via Total Correlation Explanation

no code implementations27 Jun 2017 Greg Ver Steeg

Learning by children and animals occurs effortlessly and largely without obvious supervision.

Fast structure learning with modular regularization

3 code implementations NeurIPS 2019 Greg Ver Steeg, Hrayr Harutyunyan, Daniel Moyer, Aram Galstyan

We also use our approach for estimating covariance structure for a number of real-world datasets and show that it consistently outperforms state-of-the-art estimators at a fraction of the computational cost.

Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge

1 code implementation TACL 2017 Ryan J. Gallagher, Kyle Reing, David Kale, Greg Ver Steeg

While generative models such as Latent Dirichlet Allocation (LDA) have proven fruitful in topic modeling, they often require detailed assumptions and careful specification of hyperparameters.

Toward Interpretable Topic Discovery via Anchored Correlation Explanation

no code implementations22 Jun 2016 Kyle Reing, David C. Kale, Greg Ver Steeg, Aram Galstyan

Many predictive tasks, such as diagnosing a patient based on their medical chart, are ultimately defined by the decisions of human experts.

Variational Information Maximization for Feature Selection

1 code implementation NeurIPS 2016 Shuyang Gao, Greg Ver Steeg, Aram Galstyan

We demonstrate that approximations made by existing methods are based on unrealistic assumptions.

feature selection

Sifting Common Information from Many Variables

1 code implementation7 Jun 2016 Greg Ver Steeg, Shuyang Gao, Kyle Reing, Aram Galstyan

Measuring the relationship between any pair of variables is a rich and active area of research that is central to scientific practice.

Dimensionality Reduction

Latent Space Model for Multi-Modal Social Data

no code implementations18 Oct 2015 Yoon-Sik Cho, Greg Ver Steeg, Emilio Ferrara, Aram Galstyan

We validate the proposed framework on two problems: prediction of social interactions from user attributes and behaviors, and behavior prediction exploiting network information.

The Information Sieve

2 code implementations8 Jul 2015 Greg Ver Steeg, Aram Galstyan

Each layer of the sieve recovers a single latent factor that is maximally informative about multivariate dependence in the data.

Understanding confounding effects in linguistic coordination: an information-theoretic approach

no code implementations1 Dec 2014 Shuyang Gao, Greg Ver Steeg, Aram Galstyan

We revisit some of the previous studies that reported strong signatures of stylistic accommodation, and find that a significant part of the observed coordination can be attributed to a simple confounding effect - length coordination.

Scalable Link Prediction in Dynamic Networks via Non-Negative Matrix Factorization

no code implementations13 Nov 2014 Linhong Zhu, Dong Guo, Junming Yin, Greg Ver Steeg, Aram Galstyan

We propose a scalable temporal latent space model for link prediction in dynamic social networks, where the goal is to predict links over time based on a sequence of previous graph snapshots.

Link Prediction

Efficient Estimation of Mutual Information for Strongly Dependent Variables

4 code implementations7 Nov 2014 Shuyang Gao, Greg Ver Steeg, Aram Galstyan

We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI.

Discovering Structure in High-Dimensional Data Through Correlation Explanation

3 code implementations NeurIPS 2014 Greg Ver Steeg, Aram Galstyan

We introduce a method to learn a hierarchy of successively more abstract representations of complex data based on optimizing an information-theoretic objective.

Vocal Bursts Intensity Prediction

Phase Transitions in Community Detection: A Solvable Toy Model

no code implementations2 Dec 2013 Greg Ver Steeg, Cristopher Moore, Aram Galstyan, Armen E. Allahverdyan

It predicts a first-order detectability transition whenever $q > 2$, while the finite-temperature cavity method shows that this is the case only when $q > 4$.

Community Detection

Demystifying Information-Theoretic Clustering

no code implementations15 Oct 2013 Greg Ver Steeg, Aram Galstyan, Fei Sha, Simon DeDeo

We propose a novel method for clustering data which is grounded in information-theoretic principles and requires no parametric assumptions.


Information-Theoretic Measures of Influence Based on Content Dynamics

3 code implementations22 Aug 2012 Greg Ver Steeg, Aram Galstyan

The fundamental building block of social influence is for one person to elicit a response in another.

Social and Information Networks Physics and Society Applications

Information Transfer in Social Media

1 code implementation12 Oct 2011 Greg Ver Steeg, Aram Galstyan

Recent research has explored the increasingly important role of social media by examining the dynamics of individual and group behavior, characterizing patterns of information diffusion, and identifying influential individuals.

Social and Information Networks Physics and Society Applications

Cannot find the paper you are looking for? You can Submit a new open access paper.