Search Results for author: Anshumali Shrivastava

Found 93 papers, 26 papers with code

NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention

1 code implementation2 Mar 2024 Tianyi Zhang, Jonah Wonkyu Yi, Bowen Yao, Zhaozhuo Xu, Anshumali Shrivastava

Large language model inference on Central Processing Units (CPU) is challenging due to the vast quantities of expensive Multiply-Add (MAD) matrix operations in the attention computations.

Language Modelling Large Language Model

Wisdom of Committee: Distilling from Foundation Model to Specialized Application Model

no code implementations21 Feb 2024 Zichang Liu, Qingyun Liu, Yuening Li, Liang Liu, Anshumali Shrivastava, Shuchao Bi, Lichan Hong, Ed H. Chi, Zhe Zhao

Further, to accommodate the dissimilarity among the teachers in the committee, we introduce DiverseDistill, which allows the student to understand the expertise of each teacher and extract task knowledge.

Knowledge Distillation Transfer Learning

Learning Scalable Structural Representations for Link Prediction with Bloom Signatures

no code implementations28 Dec 2023 Tianyi Zhang, Haoteng Yin, Rongzhe Wei, Pan Li, Anshumali Shrivastava

We further show that any type of neighborhood overlap-based heuristic can be estimated by a neural network that takes Bloom signatures as input.

Link Prediction

Contractive error feedback for gradient compression

no code implementations13 Dec 2023 Bingcong Li, Shuai Zheng, Parameswaran Raman, Anshumali Shrivastava, Georgios B. Giannakis

On-device memory concerns in distributed deep learning have become severe due to (i) the growth of model size in multi-GPU training, and (ii) the wide adoption of deep neural networks for federated learning on IoT devices which have limited storage.

Federated Learning Image Classification +2

Adaptive Sampling for Deep Learning via Efficient Nonparametric Proxies

no code implementations22 Nov 2023 Shabnam Daghaghi, Benjamin Coleman, Benito Geordie, Anshumali Shrivastava

To address this problem, we propose a novel sampling distribution based on nonparametric kernel regression that learns an effective importance score as the neural network trains.

regression

Heterogeneous federated collaborative filtering using FAIR: Federated Averaging in Random Subspaces

1 code implementation3 Nov 2023 Aditya Desai, Benjamin Meisburger, Zichang Liu, Anshumali Shrivastava

To include data from all devices in federated learning, we must enable collective training of embedding tables on devices with heterogeneous memory capacities.

Collaborative Filtering Federated Learning +1

Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time

1 code implementation26 Oct 2023 Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, Beidi Chen

We show that contextual sparsity exists, that it can be accurately predicted, and that we can exploit it to speed up LLM inference in wall-clock time without compromising LLM's quality or in-context learning ability.

In-Context Learning

In defense of parameter sharing for model-compression

no code implementations17 Oct 2023 Aditya Desai, Anshumali Shrivastava

In this paper, we comprehensively assess the trade-off between memory and accuracy across RPS, pruning techniques, and building smaller models.

Model Compression

Zen: Near-Optimal Sparse Tensor Synchronization for Distributed DNN Training

no code implementations23 Sep 2023 Zhuang Wang, Zhaozhuo Xu, Anshumali Shrivastava, T. S. Eugene Ng

We then systematically explore the design space of communication schemes for sparse tensors and find the optimal one.

CAPS: A Practical Partition Index for Filtered Similarity Search

1 code implementation29 Aug 2023 Gaurav Gupta, Jonah Yi, Benjamin Coleman, Chen Luo, Vihan Lakshman, Anshumali Shrivastava

With the surging popularity of approximate near-neighbor search (ANNS), driven by advances in neural representation learning, the ability to serve queries accompanied by a set of constraints has become an area of intense interest.

Representation Learning

A Theoretical Analysis Of Nearest Neighbor Search On Approximate Near Neighbor Graph

no code implementations10 Mar 2023 Anshumali Shrivastava, Zhao Song, Zhaozhuo Xu

Current theoretical literature focuses on greedy search on exact near neighbor graph while practitioners use approximate near neighbor graph (ANN-Graph) to reduce the preprocessing time.

Learning Multimodal Data Augmentation in Feature Space

1 code implementation29 Dec 2022 Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, Andrew Gordon Wilson

The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is a defining feature of intelligent systems.

Data Augmentation Image Classification +1

Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity

no code implementations29 Jan 2022 Minghao Yan, Nicholas Meisburger, Tharun Medini, Anshumali Shrivastava

We show that with reduced communication, due to sparsity, we can train close to a billion parameter model on simple 4-16 core CPU nodes connected by basic low bandwidth interconnect.

Cloud Computing

Locality Sensitive Teaching

no code implementations NeurIPS 2021 Zhaozhuo Xu, Beidi Chen, Chaojian Li, Weiyang Liu, Le Song, Yingyan Lin, Anshumali Shrivastava

However, as one of the most influential and practical MT paradigms, iterative machine teaching (IMT) is prohibited on IoT devices due to its inefficient and unscalable algorithms.

Embedding models through the lens of Stable Coloring

no code implementations29 Sep 2021 Aditya Desai, Shashank Sonkar, Anshumali Shrivastava, Richard Baraniuk

Grounded on this framework, we show that many algorithms ranging across different domains are, in fact, searching for continuous stable coloring solutions of an underlying graph corresponding to the domain.

Denoising

STORM: Sketch Toward Online Risk Minimization

no code implementations29 Sep 2021 Gaurav Gupta, Benjamin Coleman, John Chen, Anshumali Shrivastava

To this end, we propose STORM, an online sketching-based method for empirical risk minimization.

Classification regression

Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference

no code implementations4 Aug 2021 Aditya Desai, Li Chou, Anshumali Shrivastava

In this paper, we present Random Offset Block Embedding Array (ROBE) as a low memory alternative to embedding tables which provide orders of magnitude reduction in memory usage while maintaining accuracy and boosting execution speed.

Model Compression

Efficient Inference via Universal LSH Kernel

no code implementations21 Jun 2021 Zichang Liu, Benjamin Coleman, Anshumali Shrivastava

Large machine learning models achieve unprecedented performance on various tasks and have evolved as the go-to technique.

Knowledge Distillation Quantization

Sublinear Least-Squares Value Iteration via Locality Sensitive Hashing

no code implementations18 May 2021 Anshumali Shrivastava, Zhao Song, Zhaozhuo Xu

We present the first provable Least-Squares Value Iteration (LSVI) algorithms that have runtime complexity sublinear in the number of actions.

reinforcement-learning Reinforcement Learning (RL)

IRLI: Iterative Re-partitioning for Learning to Index

no code implementations17 Mar 2021 Gaurav Gupta, Tharun Medini, Anshumali Shrivastava, Alexander J Smola

Neural models have transformed the fundamental information retrieval problem of mapping a query to a giant set of items.

Information Retrieval Multi-Label Classification +1

Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and More

2 code implementations6 Mar 2021 Shabnam Daghaghi, Nicholas Meisburger, Mengnan Zhao, Yong Wu, Sameh Gobriel, Charlie Tai, Anshumali Shrivastava

Our work highlights several novel perspectives and opportunities for implementing randomized algorithms for deep learning on modern CPUs.

Density Sketches for Sampling and Estimation

no code implementations24 Feb 2021 Aditya Desai, Benjamin Coleman, Anshumali Shrivastava

We introduce Density sketches (DS): a succinct online summary of the data distribution.

Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems

1 code implementation24 Feb 2021 Aditya Desai, Yanzhou Pan, Kuangyuan Sun, Li Chou, Anshumali Shrivastava

In particular, our LMA embeddings achieve the same performance compared to standard embeddings with a 16$\times$ reduction in memory footprint.

Recommendation Systems

MONGOOSE: A Learnable LSH Framework for Efficient Neural Network Training

no code implementations ICLR 2021 Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, Christopher Re

Recent advances by practitioners in the deep learning community have breathed new life into Locality Sensitive Hashing (LSH), using it to reduce memory and time bottlenecks in neural network (NN) training.

Efficient Neural Network Language Modelling +2

A Truly Constant-time Distribution-aware Negative Sampling

no code implementations1 Jan 2021 Shabnam Daghaghi, Tharun Medini, Beidi Chen, Mengnan Zhao, Anshumali Shrivastava

Softmax classifiers with a very large number of classes naturally occur in many applications such as natural language processing and information retrieval.

Information Retrieval Retrieval

A Tale of Two Efficient and Informative Negative Sampling Distributions

no code implementations31 Dec 2020 Shabnam Daghaghi, Tharun Medini, Nicholas Meisburger, Beidi Chen, Mengnan Zhao, Anshumali Shrivastava

Unfortunately, due to the dynamically updated parameters and data samples, there is no sampling scheme that is provably adaptive and samples the negative classes efficiently.

Information Retrieval Retrieval +1

Neighbor Oblivious Learning (NObLe) for Device Localization and Tracking

no code implementations23 Nov 2020 Zichang Liu, Li Chou, Anshumali Shrivastava

In this paper, we argue that the state-of-the-art-systems are significantly worse in terms of accuracy because they are incapable of utilizing these essential structural information.

Learning Sampling Distributions Using Local 3D Workspace Decompositions for Motion Planning in High Dimensions

1 code implementation29 Oct 2020 Constantinos Chamzas, Zachary Kingston, Carlos Quintero-Peña, Anshumali Shrivastava, Lydia E. Kavraki

Earlier work has shown that reusing experience from prior motion planning problems can improve the efficiency of similar, future motion planning queries.

Motion Planning

SOLAR: Sparse Orthogonal Learned and Random Embeddings

no code implementations ICLR 2021 Tharun Medini, Beidi Chen, Anshumali Shrivastava

The label vectors are random, sparse, and near-orthogonal by design, while the query vectors are learned and sparse.

Multi-Label Classification

Bloom Origami Assays: Practical Group Testing

no code implementations21 Jul 2020 Louis Abraham, Gary Becigneul, Benjamin Coleman, Bernhard Scholkopf, Anshumali Shrivastava, Alexander Smola

Group testing is a well-studied problem with several appealing solutions, but recent biological studies impose practical constraints for COVID-19 that are incompatible with traditional methods.

Climbing the WOL: Training for Cheaper Inference

no code implementations2 Jul 2020 Zichang Liu, Zhaozhuo Xu, Alan Ji, Jonathan Li, Beidi Chen, Anshumali Shrivastava

Efficient inference for wide output layers (WOLs) is an essential yet challenging task in large scale machine learning.

Retrieval

A One-Pass Private Sketch for Most Machine Learning Tasks

no code implementations16 Jun 2020 Benjamin Coleman, Anshumali Shrivastava

Existing methods for DP kernel density estimation scale poorly, often exponentially slower with an increase in dimensions.

BIG-bench Machine Learning Density Estimation

Privacy Adversarial Network: Representation Learning for Mobile Data Privacy

no code implementations8 Jun 2020 Sicong Liu, Junzhao Du, Anshumali Shrivastava, Lin Zhong

This work departs from prior works in methodology: we leverage adversarial learning to a better balance between privacy and utility.

Representation Learning

Sub-linear RACE Sketches for Approximate Kernel Density Estimation on Streaming Data

no code implementations4 Dec 2019 Benjamin Coleman, Anshumali Shrivastava

We evaluate our method on real-world high-dimensional datasets and show that our sketch achieves 10x better compression compared to competing methods.

Density Estimation

FourierSAT: A Fourier Expansion-Based Algebraic Framework for Solving Hybrid Boolean Constraints

2 code implementations2 Dec 2019 Anastasios Kyrillidis, Anshumali Shrivastava, Moshe Y. Vardi, Zhiwei Zhang

By such a reduction to continuous optimization, we propose an algebraic framework for solving systems consisting of different types of constraints.

Fast and Accurate Stochastic Gradient Estimation

1 code implementation NeurIPS 2019 Beidi Chen, Yingchen Xu, Anshumali Shrivastava

In this paper, we break this barrier by providing the first demonstration of a scheme, Locality sensitive hashing (LSH) sampled Stochastic Gradient Descent (LGD), which leads to superior gradient estimation while keeping the sampling cost per iteration similar to that of the uniform sampling.

Lsh-sampling Breaks the Computation Chicken-and-egg Loop in Adaptive Stochastic Gradient Estimation

no code implementations30 Oct 2019 Beidi Chen, Yingchen Xu, Anshumali Shrivastava

In this paper, we break this barrier by providing the first demonstration of a scheme, Locality sensitive hashing (LSH) sampled Stochastic Gradient Descent (LGD), which leads to superior gradient estimation while keeping the sampling cost per iteration similar to that of the uniform sampling.

Adaptive Learned Bloom Filter (Ada-BF): Efficient Utilization of the Classifier

no code implementations NeurIPS 2020 Zhenwei Dai, Anshumali Shrivastava

Recent work suggests improving the performance of Bloom filter by incorporating a machine learning model as a binary classifier.

BIG-bench Machine Learning

SDM-Net: A Simple and Effective Model for Generalized Zero-Shot Learning

no code implementations10 Sep 2019 Shabnam Daghaghi, Tharun Medini, Anshumali Shrivastava

Zero-Shot Learning (ZSL) is a classification task where we do not have even a single training labeled example from a set of unseen classes.

Descriptive General Classification +3

Revisiting Consistent Hashing with Bounded Loads

1 code implementation23 Aug 2019 John Chen, Ben Coleman, Anshumali Shrivastava

We show, both theoretically and empirically, that our proposed solution is significantly superior for load balancing and is optimal in many senses.

Data Structures and Algorithms

Using Local Experiences for Global Motion Planning

no code implementations20 Mar 2019 Constantinos Chamzas, Anshumali Shrivastava, Lydia E. Kavraki

In this work, we decompose the workspace into local primitives, memorizing local experiences by these primitives in the form of local samplers, and store them in a database.

Motion Planning

Sub-linear Memory Sketches for Near Neighbor Search on Streaming Data

no code implementations18 Feb 2019 Benjamin Coleman, Richard G. Baraniuk, Anshumali Shrivastava

We present the first sublinear memory sketch that can be queried to find the nearest neighbors in a dataset.

Density Estimation

Compressing Gradient Optimizers via Count-Sketches

1 code implementation1 Feb 2019 Ryan Spring, Anastasios Kyrillidis, Vijai Mohan, Anshumali Shrivastava

The problem is becoming more severe as deep learning models continue to grow larger in order to learn from complex, large-scale datasets.

Better accuracy with quantified privacy: representations learned via reconstructive adversarial network

no code implementations ICLR 2019 Sicong Liu, Anshumali Shrivastava, Junzhao Du, Lin Zhong

This work represents a methodical departure from prior works: we balance between a measure of privacy and another of utility by leveraging adversarial learning to find a sweeter tradeoff.

BIG-bench Machine Learning General Classification

Topkapi: Parallel and Fast Sketches for Finding Top-K Frequent Elements

1 code implementation NeurIPS 2018 Ankush Mandal, He Jiang, Anshumali Shrivastava, Vivek Sarkar

In particular, for identifying top-K frequent items, Count-Min Sketch (CMS) has fantastic update time but lack the important property of reducibility which is needed for exploiting available massive data parallelism.

Probabilistic Blocking with An Application to the Syrian Conflict

no code implementations11 Oct 2018 Rebecca C. Steorts, Anshumali Shrivastava

Entity resolution seeks to merge databases as to remove duplicate entries where unique identifiers are typically unknown.

Blocking Information Retrieval +1

Extreme Classification in Log Memory

no code implementations9 Oct 2018 Qixuan Huang, Yiqiu Wang, Tharun Medini, Anshumali Shrivastava

With MACH we can train ODP dataset with 100, 000 classes and 400, 000 features on a single Titan X GPU, with the classification accuracy of 19. 28%, which is the best-reported accuracy on this dataset.

Classification General Classification

Mimicking actions is a good strategy for beginners: Fast Reinforcement Learning with Expert Action Sequences

no code implementations27 Sep 2018 Tharun Medini, Anshumali Shrivastava

Imitation Learning is the task of mimicking the behavior of an expert player in a Reinforcement Learning(RL) Environment to enhance the training of a fresh agent (called novice) beginning from scratch.

Atari Games Imitation Learning +2

Ultra Large-Scale Feature Selection using Count-Sketches

1 code implementation ICML 2018 Amirali Aghazadeh, Ryan Spring, Daniel LeJeune, Gautam Dasarathy, Anshumali Shrivastava, baraniuk

We demonstrate that MISSION accurately and efficiently performs feature selection on real-world, large-scale datasets with billions of dimensions.

BIG-bench Machine Learning feature selection

MISSION: Ultra Large-Scale Feature Selection using Count-Sketches

1 code implementation12 Jun 2018 Amirali Aghazadeh, Ryan Spring, Daniel LeJeune, Gautam Dasarathy, Anshumali Shrivastava, Richard G. Baraniuk

We demonstrate that MISSION accurately and efficiently performs feature selection on real-world, large-scale datasets with billions of dimensions.

BIG-bench Machine Learning feature selection

Scaling-up Split-Merge MCMC with Locality Sensitive Sampling (LSS)

no code implementations21 Feb 2018 Chen Luo, Anshumali Shrivastava

It is well known that state-of-the-art methods for split-merge MCMC do not scale well.

MACH: Embarrassingly parallel $K$-class classification in $O(d\log{K})$ memory and $O(K\log{K} + d\log{K})$ time, instead of $O(Kd)$

no code implementations ICLR 2018 Qixuan Huang, Anshumali Shrivastava, Yiqiu Wang

MACH is the first generic $K$-classification algorithm, with provably theoretical guarantees, which requires $O(\log{K})$ memory without any assumption on the relationship between classes.

Classification General Classification

LSH-SAMPLING BREAKS THE COMPUTATIONAL CHICKEN-AND-EGG LOOP IN ADAPTIVE STOCHASTIC GRADIENT ESTIMATION

no code implementations ICLR 2018 Beidi Chen, Yingchen Xu, Anshumali Shrivastava

In this paper, we break this barrier by providing the first demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to that of the uniform sampling.

Accelerating Dependency Graph Learning from Heterogeneous Categorical Event Streams via Knowledge Transfer

no code implementations25 Aug 2017 Chen Luo, Zhengzhang Chen, Lu-An Tang, Anshumali Shrivastava, Zhichun Li

Given a well-trained dependency graph from a source domain and an immature dependency graph from a target domain, how can we extract the entity and dependency knowledge from the source to enhance the target?

Graph Learning Intrusion Detection +1

A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models

1 code implementation15 Mar 2017 Ryan Spring, Anshumali Shrivastava

We propose a new sampling scheme and an unbiased estimator that estimates the partition function accurately in sub-linear time.

Optimal Densification for Fast and Accurate Minwise Hashing

1 code implementation ICML 2017 Anshumali Shrivastava

Minwise hashing is a fundamental and one of the most successful hashing algorithm in the literature.

Revisiting Winner Take All (WTA) Hashing for Sparse Datasets

no code implementations6 Dec 2016 Beidi Chen, Anshumali Shrivastava

WTA (Winner Take All) hashing has been successfully applied in many large scale vision applications.

General Classification Image Classification +1

Sub-Linear Privacy-Preserving Near-Neighbor Search

no code implementations6 Dec 2016 M. Sadegh Riazi, Beidi Chen, Anshumali Shrivastava, Dan Wallach, Farinaz Koushanfar

In Near-Neighbor Search (NNS), a new client queries a database (held by a server) for the most similar data (near-neighbors) given a certain similarity metric.

Privacy Preserving

Simple and Efficient Weighted Minwise Hashing

no code implementations NeurIPS 2016 Anshumali Shrivastava

Weighted minwise hashing (WMH) is one of the fundamental subroutine, required by many celebrated approximation algorithms, commonly adopted in industrial practice for large -scale search and learning.

SSH (Sketch, Shingle, & Hash) for Indexing Massive-Scale Time Series

1 code implementation24 Oct 2016 Chen Luo, Anshumali Shrivastava

However, branch and bound based pruning are only useful for very short queries (low dimensional time series), and the bounds are quite weak for longer queries.

Dynamic Time Warping Time Series +1

Scalable and Sustainable Deep Learning via Randomized Hashing

no code implementations26 Feb 2016 Ryan Spring, Anshumali Shrivastava

A unique property of the proposed hashing based back-propagation is that the updates are always sparse.

Asymmetric Minwise Hashing

1 code implementation14 Nov 2014 Anshumali Shrivastava, Ping Li

Minwise hashing (Minhash) is a widely popular indexing scheme in practice.

Retrieval

Improved Asymmetric Locality Sensitive Hashing (ALSH) for Maximum Inner Product Search (MIPS)

no code implementations20 Oct 2014 Anshumali Shrivastava, Ping Li

In the prior work, the authors use asymmetric transformations which convert the problem of approximate MIPS into the problem of approximate near neighbor search which can be efficiently solved using hashing.

In Defense of MinHash Over SimHash

no code implementations16 Jul 2014 Anshumali Shrivastava, Ping Li

To provide a common basis for comparison, we evaluate retrieval results in terms of $\mathcal{S}$ for both MinHash and SimHash.

Retrieval valid

Improved Densification of One Permutation Hashing

1 code implementation18 Jun 2014 Anshumali Shrivastava, Ping Li

The existing work on densification of one permutation hashing reduces the query processing cost of the $(K, L)$-parameterized Locality Sensitive Hashing (LSH) algorithm with minwise hashing, from $O(dKL)$ to merely $O(d + KL)$, where $d$ is the number of nonzeros of the data vector, $K$ is the number of hashes in each hash table, and $L$ is the number of hash tables.

Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)

no code implementations NeurIPS 2014 Anshumali Shrivastava, Ping Li

Our proposal is based on an interesting mathematical phenomenon in which inner products, after independent asymmetric transformations, can be converted into the problem of approximate near neighbor search.

Collaborative Filtering

Graph Kernels via Functional Embedding

no code implementations21 Apr 2014 Anshumali Shrivastava, Ping Li

We propose a representation of graph as a functional object derived from the power iteration of the underlying adjacency matrix.

General Classification Graph Classification

A New Space for Comparing Graphs

no code implementations17 Apr 2014 Anshumali Shrivastava, Ping Li

We show that the proposed matrix representation encodes the spectrum of the underlying adjacency matrix and it also contains information about the counts of small sub-structures present in the graph such as triangles and small paths.

Clustering General Classification

Coding for Random Projections and Approximate Near Neighbor Search

no code implementations31 Mar 2014 Ping Li, Michael Mitzenmacher, Anshumali Shrivastava

This technical note compares two coding (quantization) schemes for random projections in the context of sub-linear time approximate near neighbor search.

Quantization

Beyond Pairwise: Provably Fast Algorithms for Approximate k-Way Similarity Search

no code implementations NeurIPS 2013 Anshumali Shrivastava, Ping Li

We go beyond the notion of pairwise similarity and look into search problems with $k$-way similarity functions.

Retrieval

Coding for Random Projections

no code implementations9 Aug 2013 Ping Li, Michael Mitzenmacher, Anshumali Shrivastava

The method of random projections has become very popular for large-scale applications in statistical learning, information retrieval, bio-informatics and other applications.

Information Retrieval Quantization +1

Hashing Algorithms for Large-Scale Learning

no code implementations NeurIPS 2011 Ping Li, Anshumali Shrivastava, Joshua L. Moore, Arnd C. König

Minwise hashing is a standard technique in the context of search for efficiently computing set similarities.

Cannot find the paper you are looking for? You can Submit a new open access paper.