Search Results for author: Anshumali Shrivastava

Found 80 papers, 17 papers with code

Learning Multimodal Data Augmentation in Feature Space

no code implementations29 Dec 2022 Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, Andrew Gordon Wilson

The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is a defining feature of intelligent systems.

Data Augmentation Image Classification +1

Distributed SLIDE: Enabling Training Large Neural Networks on Low Bandwidth and Simple CPU-Clusters via Model Parallelism and Sparsity

no code implementations29 Jan 2022 Minghao Yan, Nicholas Meisburger, Tharun Medini, Anshumali Shrivastava

We show that with reduced communication, due to sparsity, we can train close to a billion parameter model on simple 4-16 core CPU nodes connected by basic low bandwidth interconnect.

Locality Sensitive Teaching

no code implementations NeurIPS 2021 Zhaozhuo Xu, Beidi Chen, Chaojian Li, Weiyang Liu, Le Song, Yingyan Lin, Anshumali Shrivastava

However, as one of the most influential and practical MT paradigms, iterative machine teaching (IMT) is prohibited on IoT devices due to its inefficient and unscalable algorithms.

Contractive error feedback for gradient compression

no code implementations29 Sep 2021 Bingcong Li, Shuai Zheng, Parameswaran Raman, Anshumali Shrivastava, Georgios B. Giannakis

On-device memory concerns in distributed deep learning are becoming more severe due to i) the growth of model size in multi-GPU training, and ii) the adoption of neural networks for federated learning on IoT devices with limited storage.

Federated Learning Image Classification +3

Embedding models through the lens of Stable Coloring

no code implementations29 Sep 2021 Aditya Desai, Shashank Sonkar, Anshumali Shrivastava, Richard Baraniuk

Grounded on this framework, we show that many algorithms ranging across different domains are, in fact, searching for continuous stable coloring solutions of an underlying graph corresponding to the domain.

Denoising

STORM: Sketch Toward Online Risk Minimization

no code implementations29 Sep 2021 Gaurav Gupta, Benjamin Coleman, John Chen, Anshumali Shrivastava

To this end, we propose STORM, an online sketching-based method for empirical risk minimization.

Classification regression

Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference

no code implementations4 Aug 2021 Aditya Desai, Li Chou, Anshumali Shrivastava

In this paper, we present Random Offset Block Embedding Array (ROBE) as a low memory alternative to embedding tables which provide orders of magnitude reduction in memory usage while maintaining accuracy and boosting execution speed.

Model Compression

Efficient Inference via Universal LSH Kernel

no code implementations21 Jun 2021 Zichang Liu, Benjamin Coleman, Anshumali Shrivastava

Large machine learning models achieve unprecedented performance on various tasks and have evolved as the go-to technique.

Knowledge Distillation Quantization

Sublinear Least-Squares Value Iteration via Locality Sensitive Hashing

no code implementations18 May 2021 Anshumali Shrivastava, Zhao Song, Zhaozhuo Xu

We present the first provable Least-Squares Value Iteration (LSVI) algorithms that have runtime complexity sublinear in the number of actions.

reinforcement-learning reinforcement Learning

IRLI: Iterative Re-partitioning for Learning to Index

no code implementations17 Mar 2021 Gaurav Gupta, Tharun Medini, Anshumali Shrivastava, Alexander J Smola

Neural models have transformed the fundamental information retrieval problem of mapping a query to a giant set of items.

Information Retrieval Multi-Label Classification +1

Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and More

2 code implementations6 Mar 2021 Shabnam Daghaghi, Nicholas Meisburger, Mengnan Zhao, Yong Wu, Sameh Gobriel, Charlie Tai, Anshumali Shrivastava

Our work highlights several novel perspectives and opportunities for implementing randomized algorithms for deep learning on modern CPUs.

Density Sketches for Sampling and Estimation

no code implementations24 Feb 2021 Aditya Desai, Benjamin Coleman, Anshumali Shrivastava

We introduce Density sketches (DS): a succinct online summary of the data distribution.

Semantically Constrained Memory Allocation (SCMA) for Embedding in Efficient Recommendation Systems

1 code implementation24 Feb 2021 Aditya Desai, Yanzhou Pan, Kuangyuan Sun, Li Chou, Anshumali Shrivastava

In particular, our LMA embeddings achieve the same performance compared to standard embeddings with a 16$\times$ reduction in memory footprint.

Recommendation Systems

MONGOOSE: A Learnable LSH Framework for Efficient Neural Network Training

no code implementations ICLR 2021 Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, Christopher Re

Recent advances by practitioners in the deep learning community have breathed new life into Locality Sensitive Hashing (LSH), using it to reduce memory and time bottlenecks in neural network (NN) training.

Language Modelling Recommendation Systems +1

A Truly Constant-time Distribution-aware Negative Sampling

no code implementations1 Jan 2021 Shabnam Daghaghi, Tharun Medini, Beidi Chen, Mengnan Zhao, Anshumali Shrivastava

Softmax classifiers with a very large number of classes naturally occur in many applications such as natural language processing and information retrieval.

Information Retrieval Retrieval

A Tale of Two Efficient and Informative Negative Sampling Distributions

no code implementations31 Dec 2020 Shabnam Daghaghi, Tharun Medini, Nicholas Meisburger, Beidi Chen, Mengnan Zhao, Anshumali Shrivastava

Unfortunately, due to the dynamically updated parameters and data samples, there is no sampling scheme that is provably adaptive and samples the negative classes efficiently.

Information Retrieval Retrieval

Neighbor Oblivious Learning (NObLe) for Device Localization and Tracking

no code implementations23 Nov 2020 Zichang Liu, Li Chou, Anshumali Shrivastava

In this paper, we argue that the state-of-the-art-systems are significantly worse in terms of accuracy because they are incapable of utilizing these essential structural information.

Learning Sampling Distributions Using Local 3D Workspace Decompositions for Motion Planning in High Dimensions

1 code implementation29 Oct 2020 Constantinos Chamzas, Zachary Kingston, Carlos Quintero-Peña, Anshumali Shrivastava, Lydia E. Kavraki

Earlier work has shown that reusing experience from prior motion planning problems can improve the efficiency of similar, future motion planning queries.

Motion Planning

SOLAR: Sparse Orthogonal Learned and Random Embeddings

no code implementations ICLR 2021 Tharun Medini, Beidi Chen, Anshumali Shrivastava

The label vectors are random, sparse, and near-orthogonal by design, while the query vectors are learned and sparse.

Multi-Label Classification

Bloom Origami Assays: Practical Group Testing

no code implementations21 Jul 2020 Louis Abraham, Gary Becigneul, Benjamin Coleman, Bernhard Scholkopf, Anshumali Shrivastava, Alexander Smola

Group testing is a well-studied problem with several appealing solutions, but recent biological studies impose practical constraints for COVID-19 that are incompatible with traditional methods.

Climbing the WOL: Training for Cheaper Inference

no code implementations2 Jul 2020 Zichang Liu, Zhaozhuo Xu, Alan Ji, Jonathan Li, Beidi Chen, Anshumali Shrivastava

Efficient inference for wide output layers (WOLs) is an essential yet challenging task in large scale machine learning.

Retrieval

A One-Pass Private Sketch for Most Machine Learning Tasks

no code implementations16 Jun 2020 Benjamin Coleman, Anshumali Shrivastava

Existing methods for DP kernel density estimation scale poorly, often exponentially slower with an increase in dimensions.

BIG-bench Machine Learning Density Estimation

Privacy Adversarial Network: Representation Learning for Mobile Data Privacy

no code implementations8 Jun 2020 Sicong Liu, Junzhao Du, Anshumali Shrivastava, Lin Zhong

This work departs from prior works in methodology: we leverage adversarial learning to a better balance between privacy and utility.

Representation Learning

Sub-linear RACE Sketches for Approximate Kernel Density Estimation on Streaming Data

no code implementations4 Dec 2019 Benjamin Coleman, Anshumali Shrivastava

We evaluate our method on real-world high-dimensional datasets and show that our sketch achieves 10x better compression compared to competing methods.

Density Estimation

FourierSAT: A Fourier Expansion-Based Algebraic Framework for Solving Hybrid Boolean Constraints

2 code implementations2 Dec 2019 Anastasios Kyrillidis, Anshumali Shrivastava, Moshe Y. Vardi, Zhiwei Zhang

By such a reduction to continuous optimization, we propose an algebraic framework for solving systems consisting of different types of constraints.

Fast and Accurate Stochastic Gradient Estimation

1 code implementation NeurIPS 2019 Beidi Chen, Yingchen Xu, Anshumali Shrivastava

In this paper, we break this barrier by providing the first demonstration of a scheme, Locality sensitive hashing (LSH) sampled Stochastic Gradient Descent (LGD), which leads to superior gradient estimation while keeping the sampling cost per iteration similar to that of the uniform sampling.

Lsh-sampling Breaks the Computation Chicken-and-egg Loop in Adaptive Stochastic Gradient Estimation

no code implementations30 Oct 2019 Beidi Chen, Yingchen Xu, Anshumali Shrivastava

In this paper, we break this barrier by providing the first demonstration of a scheme, Locality sensitive hashing (LSH) sampled Stochastic Gradient Descent (LGD), which leads to superior gradient estimation while keeping the sampling cost per iteration similar to that of the uniform sampling.

Adaptive Learned Bloom Filter (Ada-BF): Efficient Utilization of the Classifier

no code implementations NeurIPS 2020 Zhenwei Dai, Anshumali Shrivastava

Recent work suggests improving the performance of Bloom filter by incorporating a machine learning model as a binary classifier.

BIG-bench Machine Learning

SDM-Net: A Simple and Effective Model for Generalized Zero-Shot Learning

no code implementations10 Sep 2019 Shabnam Daghaghi, Tharun Medini, Anshumali Shrivastava

Zero-Shot Learning (ZSL) is a classification task where we do not have even a single training labeled example from a set of unseen classes.

General Classification Generalized Zero-Shot Learning +2

Revisiting Consistent Hashing with Bounded Loads

1 code implementation23 Aug 2019 John Chen, Ben Coleman, Anshumali Shrivastava

We show, both theoretically and empirically, that our proposed solution is significantly superior for load balancing and is optimal in many senses.

Data Structures and Algorithms

Using Local Experiences for Global Motion Planning

no code implementations20 Mar 2019 Constantinos Chamzas, Anshumali Shrivastava, Lydia E. Kavraki

In this work, we decompose the workspace into local primitives, memorizing local experiences by these primitives in the form of local samplers, and store them in a database.

Motion Planning

Sub-linear Memory Sketches for Near Neighbor Search on Streaming Data

no code implementations18 Feb 2019 Benjamin Coleman, Richard G. Baraniuk, Anshumali Shrivastava

We present the first sublinear memory sketch that can be queried to find the nearest neighbors in a dataset.

Density Estimation

Compressing Gradient Optimizers via Count-Sketches

1 code implementation1 Feb 2019 Ryan Spring, Anastasios Kyrillidis, Vijai Mohan, Anshumali Shrivastava

The problem is becoming more severe as deep learning models continue to grow larger in order to learn from complex, large-scale datasets.

Better accuracy with quantified privacy: representations learned via reconstructive adversarial network

no code implementations ICLR 2019 Sicong Liu, Anshumali Shrivastava, Junzhao Du, Lin Zhong

This work represents a methodical departure from prior works: we balance between a measure of privacy and another of utility by leveraging adversarial learning to find a sweeter tradeoff.

BIG-bench Machine Learning General Classification

Topkapi: Parallel and Fast Sketches for Finding Top-K Frequent Elements

1 code implementation NeurIPS 2018 Ankush Mandal, He Jiang, Anshumali Shrivastava, Vivek Sarkar

In particular, for identifying top-K frequent items, Count-Min Sketch (CMS) has fantastic update time but lack the important property of reducibility which is needed for exploiting available massive data parallelism.

Probabilistic Blocking with An Application to the Syrian Conflict

no code implementations11 Oct 2018 Rebecca C. Steorts, Anshumali Shrivastava

Entity resolution seeks to merge databases as to remove duplicate entries where unique identifiers are typically unknown.

Entity Resolution Information Retrieval +1

Extreme Classification in Log Memory

no code implementations9 Oct 2018 Qixuan Huang, Yiqiu Wang, Tharun Medini, Anshumali Shrivastava

With MACH we can train ODP dataset with 100, 000 classes and 400, 000 features on a single Titan X GPU, with the classification accuracy of 19. 28%, which is the best-reported accuracy on this dataset.

Classification General Classification

Mimicking actions is a good strategy for beginners: Fast Reinforcement Learning with Expert Action Sequences

no code implementations27 Sep 2018 Tharun Medini, Anshumali Shrivastava

Imitation Learning is the task of mimicking the behavior of an expert player in a Reinforcement Learning(RL) Environment to enhance the training of a fresh agent (called novice) beginning from scratch.

Atari Games Imitation Learning +2

Ultra Large-Scale Feature Selection using Count-Sketches

1 code implementation ICML 2018 Amirali Aghazadeh, Ryan Spring, Daniel LeJeune, Gautam Dasarathy, Anshumali Shrivastava, baraniuk

We demonstrate that MISSION accurately and efficiently performs feature selection on real-world, large-scale datasets with billions of dimensions.

BIG-bench Machine Learning

MISSION: Ultra Large-Scale Feature Selection using Count-Sketches

1 code implementation12 Jun 2018 Amirali Aghazadeh, Ryan Spring, Daniel LeJeune, Gautam Dasarathy, Anshumali Shrivastava, Richard G. Baraniuk

We demonstrate that MISSION accurately and efficiently performs feature selection on real-world, large-scale datasets with billions of dimensions.

BIG-bench Machine Learning

Scaling-up Split-Merge MCMC with Locality Sensitive Sampling (LSS)

no code implementations21 Feb 2018 Chen Luo, Anshumali Shrivastava

It is well known that state-of-the-art methods for split-merge MCMC do not scale well.

MACH: Embarrassingly parallel $K$-class classification in $O(d\log{K})$ memory and $O(K\log{K} + d\log{K})$ time, instead of $O(Kd)$

no code implementations ICLR 2018 Qixuan Huang, Anshumali Shrivastava, Yiqiu Wang

MACH is the first generic $K$-classification algorithm, with provably theoretical guarantees, which requires $O(\log{K})$ memory without any assumption on the relationship between classes.

Classification General Classification

LSH-SAMPLING BREAKS THE COMPUTATIONAL CHICKEN-AND-EGG LOOP IN ADAPTIVE STOCHASTIC GRADIENT ESTIMATION

no code implementations ICLR 2018 Beidi Chen, Yingchen Xu, Anshumali Shrivastava

In this paper, we break this barrier by providing the first demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to that of the uniform sampling.

Accelerating Dependency Graph Learning from Heterogeneous Categorical Event Streams via Knowledge Transfer

no code implementations25 Aug 2017 Chen Luo, Zhengzhang Chen, Lu-An Tang, Anshumali Shrivastava, Zhichun Li

Given a well-trained dependency graph from a source domain and an immature dependency graph from a target domain, how can we extract the entity and dependency knowledge from the source to enhance the target?

Graph Learning Intrusion Detection +1

A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models

no code implementations15 Mar 2017 Ryan Spring, Anshumali Shrivastava

We propose a new sampling scheme and an unbiased estimator that estimates the partition function accurately in sub-linear time.

Optimal Densification for Fast and Accurate Minwise Hashing

no code implementations ICML 2017 Anshumali Shrivastava

Minwise hashing is a fundamental and one of the most successful hashing algorithm in the literature.

Revisiting Winner Take All (WTA) Hashing for Sparse Datasets

no code implementations6 Dec 2016 Beidi Chen, Anshumali Shrivastava

WTA (Winner Take All) hashing has been successfully applied in many large scale vision applications.

General Classification Image Classification +1

Sub-Linear Privacy-Preserving Near-Neighbor Search

no code implementations6 Dec 2016 M. Sadegh Riazi, Beidi Chen, Anshumali Shrivastava, Dan Wallach, Farinaz Koushanfar

In Near-Neighbor Search (NNS), a new client queries a database (held by a server) for the most similar data (near-neighbors) given a certain similarity metric.

Privacy Preserving

Simple and Efficient Weighted Minwise Hashing

no code implementations NeurIPS 2016 Anshumali Shrivastava

Weighted minwise hashing (WMH) is one of the fundamental subroutine, required by many celebrated approximation algorithms, commonly adopted in industrial practice for large -scale search and learning.

SSH (Sketch, Shingle, & Hash) for Indexing Massive-Scale Time Series

1 code implementation24 Oct 2016 Chen Luo, Anshumali Shrivastava

However, branch and bound based pruning are only useful for very short queries (low dimensional time series), and the bounds are quite weak for longer queries.

Dynamic Time Warping Time Series

Scalable and Sustainable Deep Learning via Randomized Hashing

no code implementations26 Feb 2016 Ryan Spring, Anshumali Shrivastava

A unique property of the proposed hashing based back-propagation is that the updates are always sparse.

Asymmetric Minwise Hashing

1 code implementation14 Nov 2014 Anshumali Shrivastava, Ping Li

Minwise hashing (Minhash) is a widely popular indexing scheme in practice.

Retrieval

Improved Asymmetric Locality Sensitive Hashing (ALSH) for Maximum Inner Product Search (MIPS)

no code implementations20 Oct 2014 Anshumali Shrivastava, Ping Li

In the prior work, the authors use asymmetric transformations which convert the problem of approximate MIPS into the problem of approximate near neighbor search which can be efficiently solved using hashing.

In Defense of MinHash Over SimHash

no code implementations16 Jul 2014 Anshumali Shrivastava, Ping Li

To provide a common basis for comparison, we evaluate retrieval results in terms of $\mathcal{S}$ for both MinHash and SimHash.

Retrieval

Improved Densification of One Permutation Hashing

1 code implementation18 Jun 2014 Anshumali Shrivastava, Ping Li

The existing work on densification of one permutation hashing reduces the query processing cost of the $(K, L)$-parameterized Locality Sensitive Hashing (LSH) algorithm with minwise hashing, from $O(dKL)$ to merely $O(d + KL)$, where $d$ is the number of nonzeros of the data vector, $K$ is the number of hashes in each hash table, and $L$ is the number of hash tables.

Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)

no code implementations NeurIPS 2014 Anshumali Shrivastava, Ping Li

Our proposal is based on an interesting mathematical phenomenon in which inner products, after independent asymmetric transformations, can be converted into the problem of approximate near neighbor search.

Collaborative Filtering

Graph Kernels via Functional Embedding

no code implementations21 Apr 2014 Anshumali Shrivastava, Ping Li

We propose a representation of graph as a functional object derived from the power iteration of the underlying adjacency matrix.

General Classification Graph Classification

A New Space for Comparing Graphs

no code implementations17 Apr 2014 Anshumali Shrivastava, Ping Li

We show that the proposed matrix representation encodes the spectrum of the underlying adjacency matrix and it also contains information about the counts of small sub-structures present in the graph such as triangles and small paths.

General Classification

Coding for Random Projections and Approximate Near Neighbor Search

no code implementations31 Mar 2014 Ping Li, Michael Mitzenmacher, Anshumali Shrivastava

This technical note compares two coding (quantization) schemes for random projections in the context of sub-linear time approximate near neighbor search.

Quantization

Beyond Pairwise: Provably Fast Algorithms for Approximate k-Way Similarity Search

no code implementations NeurIPS 2013 Anshumali Shrivastava, Ping Li

We go beyond the notion of pairwise similarity and look into search problems with $k$-way similarity functions.

Retrieval

Coding for Random Projections

no code implementations9 Aug 2013 Ping Li, Michael Mitzenmacher, Anshumali Shrivastava

The method of random projections has become very popular for large-scale applications in statistical learning, information retrieval, bio-informatics and other applications.

Information Retrieval Quantization +1

Hashing Algorithms for Large-Scale Learning

no code implementations NeurIPS 2011 Ping Li, Anshumali Shrivastava, Joshua L. Moore, Arnd C. König

Minwise hashing is a standard technique in the context of search for efficiently computing set similarities.

Cannot find the paper you are looking for? You can Submit a new open access paper.