Search Results for author: Hanlin Tang

Found 35 papers, 10 papers with code

EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs

no code implementations5 Mar 2024 Hanlin Tang, Yifu Sun, Decheng Wu, Kai Liu, Jianchen Zhu, Zhanhui Kang

To our best knowledge, we are the first work that achieves almost lossless quantization performance for LLMs under a data-independent setting and our algorithm runs over 10 times faster than the data-dependent methods.

Data Free Quantization

MKQ-BERT: Quantized BERT with 4-bits Weights and Activations

no code implementations25 Mar 2022 Hanlin Tang, Xipeng Zhang, Kai Liu, Jianchen Zhu, Zhanhui Kang

In this work, we propose MKQ-BERT, which further improves the compression level and uses 4-bits for quantization.

Quantization

PASTO: Strategic Parameter Optimization in Recommendation Systems -- Probabilistic is Better than Deterministic

no code implementations20 Aug 2021 Weicong Ding, Hanlin Tang, Jingshuo Feng, Lei Yuan, Sen yang, Guangxu Yang, Jie Zheng, Jing Wang, Qiang Su, Dong Zheng, Xuezhong Qiu, Yongqi Liu, Yuxuan Chen, Yang Liu, Chao Song, Dongying Kong, Kai Ren, Peng Jiang, Qiao Lian, Ji Liu

In this setting with multiple and constrained goals, this paper discovers that a probabilistic strategic parameter regime can achieve better value compared to the standard regime of finding a single deterministic parameter.

Recommendation Systems

On the geometry of generalization and memorization in deep neural networks

no code implementations ICLR 2021 Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, SueYeon Chung

Understanding how large neural networks avoid memorizing training data is key to explaining their high generalization performance.

Memorization

Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models

no code implementations ACL (RepL4NLP) 2021 Matteo Alleman, Jonathan Mamou, Miguel A Del Rio, Hanlin Tang, Yoon Kim, SueYeon Chung

While vector-based language representations from pretrained language models have set a new standard for many NLP tasks, there is not yet a complete accounting of their inner workings.

Sentence

1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed

1 code implementation13 Apr 2021 Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, Yuxiong He

To this end, we design a new communication-efficient algorithm, 1-bit LAMB, which introduces a novel way to support adaptive layerwise learning rates under compression.

1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed

2 code implementations4 Feb 2021 Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He

One of the most effective methods is error-compensated compression, which offers robust convergence speed even under 1-bit compression.

Representational correlates of hierarchical phrase structure in deep language models

no code implementations1 Jan 2021 Matteo Alleman, Jonathan Mamou, Miguel A Del Rio, Hanlin Tang, Yoon Kim, SueYeon Chung

Importing from computational and cognitive neuroscience the notion of representational invariance, we perform a series of probes designed to test the sensitivity of Transformer representations to several kinds of structure in sentences.

Sentence

APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm

no code implementations26 Aug 2020 Hanlin Tang, Shaoduo Gan, Samyam Rajbhandari, Xiangru Lian, Ji Liu, Yuxiong He, Ce Zhang

Adam is the important optimization algorithm to guarantee efficiency and accuracy for training many important tasks such as BERT and ImageNet.

Emergence of Separable Manifolds in Deep Language Representations

1 code implementation ICML 2020 Jonathan Mamou, Hang Le, Miguel Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, SueYeon Chung

In addition, we find that the emergence of linear separability in these manifolds is driven by a combined reduction of manifolds' radius, dimensionality and inter-manifold correlations.

Mimic The Raw Domain: Accelerating Action Recognition in the Compressed Domain

no code implementations19 Nov 2019 Barak Battash, Haim Barad, Hanlin Tang, Amit Bleiweiss

In this paper we are approaching the task in a completely different way; we are looking at the data from the compressed stream as a one unit clip and propose that the residual frames can replace the original RGB frames from the raw domain.

Action Recognition Video Recognition +1

Central Server Free Federated Learning over Single-sided Trust Social Networks

1 code implementation11 Oct 2019 Chaoyang He, Conghui Tan, Hanlin Tang, Shuang Qiu, Ji Liu

However, in many social network scenarios, centralized federated learning is not applicable (e. g., a central agent or server connecting all users may not exist, or the communication cost to the central server is not affordable).

Federated Learning

Using Image Priors to Improve Scene Understanding

no code implementations2 Oct 2019 Brigit Schroeder, Hanlin Tang, Alexandre Alahi

We propose a simple yet effective method for leveraging these image priors to improve semantic segmentation of images from sequential driving datasets.

Autonomous Driving Scene Understanding +2

Triplet-Aware Scene Graph Embeddings

no code implementations19 Sep 2019 Brigit Schroeder, Subarna Tripathi, Hanlin Tang

We see a significant performance increase in both metrics that measure the goodness of layout prediction, mean intersection-over-union (mIoU)(52. 3% vs. 49. 2%) and relation score (61. 7% vs. 54. 1%), after the addition of triplet supervision and data augmentation.

Data Augmentation Graph Embedding +7

$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression

no code implementations17 Jul 2019 Hanlin Tang, Xiangru Lian, Shuang Qiu, Lei Yuan, Ce Zhang, Tong Zhang, Ji Liu

Since the \emph{decentralized} training has been witnessed to be superior to the traditional \emph{centralized} training in the communication restricted scenario, therefore a natural question to ask is "how to apply the error-compensated technology to the decentralized learning to further reduce the communication cost."

Generalization to Novel Objects using Prior Relational Knowledge

no code implementations26 Jun 2019 Varun Kumar Vijay, Abhinav Ganesh, Hanlin Tang, Arjun Bansal

To solve tasks in new environments involving objects unseen during training, agents must reason over prior information about those objects and their relations.

Knowledge Graphs

ATRW: A Benchmark for Amur Tiger Re-identification in the Wild

1 code implementation13 Jun 2019 Shuyuan Li, Jianguo Li, Hanlin Tang, Rui Qian, Weiyao Lin

This paper tries to fill the gap by introducing a novel large-scale dataset, the Amur Tiger Re-identification in the Wild (ATRW) dataset.

Probing emergent geometry in speech models via replica theory

no code implementations28 May 2019 Suchismita Padhy, Jenelle Feather, Cory Stephenson, Oguz Elibol, Hanlin Tang, Josh Mcdermott, SueYeon Chung

The success of deep neural networks in visual tasks have motivated recent theoretical and empirical work to understand how these networks operate.

speech-recognition Speech Recognition

DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression

no code implementations15 May 2019 Hanlin Tang, Xiangru Lian, Chen Yu, Tong Zhang, Ji Liu

For example, under the popular parameter server model for distributed learning, the worker nodes need to send the compressed local gradients to the parameter server, which performs the aggregation.

Compact Scene Graphs for Layout Composition and Patch Retrieval

no code implementations19 Apr 2019 Subarna Tripathi, Sharath Nittur Sridhar, Sairam Sundaresan, Hanlin Tang

Structured representations such as scene graphs serve as an efficient and compact representation that can be used for downstream rendering or retrieval tasks.

Image Generation Retrieval

SpaceNet MVOI: a Multi-View Overhead Imagery Dataset

no code implementations ICCV 2019 Nicholas Weir, David Lindenbaum, Alexei Bastidas, Adam Van Etten, Sean McPherson, Jacob Shermeyer, Varun Kumar, Hanlin Tang

To address this problem, we present an open source Multi-View Overhead Imagery dataset, termed SpaceNet MVOI, with 27 unique looks from a broad range of viewing angles (-32. 5 degrees to 54. 0 degrees).

object-detection Object Detection +1

Decentralized Online Learning: Take Benefits from Others' Data without Sharing Your Own to Track Global Trend

no code implementations29 Jan 2019 Yawei Zhao, Chen Yu, Peilin Zhao, Hanlin Tang, Shuang Qiu, Ji Liu

Decentralized Online Learning (online learning in decentralized networks) attracts more and more attention, since it is believed that Decentralized Online Learning can help the data providers cooperatively better solve their online problems without sharing their private data to a third party or other providers.

Using Scene Graph Context to Improve Image Generation

no code implementations11 Jan 2019 Subarna Tripathi, Anahita Bhiwandiwalla, Alexei Bastidas, Hanlin Tang

Generating realistic images from scene graphs asks neural networks to be able to reason about object relationships and compositionality.

Image Generation from Scene Graphs Open-Ended Question Answering +1

Distributed Learning over Unreliable Networks

no code implementations17 Oct 2018 Chen Yu, Hanlin Tang, Cedric Renggli, Simon Kassing, Ankit Singla, Dan Alistarh, Ce Zhang, Ji Liu

Most of today's distributed machine learning systems assume {\em reliable networks}: whenever two machines exchange information (e. g., gradients or models), the network should guarantee the delivery of the message.

BIG-bench Machine Learning

$D^2$: Decentralized Training over Decentralized Data

no code implementations ICML 2018 Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, Ji Liu

While training a machine learning model using multiple workers, each of which collects data from its own data source, it would be useful when the data collected from different workers are unique and different.

Image Classification Multi-view Subspace Clustering

D$^2$: Decentralized Training over Decentralized Data

no code implementations19 Mar 2018 Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, Ji Liu

While training a machine learning model using multiple workers, each of which collects data from their own data sources, it would be most useful when the data collected from different workers can be {\em unique} and {\em different}.

Image Classification

Communication Compression for Decentralized Training

no code implementations NeurIPS 2018 Hanlin Tang, Shaoduo Gan, Ce Zhang, Tong Zhang, Ji Liu

In this paper, We explore a natural question: {\em can the combination of both techniques lead to a system that is robust to both bandwidth and latency?}

Recurrent computations for visual pattern completion

1 code implementation7 Jun 2017 Hanlin Tang, Martin Schrimpf, Bill Lotter, Charlotte Moerman, Ana Paredes, Josue Ortega Caro, Walter Hardesty, David Cox, Gabriel Kreiman

First, subjects robustly recognized objects even when rendered <15% visible, but recognition was largely impaired when processing was interrupted by backward masking.

Image Classification

Cannot find the paper you are looking for? You can Submit a new open access paper.