Search Results for author:
Found 34 papers, 10 papers with code
MKQ-BERT: Quantized BERT with 4-bits Weights and Activations
In this work, we propose MKQ-BERT, which further improves the compression level and uses 4-bits for quantization.
PASTO: Strategic Parameter Optimization in Recommendation Systems -- Probabilistic is Better than Deterministic
In this setting with multiple and constrained goals, this paper discovers that a probabilistic strategic parameter regime can achieve better value compared to the standard regime of finding a single deterministic parameter.
On the geometry of generalization and memorization in deep neural networks
Understanding how large neural networks avoid memorizing training data is key to explaining their high generalization performance.
Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models
While vector-based language representations from pretrained language models have set a new standard for many NLP tasks, there is not yet a complete accounting of their inner workings.
1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
To this end, we design a new communication-efficient algorithm, 1-bit LAMB, which introduces a novel way to support adaptive layerwise learning rates under compression.
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
One of the most effective methods is error-compensated compression, which offers robust convergence speed even under 1-bit compression.
Representational correlates of hierarchical phrase structure in deep language models
Importing from computational and cognitive neuroscience the notion of representational invariance, we perform a series of probes designed to test the sensitivity of Transformer representations to several kinds of structure in sentences.
APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm
Adam is the important optimization algorithm to guarantee efficiency and accuracy for training many important tasks such as BERT and ImageNet.
Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning
For deep neural network accelerators, memory movement is both energetically expensive and can bound computation.
Emergence of Separable Manifolds in Deep Language Representations
In addition, we find that the emergence of linear separability in these manifolds is driven by a combined reduction of manifolds' radius, dimensionality and inter-manifold correlations.
Untangling in Invariant Speech Recognition
Higher level concepts such as parts-of-speech and context dependence also emerge in the later layers of the network.
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
This necessitates increased memory footprint and computational requirements for training.
Mimic The Raw Domain: Accelerating Action Recognition in the Compressed Domain
In this paper we are approaching the task in a completely different way; we are looking at the data from the compressed stream as a one unit clip and propose that the residual frames can replace the original RGB frames from the raw domain.
Central Server Free Federated Learning over Single-sided Trust Social Networks
However, in many social network scenarios, centralized federated learning is not applicable (e. g., a central agent or server connecting all users may not exist, or the communication cost to the central server is not affordable).
Using Image Priors to Improve Scene Understanding
We propose a simple yet effective method for leveraging these image priors to improve semantic segmentation of images from sequential driving datasets.
MLPerf Training Benchmark
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML.
Triplet-Aware Scene Graph Embeddings
We see a significant performance increase in both metrics that measure the goodness of layout prediction, mean intersection-over-union (mIoU)(52. 3% vs. 49. 2%) and relation score (61. 7% vs. 54. 1%), after the addition of triplet supervision and data augmentation.
$\texttt{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression
Since the \emph{decentralized} training has been witnessed to be superior to the traditional \emph{centralized} training in the communication restricted scenario, therefore a natural question to ask is "how to apply the error-compensated technology to the decentralized learning to further reduce the communication cost."
Generalization to Novel Objects using Prior Relational Knowledge
To solve tasks in new environments involving objects unseen during training, agents must reason over prior information about those objects and their relations.
ATRW: A Benchmark for Amur Tiger Re-identification in the Wild
This paper tries to fill the gap by introducing a novel large-scale dataset, the Amur Tiger Re-identification in the Wild (ATRW) dataset.
Probing emergent geometry in speech models via replica theory
The success of deep neural networks in visual tasks have motivated recent theoretical and empirical work to understand how these networks operate.
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
For example, under the popular parameter server model for distributed learning, the worker nodes need to send the compressed local gradients to the parameter server, which performs the aggregation.
Compact Scene Graphs for Layout Composition and Patch Retrieval
Structured representations such as scene graphs serve as an efficient and compact representation that can be used for downstream rendering or retrieval tasks.
SpaceNet MVOI: a Multi-View Overhead Imagery Dataset
To address this problem, we present an open source Multi-View Overhead Imagery dataset, termed SpaceNet MVOI, with 27 unique looks from a broad range of viewing angles (-32. 5 degrees to 54. 0 degrees).
Heuristics for Image Generation from Scene Graphs
Existing scene graph to image models have two stages: (1) a scene composition stage, and an (2) image generation stage.
Decentralized Online Learning: Take Benefits from Others' Data without Sharing Your Own to Track Global Trend
Decentralized Online Learning (online learning in decentralized networks) attracts more and more attention, since it is believed that Decentralized Online Learning can help the data providers cooperatively better solve their online problems without sharing their private data to a third party or other providers.
Using Scene Graph Context to Improve Image Generation
Generating realistic images from scene graphs asks neural networks to be able to reason about object relationships and compositionality.
Distributed Learning over Unreliable Networks
Most of today's distributed machine learning systems assume {\em reliable networks}: whenever two machines exchange information (e. g., gradients or models), the network should guarantee the delivery of the message.
$D^2$: Decentralized Training over Decentralized Data
While training a machine learning model using multiple workers, each of which collects data from its own data source, it would be useful when the data collected from different workers are unique and different.
Ranked #3 on
Multi-view Subspace Clustering
on ORL
D$^2$: Decentralized Training over Decentralized Data
While training a machine learning model using multiple workers, each of which collects data from their own data sources, it would be most useful when the data collected from different workers can be {\em unique} and {\em different}.
Communication Compression for Decentralized Training
In this paper, We explore a natural question: {\em can the combination of both techniques lead to a system that is robust to both bandwidth and latency?}
Recurrent computations for visual pattern completion
First, subjects robustly recognized objects even when rendered <15% visible, but recognition was largely impaired when processing was interrupted by backward masking.
