Search Results for author:
Found 28 papers, 7 papers with code
FLuRKA: Fast fused Low-Rank & Kernel Attention
FLuRKA provide sizable performance gains over these approximate techniques and are of high quality.
Brainformers: Trading Simplicity for Efficiency
Using this insight, we develop a complex block, named Brainformer, that consists of a diverse sets of layers such as sparsely gated feed-forward layers, dense feed-forward layers, attention layers, and various forms of layer normalization and activation functions.
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Sparse Mixture-of-Experts (MoE) is a neural architecture design that can be utilized to add learnable parameters to Large Language Models (LLMs) without increasing inference cost.
GiPH: Generalizable Placement Learning for Adaptive Heterogeneous Computing
Careful placement of a computational application within a target device cluster is critical for achieving low application completion time.
Learning Large Graph Property Prediction via Graph Segment Training
Here we propose Graph Segment Training (GST), a general framework that utilizes a divide-and-conquer approach to allow learning large graph property prediction with a constant memory footprint.
Lifelong Language Pretraining with Distribution-Specialized Experts
Compared to existing lifelong learning approaches, Lifelong-MoE achieves better few-shot performance on 19 downstream NLP tasks.
TripLe: Revisiting Pretrained Model Reuse and Progressive Learning for Efficient Vision Transformer Scaling and Searching
One promising way to accelerate transformer training is to reuse small pretrained models to initialize the transformer, as their existing representation power facilitates faster model convergence.
Toward Edge-Efficient Dense Predictions with Synergistic Multi-Task Neural Architecture Search
Our first key insight is that MultiTask Learning (MTL) and hardware-aware Neural Architecture Search (NAS) can work in synergy to greatly benefit on-device Dense Predictions (DP).
Ranked #105 on
Semantic Segmentation
on NYU Depth v2
Hardware Aware Neural Architecture Search
Neural Architecture Search
+2
Searching for Efficient Neural Architectures for On-Device ML on Edge TPUs
On-device ML accelerators are becoming a standard in modern mobile system-on-chips (SoC).
Mixture-of-Experts with Expert Choice Routing
Prior work allocates a fixed number of experts to each token using a top-k function regardless of the relative importance of different tokens.
LaMDA: Language Models for Dialog Applications
We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.
Ranked #113 on
Code Generation
on HumanEval
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing.
Ranked #10 on
Language Modelling
on LAMBADA
A Transferable Approach for Partitioning Machine Learning Models on Multi-Chip-Modules
Partitioning ML graphs for MCMs is particularly hard as the search space grows exponentially with the number of chiplets available and the number of nodes in the neural network.
Do Transformer Modifications Transfer Across Implementations and Applications?
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption.
Rethinking Co-design of Neural Architectures and Hardware Accelerators
In our work, we target the optimization of hardware and software configurations on an industry-standard edge accelerator.
Apollo: Transferable Architecture Exploration
We further show that by transferring knowledge between target architectures with different design constraints, Apollo is able to find optimal configurations faster and often with better objective value (up to 25% improvements).
NAHAS: Neural Architecture and Hardware Accelerator Search
In this paper, we study the importance of co-designing neural architectures and hardware accelerators.
Transferable Graph Optimizers for ML Compilers
Most compilers for machine learning (ML) frameworks need to solve many correlated optimization problems to generate efficient machine code.
A Learned Performance Model for Tensor Processing Units
Accurate hardware performance models are critical to efficient code generation.
ODE-CNN: Omnidirectional Depth Extension Networks
Omnidirectional 360{\deg} camera proliferates rapidly for autonomous robots since it significantly enhances the perception ability by widening the field of view(FoV).
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP).
Ranked #1 on
Sentiment Analysis
on SST-2 Binary classification
GDP: Generalized Device Placement for Dataflow Graphs
Runtime and scalability of large neural networks can be significantly affected by the placement of operations in their dataflow graphs on suitable devices.
EPNAS: Efficient Progressive Neural Architecture Search
In this paper, we propose Efficient Progressive Neural Architecture Search (EPNAS), a neural architecture search (NAS) that efficiently handles large search space through a novel progressive search policy with performance prediction based on REINFORCE~\cite{Williams. 1992. PG}.
A Proposed Hierarchy of Deep Learning Tasks
As the pace of deep learning innovation accelerates, it becomes increasingly important to organize the space of problems by relative difficultly.
Neural Voice Cloning with a Few Samples
Speaker adaptation is based on fine-tuning a multi-speaker generative model with a few cloning samples.
HybridNet: A Hybrid Neural Architecture to Speed-up Autoregressive Models
This paper introduces HybridNet, a hybrid neural network to speed-up autoregressive models for raw audio waveform generation.
Deep Learning Scaling is Predictable, Empirically
As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art.
Deep Voice 2: Multi-Speaker Neural Text-to-Speech
We introduce Deep Voice 2, which is based on a similar pipeline with Deep Voice 1, but constructed with higher performance building blocks and demonstrates a significant audio quality improvement over Deep Voice 1.
