Search Results for author:
Found 20 papers, 7 papers with code
MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
Large language models (LLMs) are typically trained on general source data for various domains, but a recent surge in domain-specific LLMs has shown their potential to outperform general-purpose models in domain-specific tasks (e. g., biomedicine).
Ranked #10 on
Question Answering
on PubMedQA
Position Interpolation Improves ALiBi Extrapolation
Linear position interpolation helps pre-trained models using rotary position embeddings (RoPE) to extrapolate to longer sequence lengths.
BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
BTLM-3B-8K is available under an Apache 2. 0 license on Hugging Face: https://huggingface. co/cerebras/btlm-3b-8k-base.
SlimPajama-DC: Understanding Data Combinations for LLM Training
This paper aims to understand the impacts of various data combinations (e. g., web text, wikipedia, github, books) on the training of large language models using SlimPajama.
Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
We release two open versions of the model -- the foundation Jais model, and an instruction-tuned Jais-chat variant -- with the aim of promoting research on Arabic LLMs.
Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster
We study recent research advances that improve large language models through efficient pre-training and scaling, and open datasets and tools.
RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network
However, training them requires substantial accelerator memory for saving large, multi-resolution activations.
Ranked #310 on
Image Classification
on ImageNet
(using extra training data)
Time and the Value of Data
In addition, we argue that increasing the stock of data by including older datasets may, in fact, damage the model's accuracy.
Time Dependency, Data Flow, and Competitive Advantage
Relating the text topics to various business areas of interest, we argue that competing in a business area in which data value decays rapidly alters strategies to acquire competitive advantage.
Efficiently Disentangle Causal Representations
This paper proposes an efficient approach to learning disentangled representations with causal mechanisms based on the difference of conditional probabilities in original and new distributions.
Memory Efficient 3D U-Net with Reversible Mobile Inverted Bottlenecks for Brain Tumor Segmentation
The inverted residual bottleneck block uses lightweight depthwise separable convolutions to reduce computation by decomposing convolutions into a pointwise convolution and a depthwise convolution.
Gradient Descent Resists Compositionality
In this paper, we argue that gradient descent is one of the reasons that make compositionality learning hard during neural network optimization.
Transferability of Compositionality
To our best knowledge, this is the first work to focus on the transferability of compositionality, and it is orthogonal to existing efforts of learning compositional representations in training distribution.
Pipelined Backpropagation at Scale: Training Large Models without Batches
New hardware can substantially increase the speed and efficiency of deep neural network training.
Compositional Generalization for Primitive Substitutions
Compositional generalization is a basic mechanism in human language learning, but current neural networks lack such ability.
Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
However, recent prior work shows that as dataset sizes grow, DL model accuracy and model size grow predictably.
Empirically Characterizing Overparameterization Impact on Convergence
A long-held conventional wisdom states that larger models train more slowly when using gradient descent.
A Proposed Hierarchy of Deep Learning Tasks
As the pace of deep learning innovation accelerates, it becomes increasingly important to organize the space of problems by relative difficultly.
Deep Learning Scaling is Predictable, Empirically
As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art.
Convolutional Recurrent Neural Networks for Small-Footprint Keyword Spotting
Keyword spotting (KWS) constitutes a major component of human-technology interfaces.
