Search Results for author:
Found 4 papers, 2 papers with code
Pre-training Small Base LMs with Fewer Tokens
Here we show that smaller LMs trained utilizing some of the layers of GPT2-medium (355M) and GPT-2-large (770M) can effectively match the val loss of their bigger counterparts when trained from scratch for the same number of training steps on OpenWebText dataset with 9B tokens.
Early Weight Averaging meets High Learning Rates for LLM Pre-training
Specifically, we pre-trained nanoGPT-2 models of varying sizes, small (125M), medium (335M), and large (770M)on the OpenWebText dataset, comprised of 9B tokens.
Data Aggregation Techniques for Internet of Things
The development of such an ambitious design involves many open challenges, this proposal envisions three major open challenges for IoT data aggregation: first, severe resource constraints of IoT nodes due to limited power and computational ability, second, the highly uncertain (unreliable) raw IoT data is not fit for decisionmaking and third, network latency and privacy issue for critical applications.
A Federated Filtering Framework for Internet of Medical Things
Based on the dominant paradigm, all the wearable IoT devices used in the healthcare sector also known as the internet of medical things (IoMT) are resource constrained in power and computational capabilities.
Networking and Internet Architecture
