Search Results for author:
Found 35 papers, 24 papers with code
Toward Inference-optimal Mixture-of-Expert Large Language Models
Like dense models, training MoEs requires answering the same question: given a training budget, what is the optimal allocation on the model size and number of tokens?
AutoGuide: Automated Generation and Selection of State-Aware Guidelines for Large Language Model Agents
The primary limitation of large language models (LLMs) is their restricted understanding of the world.
Data Engineering for Scaling Language Models to 128K Context
We demonstrate that continual pretraining of the full model on 1B-5B tokens of such data is an effective and affordable strategy for scaling the context length of language models to 128K.
OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from 650M to 34B parameters and trained on up to over 1T tokens.
ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models
This paper presents ServerlessLLM, a locality-enhanced serverless inference system for Large Language Models (LLMs).
MoE-Infinity: Activation-Aware Expert Offloading for Efficient MoE Serving
This paper presents MoE-Infinity, a cost-efficient mixture-of-expert (MoE) serving system that realizes activation-aware expert offloading.
Critical Data Size of Language Models from a Grokking Perspective
We formalize the phase transition under the grokking configuration into the Data Efficiency Hypothesis and identify data insufficiency, sufficiency, and surplus regimes in language models training dynamics.
FiLM: Fill-in Language Models for Any-Order Generation
Language models have become the backbone of today's AI systems.
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
The MAmmoTH models are trained on MathInstruct, our meticulously curated instruction tuning dataset.
Go Beyond Imagination: Maximizing Episodic Reachability with World Models
Efficient exploration is a challenging topic in reinforcement learning, especially for sparse reward tasks.
Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
As large language models (LLMs) are continuously being developed, their evaluation becomes increasingly important yet challenging.
Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback
We study whether multiple large language models (LLMs) can autonomously improve each other in a negotiation game by playing, reflecting, and criticizing.
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context.
Specializing Smaller Language Models towards Multi-Step Reasoning
by paying the price of decreased generic ability, we can clearly lift up the scaling curve of models smaller than 10B towards a specialized multi-step math reasoning ability.
TorchOpt: An Efficient Library for Differentiable Optimization
TorchOpt further provides a high-performance distributed execution runtime.
Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTE
We hypothesize that to perform this task well, the reader needs to mentally elaborate the scene being described to identify a sensible meaning of the language.
Decomposed Prompting: A Modular Approach for Solving Complex Tasks
On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks.
Complexity-Based Prompting for Multi-Step Reasoning
In this work, we propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning.
Latent Topology Induction for Understanding Contextualized Representations
With the induced network, we: (1).
Few-shot Subgoal Planning with Language Models
Pre-trained large language models have shown successful progress in many language understanding benchmarks.
Data-to-text Generation with Variational Sequential Planning
We consider the task of data-to-text generation, which aims to create textual output from non-linguistic input.
Experimental quantum advantage with quantum coupon collector
The information transmitted by the proposed game also broke the classical limit.
Scaling Structured Inference with Randomization
Here, we propose a family of randomized dynamic programming (RDP) algorithms for scaling structured models to tens of thousands of latent states.
Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation
Knowledge distillation (KD) has recently been identified as a method that can unintentionally leak private information regarding the details of a teacher model to an unauthorized student.
Discovering Latent Network Topology in Contextualized Representations with Randomized Dynamic Programming
We use RDP to analyze the representation space of pretrained language models, discovering a large-scale latent network in a fully unsupervised way.
Optimizing the Numbers of Queries and Replies in Federated Learning with Differential Privacy
In this work, we study a crucial question which has been vastly overlooked by existing works: what are the optimal numbers of queries and replies in FL with DP so that the final model accuracy is maximized.
Noisy-Labeled NER with Confidence Estimation
This work studies NER under a noisy labeled setting with calibrated confidence estimation.
Probing BERT in Hyperbolic Spaces
We introduce a Poincare probe, a structural probe projecting these embeddings into a Poincare subspace with explicitly defined hierarchies.
Prototypical Representation Learning for Relation Extraction
This approach allows us to learn meaningful, interpretable prototypes for the final classification.
On the Practicality of Differential Privacy in Federated Learning by Tuning Iteration Times
Then, we theoretically derive: 1) the conditions for the DP based FedAvg to converge as the number of global iterations (GI) approaches infinity; 2) the method to set the number of local iterations (LI) to minimize the negative influence of DP noises.
Nested Named Entity Recognition with Partially-Observed TreeCRFs
With the TreeCRF we achieve a uniform way to jointly model the observed and the latent nodes.
Ranked #11 on
Nested Named Entity Recognition
on ACE 2005
Latent Template Induction with Gumbel-CRFs
Learning to control the structure of sentences is a challenging problem in text generation.
Paraphrase Generation with Latent Bag of Words
Inspired by variational autoencoders with discrete latent structures, in this work, we propose a latent bag of words (BOW) model for paraphrase generation.
Rethinking Text Attribute Transfer: A Lexical Analysis
We apply this framework to existing datasets and models and show that: (1) the pivot words are strong features for the classification of sentence attributes; (2) to change the attribute of a sentence, many datasets only requires to change certain pivot words; (3) consequently, many transfer models only perform the lexical-level modification, while leaving higher-level sentence structures unchanged.
Natural Answer Generation with Heterogeneous Memory
Memory augmented encoder-decoder framework has achieved promising progress for natural language generation tasks.
