Search Results for author:
Found 32 papers, 24 papers with code
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
Pretraining data of large language models composes multiple domains (e. g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome models.
In-Memory Learning: A Declarative Learning Framework for Large Language Models
The exploration of whether agents can align with their environment without relying on human-labeled data presents an intriguing research topic.
Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT
Sparse dictionary learning has been a rapidly growing technique in mechanistic interpretability to attack superposition and extract more human-understandable features from model activations.
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music.
LLM can Achieve Self-Regulation via Hyperparameter Aware Generation
HAG extends the current paradigm in the text generation process, highlighting the feasibility of endowing the LLMs with self-regulate decoding strategies.
Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
To address the dropped tokens and padding, we propose the Rectify-Router, comprising the Intra-GPU Rectification and the Fill-in Rectification.
DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning
These methods regularize the representation space by pulling similar sentence representations closer and pushing away the dissimilar ones and have been proven effective in various NLP tasks, e. g., semantic textual similarity (STS) tasks.
Can AI Assistants Know What They Don't Know?
To answer this question, we construct a model-specific "I don't know" (Idk) dataset for an assistant, which contains its known and unknown questions, based on existing open-domain question answering datasets.
Agent Alignment in Evolving Social Norms
Agents based on Large Language Models (LLMs) are increasingly permeating various domains of human production and life, highlighting the importance of aligning them with human values.
LLatrieval: LLM-Verified Retrieval for Verifiable Generation
Thus, the LLM can iteratively provide feedback to retrieval and facilitate the retrieval result to fully support verifiable generation.
Flames: Benchmarking Value Alignment of LLMs in Chinese
The widespread adoption of large language models (LLMs) across various regions underscores the urgent need to evaluate their alignment with human values.
Evaluating Hallucinations in Chinese Large Language Models
We analyze the primary types of hallucinations in different types of models and their causes.
Secrets of RLHF in Large Language Models Part I: PPO
Therefore, we explore the PPO-max, an advanced version of PPO algorithm, to efficiently improve the training stability of the policy model.
CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors
A common practice is to recast the task into a text-to-text format such that generative LLMs of natural language (NL-LLMs) like GPT-3 can be prompted to solve it.
Improving Contrastive Learning of Sentence Embeddings from AI Feedback
Our method utilizes AI feedback from large pre-trained language models (LLMs) to construct sample pairs with fine-grained sample similarity scores to improve contrastive learning.
Origin Tracing and Detecting of LLMs
The extraordinary performance of large language models (LLMs) heightens the importance of detecting whether the context is generated by an AI system.
DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models
We present DiffusionBERT, a new generative masked language model based on discrete diffusion models.
Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts
Through extensive experimental results across various tasks and PTMs, we show that LPT can achieve competitive performance to full model tuning and other PETuning methods under both full-data and few-shot scenarios while possessing faster training speed and lower memory cost.
Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning
MP2 is a set of combinable prompts pre-trained on 38 Chinese tasks.
BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation
Automatic evaluation metrics are crucial to the development of generative systems.
BBTv2: Towards a Gradient-Free Future with Large Language Models
By contrast, gradient-free methods only require the forward computation of the PTM to tune the prompt, retaining the benefits of efficient tuning and deployment.
A Simple Hash-Based Early Exiting Approach For Language Understanding and Generation
Previous works usually adopt heuristic metrics such as the entropy of internal outputs to measure instance difficulty, which suffers from generalization and threshold-tuning.
Black-Box Tuning for Language-Model-as-a-Service
In such a scenario, which we call Language-Model-as-a-Service (LMaaS), the gradients of PTMs are usually unavailable.
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline
ELUE is dedicated to depict the Pareto Frontier for various language understanding tasks, such that it can tell whether and how much a method achieves Pareto improvement.
Learning to Teach with Student Feedback
This one-way interaction leads to the teacher's inability to perceive the characteristics of the student and its training progress.
Early Exiting with Ensemble Internal Classifiers
In this paper, we show that a novel objective function for the training of the ensemble internal classifiers can be naturally induced from the perspective of ensemble learning and information theory.
Accelerating BERT Inference for Sequence Labeling via Early-Exit
To alleviate this problem, we extend the recent successful early-exit mechanism to accelerate the inference of PTMs for sequence labeling tasks.
Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa
In this paper, we firstly compare the induced trees from PTMs and the dependency parsing trees on several popular models for the ABSA task, showing that the induced tree from fine-tuned RoBERTa (FT-RoBERTa) outperforms the parser-provided tree.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+1
CoLAKE: Contextualized Language and Knowledge Embedding
With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the performance gains of these models.
Pre-trained Models for Natural Language Processing: A Survey
Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era.
Learning Sparse Sharing Architectures for Multiple Tasks
Most existing deep multi-task learning models are based on parameter sharing, such as hard sharing, hierarchical sharing, and soft sharing.
