Search Results for author:
Found 29 papers, 23 papers with code
Pass off Fish Eyes for Pearls: Attacking Model Selection of Pre-trained Models
To validate our viewpoints, we design two methods to evaluate the robustness of FMS: (1) model disguise attack, which post-trains an inferior PTM with a contrastive objective, and (2) evaluation data selection, which selects a subset of the data points for FMS evaluation based on K-means clustering.
ML-Bench: Large Language Models Leverage Open-source Libraries for Machine Learning Tasks
Instead of evaluating LLMs to code from scratch, this work aims to propose a new evaluation setup where LLMs use open-source libraries to finish machine learning tasks.
ProAgent: From Robotic Process Automation to Agentic Process Automation
Empirical experiments are conducted to detail its construction and execution procedure of workflow, showcasing the feasibility of APA, unveiling the possibility of a new paradigm of automation driven by agents.
Large Language Model as Autonomous Decision Maker
While large language models (LLMs) exhibit impressive language understanding and in-context learning abilities, their decision-making ability still heavily relies on the guidance of task-specific expert knowledge when solving real-world tasks.
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
Autonomous agents empowered by Large Language Models (LLMs) have undergone significant improvements, enabling them to generalize across a broad spectrum of tasks.
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction.
Exploring Format Consistency for Instruction Tuning
We propose a framework called "Unified Instruction Tuning" (UIT), which calls OpenAI APIs for automatic format transfer among different instruction tuning datasets.
Arbitrary Few Parameters are Good Enough for Adapting Large-scale Pre-trained Language Models
From our investigations, we find that the model scaling (1) mitigates the effects of the arbitrary module structure on the performance of tuning methods, and (2) enables the tuning methods to optimize fewer parameters to achieve the full-parameter fine-tuning performance.
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
Fine-tuning on instruction data has been widely validated as an effective practice for implementing chat language models like ChatGPT.
CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Additionally, we introduce the Creation Challenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs' tool creation ability.
Recyclable Tuning for Continual Pre-training
In pilot studies, we find that after continual pre-training, the upgraded PLM remains compatible with the outdated adapted weights to some extent.
WebCPM: Interactive Web Search for Chinese Long-form Question Answering
We recruit annotators to search for relevant information using our interface and then answer questions.
Tool Learning with Foundation Models
Considering the lack of a systematic tool learning evaluation in prior works, we experiment with 18 representative tools and show the potential of current foundation models in skillfully utilizing tools.
Language-Specific Representation of Emotion-Concept Knowledge Causally Supports Emotion Inference
Understanding how language supports emotion inference remains a topic of debate in emotion science.
FPT: Improving Prompt Tuning Efficiency via Progressive Training
Inspired by these observations, we propose Fast Prompt Tuning (FPT), which starts by conducting PT using a small-scale partial PLM, and then progressively expands its depth and width until the full-model size.
Exploring Mode Connectivity for Pre-trained Language Models
(3) How does the PLM's task knowledge change along the path connecting two minima?
Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuning
To fathom the mystery, we hypothesize that the adaptations of different DETs could all be reparameterized as low-dimensional optimizations in a unified optimization subspace, which could be found by jointly decomposing independent solutions of different DETs.
ProQA: Structural Prompt-based Pre-training for Unified Question Answering
Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models
This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, dubbed as delta tuning in this paper.
ELLE: Efficient Lifelong Pre-training for Emerging Data
We experiment ELLE with streaming data from 5 domains on BERT and GPT.
On Transferability of Prompt Tuning for Natural Language Processing
To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work.
Exploring Universal Intrinsic Task Subspace via Prompt Tuning
In the experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 250-dimensional subspace found with 100 tasks, by only tuning 250 free parameters, we can recover 97% and 83% of the full prompt tuning performance for 100 seen tasks (using different training data) and 20 unseen tasks, respectively, showing great generalization ability of the found intrinsic task subspace.
bert2BERT: Towards Reusable Pretrained Language Models
However, large language model pre-training costs intensive computational resources and most of the models are trained from scratch without reusing the existing pre-trained models, which is wasteful.
Knowledge Inheritance for Pre-trained Language Models
Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs.
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks.
CPM: A Large-scale Generative Chinese Pre-trained Language Model
However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus of GPT-3 is primarily English, and the parameters are not publicly available.
Learning from Explanations with Neural Execution Tree
While deep neural networks have achieved impressive performance on a range of NLP tasks, these data-hungry models heavily rely on labeled data, which restricts their applications in scenarios where data annotation is expensive.
Improving Sequence Modeling Ability of Recurrent Neural Networks via Sememes
Sememes, the minimum semantic units of human languages, have been successfully utilized in various natural language processing applications.
