Instruction Embedding: Latent Representations of Instructions Towards Task Identification

no code implementations • 29 Sep 2024 • Yiwei Li, Jiayi Shi, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Boyuan Pan, HeDa Wang, Yao Hu, Kan Li

In this work, we introduce a new concept, instruction embedding, and construct Instruction Embedding Benchmark (IEB) for its training and evaluation.

Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents

1 code implementation • 8 Mar 2024 • Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Yurong Wu, Chenhao Ma, Jian-Guang Lou, Reynold Cheng

The challenges and costs of collecting realistic interactive logs for data analysis hinder the quantitative evaluation of Large Language Model (LLM) agents in this task.

Benchmarking Decision Making +2