Search Results for author:
Found 59 papers, 21 papers with code
Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern
The quadratic computational complexity of the attention mechanism in current Large Language Models (LLMs) renders inference with long contexts prohibitively expensive.
Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
Experiments show that the method improves the actor's exploration efficiency and solution diversity, especially on challenging queries, leading to a stronger reasoning model.
Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning
First, we introduce a predictor-corrector learning framework to minimize truncation errors, which consists of a high-order predictor and a multistep corrector.
Multi-Programming Language Sandbox for LLMs
We introduce MPLSandbox, an out-of-the-box multi-programming language sandbox designed to provide unified and comprehensive feedback from compiler and analysis tools for Large Language Models (LLMs).
FIRP: Faster LLM inference via future intermediate representation prediction
Recent advancements in Large Language Models (LLMs) have shown remarkable performance across a wide range of tasks.
Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning
Textual Attributed Graphs (TAGs) are crucial for modeling complex real-world systems, yet leveraging large language models (LLMs) for TAGs presents unique challenges due to the gap between sequential text processing and graph-structured data.
Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models
The scaling of large language models (LLMs) is a critical research area for the efficiency and effectiveness of model training and deployment.
Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models
We provide an extensive survey for limitations in this work and propose a new method called forgetting curve to measure the memorization capability of long-context models.
Length Desensitization in Direct Preference Optimization
Direct Preference Optimization (DPO) is widely utilized in the Reinforcement Learning from Human Feedback (RLHF) phase to align Large Language Models (LLMs) with human preferences, thereby enhancing both their harmlessness and efficacy.
How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data
Based on our selected data, we present XCoder, a family of models finetuned from LLaMA3.
ReMamba: Equip Mamba with Effective Long-Sequence Modeling
While the Mamba architecture demonstrates superior inference efficiency and competitive performance on short-context natural language processing (NLP) tasks, empirical evidence suggests its capacity to comprehend long contexts is limited compared to transformer-based models.
Graph-Structured Speculative Decoding
We apply GSD across a range of LLMs, including a 70-billion parameter LLaMA-2 model, and observe a remarkable speedup of 1. 73$\times$ to 1. 96$\times$, significantly surpassing standard speculative decoding.
What's Wrong with Your Code Generated by Large Language Models? An Extensive Study
The increasing development of large language models (LLMs) in code generation has drawn significant attention among researchers.
Rethinking LLM-based Preference Evaluation
We decompose the preference evaluation metric, i. e., win rate, from the perspective of human to identify the deeper factors and conclude that the win rate is affected by two axes of model response: desirability and information mass, where the former is length-independent and related to trustworthiness, and the latter is length-dependent and can be represented by conditional entropy.
EAVE: Efficient Product Attribute Value Extraction via Lightweight Sparse-layer Interaction
Specifically, we employ a heavy encoder to separately encode the product context and attribute.
Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
The recent advancements in large language models (LLMs) have been extraordinary, yet the escalating inference costs associated with them present challenges in real-world applications.
Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration
In this paper, we propose a novel parallel decoding approach, namely \textit{hidden transfer}, which decodes multiple successive tokens simultaneously in a single forward pass.
What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation
Our findings reveal several connections between the properties of perturbations and LLM performance, providing insights into the failure cases of uniform quantization and suggesting potential solutions to improve the robustness of LLM quantization.
Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection
Out-of-domain (OOD) intent detection aims to examine whether the user's query falls outside the predefined domain of the system, which is crucial for the proper functioning of task-oriented dialogue (TOD) systems.
C-ICL: Contrastive In-context Learning for Information Extraction
There has been increasing interest in exploring the capabilities of advanced large language models (LLMs) in the field of information extraction (IE), specifically focusing on tasks related to named entity recognition (NER) and relation extraction (RE).
DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning
It learns diverse instruction targets and combines a code evaluation objective to enhance its code generation ability.
Sibyl: Empowering Empathetic Dialogue Generation in Large Language Models via Sensible and Visionary Commonsense Inference
Recently, there has been a heightened interest in building chatbots based on Large Language Models (LLMs) to emulate human-like qualities in multi-turn conversations.
Improving Input-label Mapping with Demonstration Replay for In-context Learning
The effectiveness of ICL can be attributed to the strong language modeling capabilities of large language models (LLMs), which enable them to learn the mapping between input and labels based on in-context demonstrations.
Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression
In particular, our approach extracts knowledge from LLMs to construct a knowledge store, from which the small-scale model can retrieve relevant information and leverage it for effective inference.
APP: Adaptive Prototypical Pseudo-Labeling for Few-shot OOD Detection
Detecting out-of-domain (OOD) intents from user queries is essential for a task-oriented dialogue system.
Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT
The tasks of out-of-domain (OOD) intent discovery and generalized intent discovery (GID) aim to extend a closed intent classifier to open-world intent sets, which is crucial to task-oriented dialogue (TOD) systems.
mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning
A multi-view contrastive learning framework is introduced to encompass semantic contrasts between source, codeswitched, and target sentences, as well as contrasts among token-to-token relations.
MixPAVE: Mix-Prompt Tuning for Few-shot Product Attribute Value Extraction
In this work, we propose a novel prompt tuning approach with Mixed Prompts for few-shot Attribute Value Extraction, namely MixPAVE.
FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue
Pre-trained language models based on general text enable huge success in the NLP scenario.
Seen to Unseen: Exploring Compositional Generalization of Multi-Attribute Controllable Dialogue Generation
In this paper, we explore the compositional generalization for multi-attribute controllable dialogue generation where a model can learn from seen attribute values and generalize to unseen combinations.
GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment on various devices.
PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models
While transformer-based pre-trained language models (PLMs) have dominated a number of NLP applications, these models are heavy to deploy and expensive to use.
Decoupling Pseudo Label Disambiguation and Representation Learning for Generalized Intent Discovery
Previous methods suffer from a coupling of pseudo label disambiguation and representation learning, that is, the reliability of pseudo labels relies on representation learning, and representation learning is restricted by pseudo labels in turn.
RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank
In this paper, we propose a novel approach, RankCSE, for unsupervised sentence representation learning, which incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework.
Task-agnostic Distillation of Encoder-Decoder Language Models
Finetuning pretrained language models (LMs) have enabled appealing performance on a diverse array of tasks.
Lifting the Curse of Capacity Gap in Distilling Language Models
However, when the capacity gap between the teacher and the student is large, a curse of capacity gap appears, invoking a deficiency in distilling LMs.
Multi-task Transformer with Relation-attention and Type-attention for Named Entity Recognition
There are three types of NER tasks, including flat, nested and discontinuous entity recognition.
Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration
Instance segmentation in videos, which aims to segment and track multiple objects in video frames, has garnered a flurry of research attention in recent years.
UniNL: Aligning Representation Learning with Scoring Function for OOD Detection via Unified Neighborhood Learning
Specifically, we design a K-nearest neighbor contrastive learning (KNCL) objective for representation learning and introduce a KNN-based scoring function for OOD detection.
Semi-Supervised Knowledge-Grounded Pre-training for Task-Oriented Dialog Systems
Recent advances in neural approaches greatly improve task-oriented dialogue (TOD) systems which assist users to accomplish their goals.
Watch the Neighbors: A Unified K-Nearest Neighbor Contrastive Learning Framework for OOD Intent Discovery
For OOD clustering stage, we propose a KCC method to form compact clusters by mining true hard negative samples, which bridges the gap between clustering and representation learning.
XPrompt: Exploring the Extreme of Prompt Tuning
Prompt tuning learns soft prompts to condition frozen Pre-trained Language Models (PLMs) for performing downstream tasks in a parameter-efficient manner.
Generalized Intent Discovery: Learning from Open World Dialogue System
Traditional intent classification models are based on a pre-defined intent set and only recognize limited in-domain (IND) intent classes.
Structural Bias for Aspect Sentiment Triplet Extraction
Thus, a natural question arises: Is structural bias still a necessity in the context of PLMs?
Unified Knowledge Prompt Pre-training for Customer Service Dialogues
In this paper, we propose a novel unified knowledge prompt pre-training framework, UFA (\textbf{U}nified Model \textbf{F}or \textbf{A}ll Tasks), for customer service dialogues.
CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations
However, most current models use Chinese characters as inputs and are not able to encode semantic information contained in Chinese words.
MiniDisc: Minimal Distillation Schedule for Language Model Compression
In particular, motivated by the finding that the performance of the student is positively correlated to the scale-performance tradeoff of the teacher assistant, MiniDisc is designed with a $\lambda$-tradeoff to measure the optimality of the teacher assistant without trial distillation to the student.
Making Pretrained Language Models Good Long-tailed Learners
Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge.
GNN-encoder: Learning a Dual-encoder Architecture via Graph Neural Networks for Dense Passage Retrieval
To enhance the performance of dense retrieval models without loss of efficiency, we propose a GNN-encoder model in which query (passage) information is fused into passage (query) representations via graph neural networks that are constructed by queries and their top retrieved passages.
Deep Partial Multiplex Network Embedding
Network embedding is an effective technique to learn the low-dimensional representations of nodes in networks.
VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction
With the booming of pre-trained transformers, representation-based models based on Siamese transformer encoders have become mainstream techniques for efficient text matching.
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification.
Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval
Recently, the retrieval models based on dense representations have been gradually applied in the first stage of the document retrieval tasks, showing better performance than traditional sparse vector space models.
ASAP: A Chinese Review Dataset Towards Aspect Category Sentiment Analysis and Rating Prediction
Aspect category sentiment analysis (ACSA) and review rating prediction (RP) are two essential tasks to detect the fine-to-coarse sentiment polarities.
Query-aware Tip Generation for Vertical Search
Existing work on tip generation does not take query into consideration, which limits the impact of tips in search scenarios.
Earlier Attention? Aspect-Aware LSTM for Aspect-Based Sentiment Analysis
This paper proposes a novel variant of LSTM, termed as aspect-aware LSTM (AA-LSTM), which incorporates aspect information into LSTM cells in the context modeling stage before the attention mechanism.
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
Alibaba's Neural Machine Translation Systems for WMT18
We participated in 5 translation directions including English ↔ Russian, English ↔ Turkish in both directions and English → Chinese.
A Multi-task Learning Approach for Improving Product Title Compression with User Search Log Data
It is a challenging and practical research problem to obtain effective compression of lengthy product titles for E-commerce.
