Search Results for author:
Found 105 papers, 74 papers with code
Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach
In recent years, pre-trained language models (PLMs) have been shown to capture factual knowledge from massive texts, which encourages the proposal of PLM-based knowledge graph completion (KGC) models.
CodRED: A Cross-Document Relation Extraction Dataset for Acquiring Knowledge in the Wild
Existing relation extraction (RE) methods typically focus on extracting relational facts between entity pairs within single sentences or documents.
Exploring Backdoor Vulnerabilities of Chat Models
Chat models are extensively adopted across various real-world scenarios, thus the security of chat models deserves increasing attention.
Advancing LLM Reasoning Generalists with Preference Trees
We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning.
USimAgent: Large Language Models for Simulating Search Users
However, the potential of using LLMs in simulating search behaviors has not yet been fully explored.
Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
In practice, the multifaceted nature of human preferences inadvertently introduces what is known as the "alignment tax" -a compromise where enhancements in alignment within one objective (e. g., harmlessness) can diminish performance in others (e. g., helpfulness).
RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation
Generative models have demonstrated considerable potential in software engineering, particularly in tasks such as code generation and debugging.
Large Language Model-based Human-Agent Collaboration for Complex Task Solving
We construct a human-agent collaboration dataset to train this policy model in an offline reinforcement learning environment.
Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
We first formulate a general framework of agent backdoor attacks, then we present a thorough analysis on the different forms of agent backdoor attacks.
Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents
Current language model-driven agents often lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions.
InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory
To alleviate these issues, existing efforts employ sliding attention windows and discard distant tokens to achieve the processing of extremely long sequences.
ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs
To find the most efficient activation function for sparse computation, we propose a systematic framework to examine the sparsity of LLMs from three aspects: the trade-off between sparsity and performance, the predictivity of sparsity, and the hardware affinity.
UniMem: Towards a Unified View of Long-Context Large Language Models
Although there exist various methods devoted to enhancing the long-context processing ability of large language models (LLMs), they are developed in an isolated manner and lack systematic analysis and integration of their strengths, hindering further developments.
Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution
This paper introduces Investigate-Consolidate-Exploit (ICE), a novel strategy for enhancing the adaptability and flexibility of AI agents through inter-task self-evolution.
DebugBench: Evaluating Debugging Capability of Large Language Models
Previous evaluations of LLMs' debugging ability are significantly limited by the risk of data leakage, the scale of the dataset, and the variety of tested bugs.
GitAgent: Facilitating Autonomous Agent with GitHub by Tool Extension
As GitHub has hosted a multitude of repositories which can be seen as a good resource for tools, a promising solution is that LLM-based agents can autonomously integrate the repositories in GitHub according to the user queries to extend their tool set.
MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation
Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-event relationships.
Enabling Large Language Models to Learn from Rules
The current knowledge learning paradigm of LLMs is mainly based on learning from examples, in which LLMs learn the internal rule implicitly from a certain number of supervised examples.
Don't Make Your LLM an Evaluation Benchmark Cheater
Large language models~(LLMs) have greatly advanced the frontiers of artificial intelligence, attaining remarkable improvement in model capacity.
ProAgent: From Robotic Process Automation to Agentic Process Automation
Empirical experiments are conducted to detail its construction and execution procedure of workflow, showcasing the feasibility of APA, unveiling the possibility of a new paradigm of automation driven by agents.
Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
Pre-trained language models (PLMs) have achieved remarkable results on NLP tasks but at the expense of huge parameter sizes and the consequent computational costs.
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models
Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments, enabling substantial reductions in model storage and memory costs without significant performance compromise.
Predicting Emergent Abilities with Infinite Resolution Evaluation
With PassUntil, we conduct a quantitative investigation into the scaling law of task performance.
Rational Decision-Making Agent with Internalized Utility Judgment
Central to the development of rationality is the construction of an internalized utility judgment, capable of assigning numerical utilities to each decision.
Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Building a competitive counterpart in other languages is highly challenging due to the low-resource nature of non-English multimodal data (i. e., lack of large-scale, high-quality image-text data).
A Survey on Large Language Model based Autonomous Agents
In this paper, we present a comprehensive survey of these studies, delivering a systematic review of the field of LLM-based autonomous agents from a holistic perspective.
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction.
Ranked #3 on
Trajectory Planning
on ToolBench
Towards Codable Watermarking for Injecting Multi-bits Information to LLMs
As large language models (LLMs) generate texts with increasing fluency and realism, there is a growing need to identify the source of texts to prevent the abuse of LLMs.
User Behavior Simulation with Large Language Model based Agents
Simulating high quality user behavior data has always been a fundamental problem in human-centered applications, where the major difficulty originates from the intricate mechanism of human decision process.
Exploring the Impact of Model Scaling on Parameter-Efficient Tuning
Our investigations reveal that model scaling (1) mitigates the effects of the positions of tunable parameters on performance, and (2) enables tuning methods to achieve performance comparable to full-parameter fine-tuning by optimizing fewer tunable parameters.
Plug-and-Play Document Modules for Pre-trained Models
By inserting document plugins into the backbone PTM for downstream tasks, we can encode a document one time to handle multiple tasks, which is more efficient than conventional encoding-task coupling methods that simultaneously encode documents and input queries using task-specific encoders.
Plug-and-Play Knowledge Injection for Pre-trained Language Models
Experimental results on three knowledge-driven NLP tasks show that existing injection methods are not suitable for the new paradigm, while map-tuning effectively improves the performance of downstream models.
Stochastic Bridges as Effective Regularizers for Parameter-Efficient Tuning
Since it is non-trivial to directly model the intermediate states and design a running cost function, we propose to use latent stochastic bridges to regularize the intermediate states and use the regularization as the running cost of PETs.
Emergent Modularity in Pre-trained Transformers
In analogy to human brains, we consider two main characteristics of modularity: (1) functional specialization of neurons: we evaluate whether each neuron is mainly specialized in a certain function, and find that the answer is yes.
Decouple knowledge from parameters for plug-and-play language modeling
The key intuition is to decouple the knowledge storage from model parameters with an editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the DPM.
Recyclable Tuning for Continual Pre-training
In pilot studies, we find that after continual pre-training, the upgraded PLM remains compatible with the outdated adapted weights to some extent.
WebCPM: Interactive Web Search for Chinese Long-form Question Answering
We recruit annotators to search for relevant information using our interface and then answer questions.
UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models
In this paper, we propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge.
Tool Learning with Foundation Models
Considering the lack of a systematic tool learning evaluation in prior works, we experiment with 18 representative tools and show the potential of current foundation models in skillfully utilizing tools.
When to Trust Aggregated Gradients: Addressing Negative Client Sampling in Federated Learning
Federated Learning has become a widely-used framework which allows learning a global model on decentralized local datasets under the condition of protecting local data privacy.
Plug-and-Play Secondary Control for Safety of LTI Systems under Attacks
To further ensure the safety of the states of the system, we choose a subset of sensors that can be locally secured and made free of attacks.
MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction
It contains 103, 193 event coreference chains, 1, 216, 217 temporal relations, 57, 992 causal relations, and 15, 841 subevent relations, which is larger than existing datasets of all the ERE tasks by at least an order of magnitude.
Exploring Mode Connectivity for Pre-trained Language Models
(3) How does the PLM's task knowledge change along the path connecting two minima?
Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuning
To fathom the mystery, we hypothesize that the adaptations of different DETs could all be reparameterized as low-dimensional optimizations in a unified optimization subspace, which could be found by jointly decomposing independent solutions of different DETs.
ROSE: Robust Selective Fine-tuning for Pre-trained Language Models
Even though the large-scale language models have achieved excellent performances, they suffer from various adversarial attacks.
Step out of KG: Knowledge Graph Completion via Knowledgeable Retrieval and Reading Comprehension
To alleviate this problem, we propose a new model based on information retrieval and reading comprehension, namely IR4KGC.
From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models
We then design a Model Uncertainty--aware Knowledge Integration (MUKI) framework to recover the golden supervision for the student.
Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models
Prompting, which casts downstream applications as language modeling tasks, has shown to be sample efficient compared to standard fine-tuning with pre-trained models.
Manual-Guided Dialogue for Flexible Conversational Agents
Modeling these subtasks is consistent with the human agent's behavior patterns.
Towards a General Pre-training Framework for Adaptive Learning in MOOCs
Adaptive learning aims to stimulate and meet the needs of individual learners, which requires sophisticated system-level coordination of diverse tasks, including modeling learning resources, estimating student states, and making personalized recommendations.
CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation
Existing reference-free metrics have obvious limitations for evaluating controlled text generation models.
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
ELLE: Efficient Lifelong Pre-training for Emerging Data
We experiment ELLE with streaming data from 5 domains on BERT and GPT.
A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models
Pre-trained language models (PLMs) cannot well recall rich factual knowledge of entities exhibited in large-scale corpora, especially those rare entities.
Model Uncertainty-Aware Knowledge Amalgamation for Pre-Trained Language Models
As many fine-tuned pre-trained language models~(PLMs) with promising performance are generously released, investigating better ways to reuse these models is vital as it can greatly reduce the retraining computational cost and the potential environmental side-effects.
On Transferability of Prompt Tuning for Natural Language Processing
To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work.
RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models
Motivated by this observation, we construct a word-based robustness-aware perturbation to distinguish poisoned samples from clean samples to defend against the backdoor attacks on natural language processing (NLP) models.
Exploring Universal Intrinsic Task Subspace via Prompt Tuning
In the experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 250-dimensional subspace found with 100 tasks, by only tuning 250 free parameters, we can recover 97% and 83% of the full prompt tuning performance for 100 seen tasks (using different training data) and 20 unseen tasks, respectively, showing great generalization ability of the found intrinsic task subspace.
Topology-Imbalance Learning for Semi-Supervised Node Classification
The class imbalance problem, as an important issue in learning node representations, has drawn increasing attention from the community.
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs.
Dynamic Knowledge Distillation for Pre-trained Language Models
Knowledge distillation~(KD) has been proved effective for compressing large-scale pre-trained language models.
Packed Levitated Marker for Entity and Relation Extraction
In particular, we propose a neighborhood-oriented packing strategy, which considers the neighbor spans integrally to better model the entity boundary information.
Ranked #1 on
Named Entity Recognition (NER)
on Few-NERD (SUP)
On Length Divergence Bias in Textual Matching Models
To determine whether TM models have adopted such heuristic, we introduce an adversarial evaluation scheme which invalidates the heuristic.
Rethinking Stealthiness of Backdoor Attack against NLP Models
In this work, we point out a potential problem of current backdoor attacking research: its evaluation ignores the stealthiness of backdoor attacks, and most of existing backdoor attacking methods are not stealthy either to system deployers or to system users.
Fully Hyperbolic Neural Networks
Hyperbolic neural networks have shown great potential for modeling complex data.
CLEVE: Contrastive Pre-training for Event Extraction
Event extraction (EE) has considerably benefited from pre-trained language models (PLMs) by fine-tuning.
Knowledge Inheritance for Pre-trained Language Models
Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs.
TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
To address this issue, we propose a dynamic token reduction approach to accelerate PLMs' inference, named TR-BERT, which could flexibly adapt the layer number of each token in inference to avoid redundant calculation.
Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction
Distantly supervised (DS) relation extraction (RE) has attracted much attention in the past few years as it can utilize large-scale auto-labeled data.
Representation Learning for Natural Language Processing
This book aims to review and present the recent advances of distributed representation learning for NLP, including why representation learning can improve NLP, how representation learning takes part in various important topics of NLP, and what challenges are still not well addressed by distributed representation.
CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
We then perform contrastive semi-supervised learning on both the retrieved unlabeled and original labeled instances to help PLMs capture crucial task-related semantic features.
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks.
CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade
On the other hand, the exiting decisions made by internal classifiers are unreliable, leading to wrongly emitted early predictions.
Rethinking the Promotion Brought by Contrastive Learning to Semi-Supervised Node Classification
Graph Contrastive Learning (GCL) has proven highly effective in promoting the performance of Semi-Supervised Node Classification (SSNC).
DisenE: Disentangling Knowledge Graph Embeddings
Knowledge graph embedding (KGE), aiming to embed entities and relations into low-dimensional vectors, has attracted wide attention recently.
Disentangle-based Continual Graph Representation Learning
Graph embedding (GE) methods embed nodes (and/or edges) in graph into a low-dimensional semantic space, and have shown its effectiveness in modeling multi-relational data.
Learning from Context or Names? An Empirical Study on Neural Relation Extraction
We find that (i) while context is the main source to support the predictions, RE models also heavily rely on the information from entity mentions, most of which is type information, and (ii) existing datasets may leak shallow heuristics via entity mentions and thus contribute to the high performance on RE benchmarks.
Ranked #23 on
Relation Extraction
on TACRED
CokeBERT: Contextual Knowledge Selection and Embedding towards Enhanced Pre-Trained Language Models
In this paper, we propose a novel framework named Coke to dynamically select contextual knowledge and embed knowledge context according to textual context for PLMs, which can avoid the effect of redundant and ambiguous knowledge in KGs that cannot match the input text.
Continual Relation Learning via Episodic Memory Activation and Reconsolidation
Continual relation learning aims to continually train a model on new data to learn incessantly emerging novel relations while avoiding catastrophically forgetting old relations.
Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation
By dynamically determining segment length and deleting repetitive segments, RecoverSAT is capable of recovering from repetitive and missing token errors.
MAVEN: A Massive General Domain Event Detection Dataset
Most existing datasets exhibit the following issues that limit further development of ED: (1) Data scarcity.
Coreferential Reasoning Learning for Language Representation
Language representation models such as BERT could effectively capture contextual semantic information from plain text, and have been proved to achieve promising results in lots of downstream NLP tasks with appropriate fine-tuning.
Ranked #31 on
Relation Extraction
on DocRED
More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction
Relational facts are an important component of human knowledge, which are hidden in vast amounts of text.
HighwayGraph: Modelling Long-distance Node Relations for Improving General Graph Neural Network
To address this issue, we propose to model long-distance node relations by simply relying on shallow GNN architectures with two solutions: (1) Implicitly modelling by learning to predict node pair relations (2) Explicitly modelling by adding edges between nodes that potentially have the same label.
Multi-Paragraph Reasoning with Knowledge-enhanced Graph Neural Network
Multi-paragraph reasoning is indispensable for open-domain question answering (OpenQA), which receives less attention in the current OpenQA systems.
Ranked #58 on
Question Answering
on HotpotQA
Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
Non-autoregressive neural machine translation (NAT) generates each target word in parallel and has achieved promising inference acceleration.
NumNet: Machine Reading Comprehension with Numerical Reasoning
Numerical reasoning, such as addition, subtraction, sorting and counting is a critical skill in human's reading comprehension, which has not been well considered in existing machine reading comprehension (MRC) systems.
Ranked #10 on
Question Answering
on DROP Test
Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View
Graph Neural Networks (GNNs) have achieved promising performance on a wide range of graph-based tasks.
Ranked #52 on
Node Classification
on Cora
XQA: A Cross-lingual Open-domain Question Answering Dataset
Experimental results show that the multilingual BERT model achieves the best results in almost all target languages, while the performance of cross-lingual OpenQA is still much lower than that of English.
DocRED: A Large-Scale Document-Level Relation Extraction Dataset
Multiple entities in a document generally exhibit complex inter-sentence relations, and cannot be well handled by existing relation extraction (RE) methods that typically focus on extracting intra-sentence relations for single entity pairs.
Ranked #59 on
Relation Extraction
on DocRED
Fact Discovery from Knowledge Base via Facet Decomposition
We also propose a novel auto-encoder based facet component to estimate some facets of the fact.
Graph Neural Networks with Generated Parameters for Relation Extraction
Recently, progress has been made towards improving relational reasoning in machine learning field.
Knowledge Representation Learning: A Quantitative Review
Knowledge representation learning (KRL) aims to represent entities and relations in knowledge graph in low-dimensional semantic space, which have been widely used in massive knowledge-driven tasks.
DIAG-NRE: A Neural Pattern Diagnosis Framework for Distantly Supervised Neural Relation Extraction
To demonstrate the effectiveness of DIAG-NRE, we apply it to two real-world datasets and present both significant and interpretable improvements over state-of-the-art methods.
OpenKE: An Open Toolkit for Knowledge Embedding
We release an open toolkit for knowledge embedding (OpenKE), which provides a unified framework and various fundamental models to embed knowledge graphs into a continuous low-dimensional space.
Cross-lingual Lexical Sememe Prediction
We propose a novel framework to model correlations between sememes and multi-lingual words in low-dimensional semantic space for sememe prediction.
Denoise while Aggregating: Collaborative Learning in Open-Domain Question Answering
The open-domain question answering (OpenQA) task aims to extract answers that match specific questions from a distantly supervised corpus.
Adversarial Multi-lingual Neural Relation Extraction
To address these issues, we propose an adversarial multi-lingual neural relation extraction (AMNRE) model, which builds both consistent and individual representations for each sentence to consider the consistency and diversity among languages.
Denoising Distantly Supervised Open-Domain Question Answering
Distantly supervised open-domain question answering (DS-QA) aims to find answers in collections of unlabeled text.
Ranked #2 on
Open-Domain Question Answering
on Quasar
Neural Relation Extraction with Multi-lingual Attention
Relation extraction has been widely used for finding unknown relational facts from plain text.
Incorporating Relation Paths in Neural Relation Extraction
Distantly supervised relation extraction has been widely used to find novel relational facts from plain text.
Modeling Relation Paths for Representation Learning of Knowledge Bases
Representation learning of knowledge bases (KBs) aims to embed both entities and relations into a low-dimensional space.
Learning Entity and Relation Embeddings for Knowledge Graph Completion
Knowledge graph completion aims to perform link prediction between entities.
