Search Results for author:
Found 35 papers, 26 papers with code
Few-shot Relation Extraction via Bayesian Meta-learning on Task Graphs
This paper studies few-shot relation extraction, which aims at predicting the relation for a pair of entities in a sentence by training with a few labeled examples in each relation.
Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients
In this work, we propose a novel framework, VLoRP, that extends low-rank gradient projection by introducing an additional degree of freedom for controlling the trade-off between memory efficiency and performance, beyond the rank hyper-parameter.
Metadata Conditioning Accelerates Language Model Pre-training
The vast diversity of styles, domains, and quality levels present in language model pre-training corpora is essential in developing general model capabilities, but efficiently learning and deploying the correct behaviors exemplified in each of these heterogeneous data sources is challenging.
How to Train Long-Context Language Models (Effectively)
We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information.
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly
There have been many benchmarks for evaluating long-context language models (LCLMs), but developers often rely on synthetic tasks like needle-in-a-haystack (NIAH) or arbitrary subsets of tasks.
LitSearch: A Retrieval Benchmark for Scientific Literature Search
LitSearch is constructed using a combination of (1) questions generated by GPT-4 based on paragraphs containing inline citations from research papers and (2) questions manually written by authors about their recently published papers.
Long-Context Language Modeling with Parallel Context Encoding
However, the substantial computational cost of transformers and limited generalization of positional encoding restrict the size of their context window.
Improving Language Understanding from Screenshots
An emerging family of language models (LMs), capable of processing both text and images within a single visual view, has the promise to unlock complex tasks such as chart understanding and UI navigation.
Harmonizing SO(3)-Equivariance with Neural Expressiveness: a Hybrid Deep Learning Framework Oriented to the Prediction of Electronic Structure Hamiltonian
Deep learning for predicting the electronic structure Hamiltonian of quantum systems necessitates satisfying the covariance laws, among which achieving SO(3)-equivariance without sacrificing the non-linear expressive capability of networks remains unsolved.
Evaluating Large Language Models at Evaluating Instruction Following
As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models.
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
In this work, we study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models.
Ranked #45 on
Question Answering
on PIQA
Fine-Tuning Language Models with Just Forward Passes
Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory.
Enabling Large Language Models to Generate Text with Citations
We propose ALCE, the first benchmark for Automatic LLMs' Citation Evaluation.
What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning
Large language models (LLMs) exploit in-context learning (ICL) to solve tasks with only a few demonstrations, but its mechanisms are not yet well-understood.
STM-UNet: An Efficient U-shaped Architecture Based on Swin Transformer and Multi-scale MLP for Medical Image Segmentation
Deep learning based models for medical image segmentation have made great progress in recent years.
The CRINGE Loss: Learning what language not to model
Standard language model training employs gold human documents or human-human interaction data, and treats all training data as positive examples.
Transformer-based dimensionality reduction
Recently, Transformer is much popular and plays an important role in the fields of Machine Learning (ML), Natural Language Processing (NLP), and Computer Vision (CV), etc.
Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models
Prompting, which casts downstream applications as language modeling tasks, has shown to be sample efficient compared to standard fine-tuning with pre-trained models.
Recovering Private Text in Federated Learning of Language Models
For the first time, we show the feasibility of recovering text from large batch sizes of up to 128 sentences.
Should You Mask 15% in Masked Language Modeling?
In this work, we revisit this important choice of MLM pre-training.
Ditch the Gold Standard: Re-evaluating Conversational Question Answering
In this work, we conduct the first large-scale human evaluation of state-of-the-art conversational QA systems, where human evaluators converse with models and judge the correctness of their answers.
Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction
Distantly supervised (DS) relation extraction (RE) has attracted much attention in the past few years as it can utilize large-scale auto-labeled data.
SimCSE: Simple Contrastive Learning of Sentence Embeddings
This paper presents SimCSE, a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings.
Ranked #3 on
Linear-Probe Classification
on SentEval
Making Pre-trained Language Models Better Few-shot Learners
We present LM-BFF--better few-shot fine-tuning of language models--a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples.
Ranked #6 on
Zero-Shot Text Classification
on AG News
TADO: Time-varying Attention with Dual-Optimizer Model
As the first attempt in this field to address this problem, we propose a flexible dual-optimizer model to gain robustness from both regression loss and classification loss.
Meta-Information Guided Meta-Learning for Few-Shot Relation Classification
Few-shot classification requires classifiers to adapt to new classes with only a few training instances.
Learning from Context or Names? An Empirical Study on Neural Relation Extraction
We find that (i) while context is the main source to support the predictions, RE models also heavily rely on the information from entity mentions, most of which is type information, and (ii) existing datasets may leak shallow heuristics via entity mentions and thus contribute to the high performance on RE benchmarks.
Ranked #24 on
Relation Extraction
on TACRED
Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs
To more effectively generalize to new relations, in this paper we study the relationships between different relations and propose to leverage a global relation graph.
Continual Relation Learning via Episodic Memory Activation and Reconsolidation
Continual relation learning aims to continually train a model on new data to learn incessantly emerging novel relations while avoiding catastrophically forgetting old relations.
More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction
Relational facts are an important component of human knowledge, which are hidden in vast amounts of text.
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text.
FewRel 2.0: Towards More Challenging Few-Shot Relation Classification
We present FewRel 2. 0, a more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a handful of instances?
OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction
OpenNRE is an open-source and extensible toolkit that provides a unified framework to implement neural models for relation extraction (RE).
Neural Snowball for Few-Shot Relation Learning
To address new relations with few-shot instances, we propose a novel bootstrapping approach, Neural Snowball, to learn new relations by transferring semantic knowledge about existing relations.
