Search Results for author:
Found 63 papers, 28 papers with code
damo_nlp at MEDIQA 2021: Knowledge-based Preprocessing and Coverage-oriented Reranking for Medical Question Summarization
Medical question summarization is an important but difficult task, where the input is often complex and erroneous while annotated data is expensive to acquire.
S^4-Tuning: A Simple Cross-lingual Sub-network Tuning Method
The emergence of multilingual pre-trained language models makes it possible to adapt to target languages with only few labeled examples. However, vanilla fine-tuning tends to achieve degenerated and unstable results, owing to the Language Interference among different languages, and Parameter Overload under the few-sample transfer learning scenarios. To address two problems elegantly, we propose S^4-Tuning, a Simple Cross-lingual Sub-network Tuning method.
PALM: Pre-training an Autoencoding\&Autoregressive Language Model for Context-conditioned Generation
An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.
Abstractive Text Summarization
Conversational Response Generation
+9
Rethinking Denoised Auto-Encoding in Language Pre-Training
Pre-trained self-supervised models such as BERT have achieved striking success in learning sequence representations, especially for natural language processing.
Unlocking the Potential of Model Merging for Low-Resource Languages
Adapting large language models (LLMs) to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT).
Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models
Experimental results demonstrate that our approach surpasses the performance of both the large and small language models individually, forming a complementary advantage.
Harder Tasks Need More Experts: Dynamic Routing in MoE Models
In this paper, we introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models, aiming to enhance computational efficiency and model performance by adjusting the number of activated experts based on input difficulty.
BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization
Specifically, We incorporate a Text-Semantics-Aware Patch Selector (TSPS) into the ViT backbone to perform a coarse-grained visual token extraction and then attach a flexible Transformer-based Patch Abstraction Decoder (PAD) upon the backbone for top-level visual abstraction.
DiffDTM: A conditional structure-free framework for bioactive molecules generation targeted for dual proteins
However, molecule generation targeted for dual protein targets still faces formidable challenges including protein 3D structure data requisition for model training, auto-regressive sampling, and model generalization for unseen targets.
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning
In addition, taking the gate as a probing, we validate the efficiency and effectiveness of the variable prefix.
VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning
Recent studies have demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages.
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models.
Revisiting Automatic Question Summarization Evaluation in the Biomedical Domain
To better understand whether commonly used evaluation metrics are capable of evaluating automatic summarization in the biomedical domain, we conduct human evaluations of summarization quality from four different aspects of a biomedical question summarization task.
How well do Large Language Models perform in Arithmetic tasks?
Large language models have emerged abilities including chain-of-thought to answer math word problems step by step.
RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training
We propose to retrieve similar image-text pairs based on ITC from pretraining datasets and introduce a novel retrieval-attention module to fuse the representation of the image and the question with the retrieved images and texts.
Molecular Geometry-aware Transformer for accurate 3D Atomic System modeling
To alleviate this limitation, we propose Moleformer, a novel Transformer architecture that takes nodes (atoms) and edges (bonds and nonbonding atom pairs) as inputs and models the interactions among them using rotational and translational invariant geometry-aware spatial encoding.
Graph Neural Network
Initial Structure to Relaxed Energy (IS2RE), Direct
mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video
In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal modules for modality collaboration and disentangling different modality modules to deal with modality entanglement.
Ranked #1 on
Video Captioning
on MSR-VTT
Learning Trajectory-Word Alignments for Video-Language Tasks
To amend this, we propose a novel TW-BERT to learn Trajectory-Word alignment by a newly designed trajectory-to-word (T2W) attention for solving video-language tasks.
BUS: Efficient and Effective Vision-Language Pre-Training with Bottom-Up Patch Summarization.
In this paper, we propose a Bottom-Up Patch Summarization approach named BUS which is inspired by the Document Summarization Task in NLP to learn a concise visual summary of lengthy visual token sequences, guided by textual semantics.
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation
Unlike previous works that only add noise to inputs or parameters, we argue that the hidden representations of Transformers layers convey more diverse and meaningful language information.
SpanProto: A Two-stage Span-based Prototypical Network for Few-shot Named Entity Recognition
Few-shot Named Entity Recognition (NER) aims to identify named entities with very little annotated data.
A Survey on Open Information Extraction from Rule-based Model to Large Language Model
Open Information Extraction (OpenIE) represents a crucial NLP task aimed at deriving structured information from unstructured text, unrestricted by relation type or domain.
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections
Large-scale pretrained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks.
Ranked #1 on
Image Captioning
on COCO Captions
Parameter-Efficient Sparsity for Large Language Models Fine-Tuning
With the dramatically increased number of parameters in language models, sparsity methods have received ever-increasing research focus to compress and accelerate the models.
Towards Unified Prompt Tuning for Few-shot Text Classification
Prompt-based fine-tuning has boosted the performance of Pre-trained Language Models (PLMs) on few-shot text classification by employing task-specific prompts.
Image Captioning In the Transformer Age
This drawback inspires the researchers to develop a homogeneous architecture that facilitates end-to-end training, for which Transformer is the perfect one that has proven its huge potential in both vision and language domains and thus can be used as the basic component of the visual encoder and language decoder in an IC pipeline.
Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency
Structured pruning has been extensively studied on monolingual pre-trained language models and is yet to be fully evaluated on their multilingual counterparts.
Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning
Pre-trained Language Models (PLMs) have achieved remarkable performance for various language understanding tasks in IR systems, which require the fine-tuning process based on labeled training data.
Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding
Automatic ICD coding is defined as assigning disease codes to electronic medical records (EMRs).
Ranked #6 on
Medical Code Prediction
on MIMIC-III
From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression
Unified in contrastive learning, CAP enables the pruned model to learn from the pre-trained model for task-agnostic knowledge, and fine-tuned model for task-specific knowledge.
LOGEN: Few-shot Logical Knowledge-Conditioned Text Generation with Self-training
Previous works leverage logical forms to facilitate logical knowledge-conditioned text generation.
Achieving Human Parity on Visual Question Answering
The Visual Question Answering (VQA) task utilizes both visual image and language analysis to answer a textual question with respect to an image.
Ranked #8 on
Visual Question Answering (VQA)
on VQA v2 test-dev
A Chinese Multi-type Complex Questions Answering Dataset over Wikidata
We finally analyze the performance of SOTA KBQA models on this dataset and identify the challenges facing Chinese KBQA.
Fusing Heterogeneous Factors with Triaffine Mechanism for Nested Named Entity Recognition
To fuse these heterogeneous factors, we propose a novel triaffine mechanism including triaffine attention and scoring.
Ranked #1 on
Nested Named Entity Recognition
on TAC-KBP 2017
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning
Recent pretrained language models extend from millions to billions of parameters.
Addressing Semantic Drift in Generative Question Answering with Auxiliary Extraction
This work focuses on generative QA which aims to generate an abstractive answer to a given question instead of extracting an answer span from a provided passage.
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Vision-language pre-training (VLP) on large-scale image-text pairs has achieved huge success for the cross-modal downstream tasks.
StructuralLM: Structural Pre-training for Form Understanding
Large pre-trained language models achieve state-of-the-art results when fine-tuned on downstream NLP tasks.
Improving Biomedical Pretrained Language Models with Knowledge
To this end, we propose KeBioLM, a biomedical pretrained language model that explicitly leverages knowledge from the UMLS knowledge bases.
Ranked #1 on
Named Entity Recognition (NER)
on JNLPBA
Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models
Further analysis shows that Lattice-BERT can harness the lattice structures, and the improvement comes from the exploration of redundant information and multi-granularity representations.
Noisy-Labeled NER with Confidence Estimation
This work studies NER under a noisy labeled setting with calibrated confidence estimation.
SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
Vision-language pre-training (VLP) on large-scale image-text pairs has recently witnessed rapid progress for learning cross-modal representations.
Biomedical Question Answering: A Survey of Approaches and Challenges
Automatic Question Answering (QA) has been successfully applied in various domains such as search engines and chatbots.
Contrastive Prototype Learning with Augmented Embeddings for Few-Shot Learning
First, data augmentations are introduced to both the support and query sets with each sample now being represented as an augmented embedding (AE) composed of concatenated embeddings of both the original and augmented versions.
MELR: Meta-Learning via Modeling Episode-Level Relationships for Few-Shot Learning
Most recent few-shot learning (FSL) approaches are based on episodic training whereby each episode samples few training instances (shots) per class to imitate the test condition.
Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw
With the constrained jigsaw puzzles, instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable but meanwhile still ensure that the learned representation is sensitive to spatiotemporal continuity at both the local and global levels.
IEPT: Instance-Level and Episode-Level Pretext Tasks for Few-Shot Learning
Importantly, at the episode-level, two SSL-FSL hybrid learning objectives are devised: (1) The consistency across the predictions of an FSL classifier from different extended episodes is maximized as an episode-level pretext task.
Nested Named Entity Recognition with Partially-Observed TreeCRFs
With the TreeCRF we achieve a uniform way to jointly model the observed and the latent nodes.
Ranked #12 on
Nested Named Entity Recognition
on ACE 2005
VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
Existing work in multilingual pretraining has demonstrated the potential of cross-lingual transferability by training a unified Transformer encoder for multiple languages.
Predicting Clinical Trial Results by Implicit Evidence Integration
In the CTRP framework, a model takes a PICO-formatted clinical trial proposal with its background as input and predicts the result, i. e. how the Intervention group compares with the Comparison group in terms of the measured Outcome in the studied Population.
VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation
Recent studies about learning multilingual representations have achieved significant performance gains across a wide range of downstream cross-lingual tasks.
PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.
Ranked #1 on
Text Generation
on CNN/Daily Mail
Abstractive Text Summarization
Conversational Response Generation
+9
Encoding Implicit Relation Requirements for Relation Extraction: A Joint Inference Approach
Relation extraction is the task of identifying predefined relationship between entities, and plays an essential role in information extraction, knowledge base construction, question answering and so on.
Marrying up Regular Expressions with Neural Networks: A Case Study for Spoken Language Understanding
The success of many natural language processing (NLP) tasks is bound by the number and quality of annotated data, but there is often a shortage of such training data.
Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrix
We show that the dynamic transition matrix can effectively characterize the noise in the training data built by distant supervision.
Hybrid Question Answering over Knowledge Base and Free Text
While these systems are able to provide more precise answers than information retrieval (IR) based QA systems, the natural incompleteness of KB inevitably limits the question scope that the system can answer.
Question Answering on Freebase via Relation Extraction and Textual Evidence
Existing knowledge-based question answering systems often rely on small annotated training data.
Semantic Relation Classification via Convolutional Neural Networks with Simple Negative Sampling
Syntactic features play an essential role in identifying relationship in a sentence.
Ranked #3 on
Relation Classification
on SemEval 2010 Task 8
