Search Results for author:
Found 35 papers, 17 papers with code
Question Generation using a Scratchpad Encoder
In this paper we introduce the Scratchpad Encoder, a novel addition to the sequence to sequence (seq2seq) framework and explore its effectiveness in generating natural language questions from a given logical form.
The Llama 3 Herd of Models
This paper presents a new set of foundation models, called Llama 3.
Ranked #3 on
Multi-task Language Understanding
on MMLU
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
We introduce Llama Guard, an LLM-based input-output safeguard model geared towards Human-AI conversation use cases.
RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training
Under a unified evaluation of fine-tuned LMs by incorporating four representative perspectives of model robustness, we demonstrate the effectiveness of RoAST compared to state-of-the-art fine-tuning methods on six different types of LMs, which indicates its usefulness in practice.
MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
Specifically, an adversarial LLM and a target LLM interplay with each other in an iterative manner, where the adversarial LLM aims to generate challenging prompts that elicit unsafe responses from the target LLM, while the target LLM is fine-tuned with safety aligned data on these adversarial prompts.
On the Equivalence of Graph Convolution and Mixup
We establish this equivalence mathematically by demonstrating that graph convolution networks (GCN) and simplified graph convolution (SGC) can be expressed as a form of Mixup.
Effective Long-Context Scaling of Foundation Models
We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
We use this dataset to evaluate the capabilities of multilingual masked language models (MLMs) and large language models (LLMs).
Llama 2: Open Foundation and Fine-Tuned Chat Models
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.
Ranked #2 on
Question Answering
on PubChemQA
MixPAVE: Mix-Prompt Tuning for Few-shot Product Attribute Value Extraction
In this work, we propose a novel prompt tuning approach with Mixed Prompts for few-shot Attribute Value Extraction, namely MixPAVE.
Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization
In this work, we introduce Residual Prompt Tuning - a simple and efficient method that significantly improves the performance and stability of prompt tuning.
MMViT: Multiscale Multiview Vision Transformers
Our model encodes different views of the input signal and builds several channel-resolution feature stages to process the multiple views of the input at different resolutions in parallel.
SVT: Supertoken Video Transformer for Efficient Video Understanding
Whether by processing videos with fixed resolution from start to end or incorporating pooling and down-scaling strategies, existing video transformers process the whole video content throughout the network without specially handling the large portions of redundant information.
Progressive Prompts: Continual Learning for Language Models
We introduce Progressive Prompts - a simple and efficient approach for continual learning in language models.
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
Large multilingual language models typically rely on a single vocabulary shared across 100+ languages.
Uniform Masking Prevails in Vision-Language Pretraining
Masked Language Modeling (MLM) has proven to be an essential component of Vision-Language (VL) pretraining.
Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI
Evaluating an explanation's faithfulness is desired for many reasons such as trust, interpretability and diagnosing the sources of model's errors.
A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision
In this paper, we focus on teasing out what parts of the language supervision are essential for training zero-shot image classification models.
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks
When a neural language model (LM) is adapted to perform a new task, what aspects of the task predict the eventual performance of the model?
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models
Distilling state-of-the-art transformer models into lightweight student models is an effective way to reduce computation cost at inference time.
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning
Recent parameter-efficient language model tuning (PELT) methods manage to match the performance of fine-tuning with much fewer trainable parameters and perform especially well when training data is limited.
Entailment as Few-Shot Learner
Large pre-trained language models (LMs) have demonstrated remarkable ability as few-shot learners.
Ranked #1 on
Topic Classification
on OS
On the Influence of Masking Policies in Intermediate Pre-training
Current NLP models are predominantly trained through a two-stage "pre-train then fine-tune" pipeline.
On Unifying Misinformation Detection
In this paper, we introduce UnifiedM2, a general-purpose misinformation model that jointly models multiple domains of misinformation with a single, unified setup.
Towards Few-Shot Fact-Checking via Perplexity
Through experiments, we empirically verify the plausibility of the rather surprising usage of the perplexity score in the context of fact-checking and highlight the strength of our few-shot methodology by comparing it to strong fine-tuning-based baseline models.
Studying Strategically: Learning to Mask for Closed-book QA
Thus, our policy packs task-relevant knowledge into the parameters of a language model.
CLEAR: Contrastive Learning for Sentence Representation
Pre-trained language models have proven their unique powers in capturing implicit language features.
Ranked #5 on
Question Answering
on Quora Question Pairs
To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks
Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks.
To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks
Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks.
Linformer: Self-Attention with Linear Complexity
Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications.
Language Models as Fact Checkers?
Recent work has suggested that language models (LMs) store both common-sense and factual knowledge learned from pre-training data.
Keeping Notes: Conditional Natural Language Generation with a Scratchpad Encoder
We introduce the Scratchpad Mechanism, a novel addition to the sequence-to-sequence (seq2seq) neural network architecture and demonstrate its effectiveness in improving the overall fluency of seq2seq models for natural language generation tasks.
Keeping Notes: Conditional Natural Language Generation with a Scratchpad Mechanism
We introduce the Scratchpad Mechanism, a novel addition to the sequence-to-sequence (seq2seq) neural network architecture and demonstrate its effectiveness in improving the overall fluency of seq2seq models for natural language generation tasks.
Adversarial Training for Community Question Answer Selection Based on Multi-scale Matching
We frame this task as a binary (relevant/irrelevant) classification problem, and present an adversarial training framework to alleviate label imbalance issue.
Actionable Email Intent Modeling with Reparametrized RNNs
Emails in the workplace are often intentional calls to action for its recipients.
