Search Results for author:
Found 60 papers, 43 papers with code
OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation
Office automation significantly enhances human productivity by automatically finishing routine tasks in the workflow.
The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism
Current evaluations of large language models (LLMs) often overlook non-determinism, typically focusing on a single output per example.
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs
We introduce WildGuard -- an open, light-weight moderation tool for LLM safety that achieves three goals: (1) identifying malicious intent in user prompts, (2) detecting safety risks of model responses, and (3) determining model refusal rate.
ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates
We demonstrate that a malicious user can exploit the ChatBug vulnerability of eight state-of-the-art (SOTA) LLMs and effectively elicit unintended responses from these models.
WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences
Recent breakthroughs in vision-language models (VLMs) emphasize the necessity of benchmarking human preferences in real-world multimodal interactions.
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
High-quality instruction data is critical for aligning large language models (LLMs).
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models
To overcome these limitations, we introduce the BiGGen Bench, a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks.
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
For automated evaluation with WildBench, we have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs such as GPT-4-turbo.
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
Proprietary LMs such as GPT-4 are often employed to assess the quality of responses from various LMs.
CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting
As the utilization of large language models (LLMs) has proliferated world-wide, it is crucial for them to have adequate knowledge and fair representation for diverse global cultures.
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks.
RewardBench: Evaluating Reward Models for Language Modeling
To enhance scientific understanding of reward models, we present RewardBench, a benchmark dataset and code-base for evaluation.
Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
This iterative cycle of exploration and training fosters continued improvement in the agents.
Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
Selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain.
OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Interpreter.
SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding
Our results show that SafeDecoding significantly reduces the attack success rate and harmfulness of jailbreak attacks without compromising the helpfulness of responses to benign user queries.
NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation
We present NovaCOMET, an open commonsense knowledge model, that combines the best aspects of knowledge and general task models.
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
We analyze the effect of alignment tuning by examining the token distribution shift between base LLMs and their aligned counterpart.
Agent Lumos: Unified and Modular Training for Open-Source Language Agents
To foster generalizable agent learning, we collect large-scale, unified, and high-quality training annotations derived from diverse ground-truth reasoning rationales across various complex interactive tasks.
Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
In this work, we study Reinforcement Learning from Personalized Human Feedback (RLPHF) problem, wherein LLMs are aligned to multiple (sometimes conflicting) preferences by modeling alignment as a Multi-Objective Reinforcement Learning (MORL) problem.
TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks
To quantitatively assess our metric, we evaluate its correlation with human ratings on 5 held-in datasets, 2 held-out datasets and show that TIGERScore can achieve the open-source SoTA correlation with human ratings across these datasets and almost approaches GPT-4 evaluator.
Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4
Unlike perfect information games, where all elements are known to every player, imperfect information games emulate the real-world complexities of decision-making under uncertain or incomplete information.
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks.
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion
We present LLM-Blender, an ensembling framework designed to attain consistently superior performance by leveraging the diverse strengths of multiple open-source large language models (LLMs).
Faith and Fate: Limits of Transformers on Compositionality
We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures.
SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks
The Swift module is a small encoder-decoder LM fine-tuned on the oracle agent's action trajectories, while the Sage module employs LLMs such as GPT-4 for subgoal planning and grounding.
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning
While extreme-scale language models have demonstrated exceptional performance on a variety of language tasks, the degree of control over these language models through pure prompting can often be limited.
PairReranker: Pairwise Reranking for Natural Language Generation
Pre-trained language models have been successful in natural language generation (NLG) tasks.
Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality
Human communication relies on common ground (CG), the mutual knowledge and beliefs shared by participants, to produce coherent and interesting conversations.
On Grounded Planning for Embodied Tasks with Language Models
Language models (LMs) have demonstrated their capability in possessing commonsense knowledge of the physical world, a crucial aspect of performing tasks in everyday life.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
On Continual Model Refinement in Out-of-Distribution Data Streams
Real-world natural language processing (NLP) models need to be continually updated to fix the prediction errors in out-of-distribution (OOD) data streams while overcoming catastrophic forgetting.
Unsupervised Cross-Task Generalization via Retrieval Augmentation
Humans can perform unseen tasks by recalling relevant skills acquired previously and then generalizing them to the target tasks, even if there is no supervision at all.
On the Robustness of Reading Comprehension Models to Entity Renaming
We study the robustness of machine reading comprehension (MRC) models to entity renaming -- do models make more wrong predictions when the same questions are asked about an entity whose name has been changed?
RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models
To audit the robustness of named entity recognition (NER) models, we propose RockNER, a simple yet effective method to create natural adversarial examples.
AutoTriggER: Label-Efficient and Robust Named Entity Recognition with Auxiliary Trigger Extraction
Deep neural models for named entity recognition (NER) have shown impressive results in overcoming label scarcity and generalizing to unseen entities by leveraging distant supervision and auxiliary information such as explanations.
Low Resource Named Entity Recognition
named-entity-recognition
+2
Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning
In addition, we also create two new datasets, X-CSQA and X-CODAH, by translating their English versions to 15 other languages, so that we can evaluate popular ML-LMs for cross-lingual commonsense reasoning.
Probing Commonsense Explanation in Dialogue Response Generation
Humans use commonsense reasoning (CSR) implicitly to produce natural and coherent responses in conversations.
FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks
Increasing concerns and regulations about data privacy and sparsity necessitate the study of privacy-preserving, decentralized learning methods for natural language processing (NLP) tasks.
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks.
Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning
The ability to continuously expand knowledge over time and utilize it to rapidly generalize to new tasks is a key feature of human linguistic intelligence.
RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge
Question: I have five fingers but I am not alive.
Differentiable Open-Ended Commonsense Reasoning
As a step towards making commonsense reasoning research more realistic, we propose to study open-ended commonsense reasoning (OpenCSR) -- the task of answering a commonsense question without any pre-defined choices -- using as a resource only a corpus of commonsense facts written in natural language.
Pre-training Text-to-Text Transformers for Concept-centric Common Sense
Pre-trained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks.
FreeDOM: A Transferable Neural Architecture for Structured Information Extraction on Web Documents
By combining these stages, FreeDOM is able to generalize to unseen sites after training on a small number of seed sites from that vertical without requiring expensive hand-crafted features over visual renderings of the page.
RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms
Pre-trained language models (PTLMs) have achieved impressive performance on commonsense inference benchmarks, but their ability to employ commonsense to make robust inferences, which is crucial for effective communications with humans, is debated.
IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization
Fine-tuning pre-trained language models (PTLMs), such as BERT and its better variant RoBERTa, has been a common practice for advancing performance in natural language understanding (NLU) tasks.
Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models
Recent works show that pre-trained language models (PTLMs), such as BERT, possess certain commonsense and factual knowledge.
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering
Existing work on augmenting question answering (QA) models with external knowledge (e. g., knowledge graphs) either struggle to model multi-hop relations efficiently, or lack transparency into the model's prediction rationale.
LEAN-LIFE: A Label-Efficient Annotation Framework Towards Learning from Explanation
Successfully training a deep neural network demands a huge corpus of labeled data.
TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition
In this paper, we introduce "entity triggers," an effective proxy of human explanations for facilitating label-efficient learning of NER models.
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning
In this paper, we present a constrained text generation task, CommonGen associated with a benchmark dataset, to explicitly test machines for the ability of generative commonsense reasoning.
Ranked #1 on
Text Generation
on CommonGen
Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling
Its performance is largely influenced by the annotation quality and quantity in supervised learning scenarios, and obtaining ground truth labels is often costly.
NERO: A Neural Rule Grounding Framework for Label-Efficient Relation Extraction
The soft matching module learns to match rules with semantically similar sentences such that raw corpora can be automatically labeled and leveraged by the RE module (in a much better coverage) as augmented supervision, in addition to the exactly matched sentences.
KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning
Commonsense reasoning aims to empower machines with the human ability to make presumptions about ordinary situations in our daily life.
Ranked #30 on
Common Sense Reasoning
on CommonsenseQA
(using extra training data)
AlpacaTag: An Active Learning-based Crowd Annotation Framework for Sequence Tagging
We introduce an open-source web-based data annotation framework (AlpacaTag) for sequence tagging tasks such as named-entity recognition (NER).
Neural Adaptation Layers for Cross-domain Named Entity Recognition
Recent research efforts have shown that neural architectures can be effective in conventional information extraction tasks such as named entity recognition, yielding state-of-the-art results on standard newswire datasets.
ExtRA: Extracting Prominent Review Aspects from Customer Feedback
Many existing systems for analyzing and summarizing customer reviews about products or service are based on a number of prominent review aspects.
Mining Cross-Cultural Differences and Similarities in Social Media
Cross-cultural differences and similarities are common in cross-lingual natural language understanding, especially for research in social media.
Automatic Extraction of Commonsense LocatedNear Knowledge
LocatedNear relation is a kind of commonsense knowledge describing two physical objects that are typically found near each other in real life.
