Search Results for author: Pedro Henrique Luz de Araujo

Found 11 papers, 6 papers with code

Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles

no code implementations7 Jan 2025 Yuxi Xia, Pedro Henrique Luz de Araujo, Klim Zaporojets, Benjamin Roth

Concretely, we build Calib-n, a novel framework that trains an auxiliary model for confidence estimation that aggregates responses from multiple LLMs to capture inter-model agreement.

Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior

1 code implementation2 Jul 2024 Pedro Henrique Luz de Araujo, Benjamin Roth

We also compare persona's generations to two baseline settings: a control persona setting with 30 paraphrases of "a helpful assistant" to control for models' prompt sensitivity, and an empty persona setting where no persona is assigned.

Language Modeling Language Modelling +1

Exploring prompts to elicit memorization in masked language model-based named entity recognition

no code implementations5 May 2024 Yuxi Xia, Anastasiia Sedova, Pedro Henrique Luz de Araujo, Vasiliki Kougia, Lisa Nußbaumer, Benjamin Roth

Finally, the prompt performance of detecting model memorization is quantified by the percentage of name pairs for which the model has higher confidence for the name from the training set.

Language Modeling Language Modelling +4

Specification Overfitting in Artificial Intelligence

no code implementations13 Mar 2024 Benjamin Roth, Pedro Henrique Luz de Araujo, Yuxi Xia, Saskia Kaltenbrunner, Christoph Korab

Machine learning (ML) and artificial intelligence (AI) approaches are often criticized for their inherent bias and for their lack of control, accountability, and transparency.

Fairness

Functionality learning through specification instructions

no code implementations14 Nov 2023 Pedro Henrique Luz de Araujo, Benjamin Roth

We combine the specification instructions to create specification-augmented prompts, which we feed to language models pre-trained on natural instruction data.

Fairness

Cross-functional Analysis of Generalisation in Behavioural Learning

1 code implementation22 May 2023 Pedro Henrique Luz de Araujo, Benjamin Roth

In behavioural testing, system functionalities underrepresented in the standard evaluation setting (with a held-out test set) are validated through controlled input-output pairs.

Paraphrase Identification Reading Comprehension +1

Checking HateCheck: a cross-functional analysis of behaviour-aware learning for hate speech detection

1 code implementation nlppower (ACL) 2022 Pedro Henrique Luz de Araujo, Benjamin Roth

Behavioural testing -- verifying system capabilities by validating human-designed input-output pairs -- is an alternative evaluation method of natural language processing systems proposed to address the shortcomings of the standard approach: computing metrics on held-out data.

Hate Speech Detection

Topic Modelling Brazilian Supreme Court Lawsuits

1 code implementation1 Dec 2020 Pedro Henrique Luz de Araujo, Teófilo Emidio de Campos

The data consist of a corpus of 45, 532 lawsuits manually annotated by the Court’s experts with theme labels, a multi-class and multi-label classification task.

Multi-Label Classification MUlTI-LABEL-ClASSIFICATION

VICTOR: a Dataset for Brazilian Legal Documents Classification

1 code implementation LREC 2020 Pedro Henrique Luz de Araujo, Te{\'o}filo Em{\'\i}dio de Campos, Fabricio Ataides Braz, Nilton Correia da Silva

This paper describes VICTOR, a novel dataset built from Brazil{'}s Supreme Court digitalized legal documents, composed of more than 45 thousand appeals, which includes roughly 692 thousand documents{---}about 4. 6 million pages.

Classification General Classification

Cannot find the paper you are looking for? You can Submit a new open access paper.