no code implementations • COLING (ArgMining) 2020 • Prakash Poudyal, Jaromir Savelka, Aagje Ieven, Marie Francine Moens, Teresa Goncalves, Paulo Quaresma
The corpus is annotated in terms of three types of clauses useful in argument mining: premise, conclusion, and non-argument parts of the text.
no code implementations • 14 Apr 2024 • Arav Agarwal, Karthik Mittal, Aidan Doyle, Pragnya Sridhar, Zipiao Wan, Jacob Arthur Doughty, Jaromir Savelka, Majd Sakr
We conduct a preliminary study of the effect of GPT's temperature parameter on the diversity of GPT4-generated questions.
no code implementations • 5 Dec 2023 • Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, Christopher Bogart, Eric Keylor, Can Kultur, Jaromir Savelka, Majd Sakr
While there is a growing body of research in computing education on utilizing large language models (LLMs) in generation and engagement with coding exercises, the use of LLMs for generating programming MCQs has not been extensively explored.
1 code implementation • 1 Nov 2023 • Samyar Janatian, Hannes Westermann, Jinzhe Tan, Jaromir Savelka, Karim Benyekhlef
We use LLMs to create pathways from legislation, according to the JusticeBot methodology for legal decision support systems, evaluate the pathways and compare them to manually created pathways.
no code implementations • 31 Oct 2023 • Jaromir Savelka, Paul Denny, Mark Liffiton, Brad Sheese
This study evaluates the performance of the GPT-3. 5 and GPT-4 models for classifying help requests from students in an introductory programming class.
no code implementations • 28 Oct 2023 • Jakub Drápal, Hannes Westermann, Jaromir Savelka
We propose a novel framework facilitating effective collaboration of a legal expert with a large language model (LLM) for generating initial codes (phase 2 of thematic analysis), searching for themes (phase 3), and classifying the data in terms of the themes (to kick-start phase 4).
no code implementations • 1 Oct 2023 • James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, Stephen MacNeil, Andrew Peterson, Raymond Pettit, Brent N. Reeves, Jaromir Savelka
Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards LLMs and their use in computing education contexts.
no code implementations • 27 Jul 2023 • Hannes Westermann, Jaromir Savelka, Karim Benyekhlef
In this article, we introduce LLMediator, an experimental platform designed to enhance online dispute resolution (ODR) by utilizing capabilities of state-of-the-art large language models (LLMs) such as GPT-4.
no code implementations • 30 Jun 2023 • Pragnya Sridhar, Aidan Doyle, Arav Agarwal, Christopher Bogart, Jaromir Savelka, Majd Sakr
We evaluated 127 LOs that were automatically generated based on a carefully crafted prompt (detailed guidelines on high-quality LOs authoring) submitted to GPT-4 for conceptual modules and projects of an AI Practitioner course.
no code implementations • 24 Jun 2023 • Jaromir Savelka, Kevin D. Ashley, Morgan A Gray, Hannes Westermann, Huihui Xu
We observed that, with a relatively minor decrease in performance, GPT-4 can perform batch predictions leading to significant cost reductions.
no code implementations • 15 Jun 2023 • Jaromir Savelka, Kevin D. Ashley, Morgan A. Gray, Hannes Westermann, Huihui Xu
We compare the performance of a baseline setup, where GPT-4 is directly asked to explain a legal term, to an augmented approach, where a legal information retrieval module is used to provide relevant context to the model, in the form of sentences from case law.
no code implementations • 15 Jun 2023 • Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, Majd Sakr
Additionally, we analyze the assessments that were not handled well by GPT-4 to understand the current limitations of the model, as well as its capabilities to leverage feedback provided by an auto-grader.
no code implementations • 8 May 2023 • Jaromir Savelka
We evaluated the capability of a state-of-the-art generative pre-trained transformer (GPT) model to perform semantic annotation of short text snippets (one to few sentences) coming from legal documents of various types.
no code implementations • 16 Mar 2023 • Jaromir Savelka, Arav Agarwal, Christopher Bogart, YiFan Song, Majd Sakr
We evaluated the capability of generative pre-trained transformers (GPT), to pass assessments in introductory and intermediate Python programming courses at the postsecondary level.
no code implementations • 9 Mar 2023 • Jaromir Savelka, Arav Agarwal, Christopher Bogart, Majd Sakr
While questions requiring to fill-in a blank in the code or completing a natural language statement about the snippet are handled rather successfully, MCQs that require analysis and/or reasoning about the code (e. g., what is true/false about the snippet, or what is its output) appear to be the most challenging.
no code implementations • 24 Oct 2022 • Hannes Westermann, Jaromir Savelka, Vern R. Walker, Kevin D. Ashley, Karim Benyekhlef
We propose an adaptive environment (CABINET) to support caselaw analysis (identifying key argument elements) based on a novel cognitive computing framework that carefully matches various machine learning (ML) capabilities to the proficiency of a user.
no code implementations • 17 Jan 2022 • Hannes Westermann, Jaromir Savelka, Vern R. Walker, Kevin D. Ashley, Karim Benyekhlef
The results also indicate that enhancements to a data set could be considered, alongside the advancement of the ML models, as an additional path for increasing classification performance on various tasks in AI & Law.
no code implementations • 21 Dec 2021 • Hannes Westermann, Jaromir Savelka, Vern R. Walker, Kevin D. Ashley, Karim Benyekhlef
We use this observation in allowing annotators to quickly view and annotate sentences that are semantically similar to a given sentence, across an entire corpus of documents.
no code implementations • 15 Dec 2021 • Jaromir Savelka, Hannes Westermann, Karim Benyekhlef
We analyze the ability of pre-trained language models to transfer knowledge among datasets annotated with different type systems and to generalize beyond the domain and dataset they were trained on.
1 code implementation • 15 Dec 2021 • Jaromir Savelka, Hannes Westermann, Karim Benyekhlef, Charlotte S. Alexander, Jayla C. Grant, David Restrepo Amariles, Rajaa El Hamdani, Sébastien Meeùs, Michał Araszkiewicz, Kevin D. Ashley, Alexandra Ashley, Karl Branting, Mattia Falduti, Matthias Grabmair, Jakub Harašta, Tereza Novotná, Elizabeth Tippett, Shiwanni Johnson
In this paper, we examine the use of multi-lingual sentence embeddings to transfer predictive models for functional segmentation of adjudicatory decisions across jurisdictions, legal systems (common and civil law), languages, and domains (i. e. contexts).
1 code implementation • Findings (EMNLP) 2021 • Jaromir Savelka, Kevin D. Ashley
Legal texts routinely use concepts that are difficult to understand.
no code implementations • 10 Dec 2021 • Hannes Westermann, Jaromir Savelka, Vern R. Walker, Kevin D. Ashley, Karim Benyekhlef
In this paper, we present a method of building strong, explainable classifiers in the form of Boolean search rules.