Spider4SPARQL: A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems

1 code implementation • 28 Sep 2023 • Catherine Kosten, Philippe Cudré-Mauroux, Kurt Stockinger

With the recent spike in the number and availability of Large Language Models (LLMs), it has become increasingly important to provide large and realistic benchmarks for evaluating Knowledge Graph Question Answering (KGQA) systems.

Graph Question Answering Knowledge Graphs +3

INODE: Building an End-to-End Data Exploration System in Practice [Extended Vision]

no code implementations • 9 Apr 2021 • Sihem Amer-Yahia, Georgia Koutrika, Frederic Bastian, Theofilos Belmpas, Martin Braschler, Ursin Brunner, Diego Calvanese, Maximilian Fabricius, Orest Gkini, Catherine Kosten, Davide Lanti, Antonis Litke, Hendrik Lücke-Tieke, Francesco Alessandro Massucci, Tarcisio Mendes de Farias, Alessandro Mosca, Francesco Multari, Nikolaos Papadakis, Dimitris Papadopoulos, Yogendra Patil, Aurélien Personnaz, Guillem Rull, Ana Sima, Ellery Smith, Dimitrios Skoutas, Srividya Subramanian, Guohui Xiao, Kurt Stockinger

We demonstrate that our system is uniquely accessible to a wide range of users from larger scientific communities to the public.

Management

ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems

no code implementations • 7 Jun 2023 • Yi Zhang, Jan Deriu, George Katsogiannis-Meimarakis, Catherine Kosten, Georgia Koutrika, Kurt Stockinger

Thus, the challenge is many-fold: creating NL-to-SQL systems for highly complex domains with a small amount of hand-made training data augmented with synthetic data.

Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries

no code implementations • 13 Feb 2024 • Jonathan Fürst, Catherine Kosten, Farhard Nooralahzadeh, Yi Zhang, Kurt Stockinger

All of our data is based on real user questions that were asked live to the system.

Language Modelling Text-To-SQL