Search Results for author:
Found 59 papers, 33 papers with code
Evaluation of Coreference Resolution Systems Under Adversarial Attacks
A substantial overlap of coreferent mentions in the CoNLL dataset magnifies the recent progress on coreference resolution.
TUDA-Reproducibility @ ReproGen: Replicability of Human Evaluation of Text-to-Text and Concept-to-Text Generation
This paper describes our contribution to the Shared Task ReproGen by Belz et al. (2021), which investigates the reproducibility of human evaluations in the context of Natural Language Generation.
End-to-end style-conditioned poetry generation: What does it take to learn from examples alone?
In this work, we design an end-to-end model for poetry generation based on conditioned recurrent neural network (RNN) language models whose goal is to learn stylistic features (poem length, sentiment, alliteration, and rhyming) from examples alone.
TUDa at WMT21: Sentence-Level Direct Assessment with Adapters
This paper presents our submissions to the WMT2021 Shared Task on Quality Estimation, Task 1 Sentence-Level Direct Assessment.
Reproducibility Issues for BERT-based Evaluation Metrics
We find that reproduction of claims and results often fails because of (i) heavy undocumented preprocessing involved in the metrics, (ii) missing code and (iii) reporting weaker results for the baseline metrics.
Towards Explainable Evaluation Metrics for Natural Language Generation
We also provide a synthesizing overview over recent approaches for explainable machine translation metrics and discuss how they relate to those goals and properties.
Detecting Stance in Scientific Papers: Did we get more Negative Recently?
In this paper, we classify scientific articles in the domain of natural language processing (NLP) and machine learning (ML) into whether (i) they extend the current state-of-the-art by introduction of novel techniques which beat existing models or whether (ii) they mainly criticize the existing state-of-the-art, i. e., that it is deficient with respect to some property (e. g., wrong evaluation, wrong datasets, misleading task specification).
USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation
In particular, we use an unsupervised evaluation metric to mine pseudo-parallel data, which we use to remap deficient underlying vector spaces (in an iterative manner) and to induce an unsupervised MT system, which then provides pseudo-references as an additional component in the metric.
Constrained Density Matching and Modeling for Cross-lingual Alignment of Contextualized Representations
In this work, we attribute the data hungriness of previous alignment techniques to two limitations: (i) the inability to sufficiently leverage data and (ii) these techniques are not trained properly.
DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence
We find that (i) the majority of BERT-based metrics correlate much worse with human rated coherence than early discourse metrics, invented a decade ago; (ii) the recent state-of-the-art BARTScore is weak when operated at system level -- which is particularly problematic as systems are typically compared in this manner.
Better than Average: Paired Evaluation of NLP Systems
Evaluation in NLP is usually done by comparing the scores of competing systems independently averaged over a common set of test instances.
The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results
In this paper, we introduce the Eval4NLP-2021shared task on explainable quality estimation.
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors
Evaluation metrics are a key ingredient for progress of text generation systems.
Constrained Density Matching and Modeling for Effective Contextualized Alignment
In this work, we analyze the limitations according to which previous alignments become very resource-intensive, \emph{viz.,} (i) the inability to sufficiently leverage data and (ii) that alignments are not trained properly.
Diachronic Analysis of German Parliamentary Proceedings: Ideological Shifts through the Lens of Political Biases
We analyze bias in historical corpora as encoded in diachronic distributional semantic models by focusing on two specific forms of bias, namely a political (i. e., anti-communism) and racist (i. e., antisemitism) one.
Changes in European Solidarity Before and During COVID-19: Evidence from a Large Crowd- and Expert-Annotated Twitter Dataset
We use these annotations to train a BERT model with multiple data augmentation strategies.
Graph Routing between Capsules
Therefore, in this paper, we introduce a new capsule network with graph routing to learn both relationships, where capsules in each layer are treated as the nodes of a graph.
BERT-Defense: A Probabilistic Model Based on BERT to Combat Cognitively Inspired Orthographic Adversarial Attacks
Adversarial attacks expose important blind spots of deep learning systems.
CMCE at SemEval-2020 Task 1: Clustering on Manifolds of Contextualized Embeddings to Detect Historical Meaning Shifts
Remarkably, with only 10 dimensional MBERT embeddings (reduced from the original size of 768), our submitted model performs best on subtask 1 for English and ranks third in subtask 2 for English.
From Hero to Z\'eroe: A Benchmark of Low-Level Adversarial Attacks
Adversarial attacks are label-preserving modifications to inputs of machine learning classifiers designed to fool machines but not humans.
Probing Multilingual BERT for Genetic and Typological Signals
We probe the layers in multilingual BERT (mBERT) for phylogenetic and geographic language signals across 100 languages and compute language distances based on the mBERT representations.
Vec2Sent: Probing Sentence Embeddings with Natural Language Generation
We introspect black-box sentence embeddings by conditionally generating from them with the objective to retrieve the underlying discrete sentence.
From Hero to Zéroe: A Benchmark of Low-Level Adversarial Attacks
Adversarial attacks are label-preserving modifications to inputs of machine learning classifiers designed to fool machines but not humans.
Inducing Language-Agnostic Multilingual Representations
Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world.
How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation
We then probe embeddings in a multilingual setup with design choices that lie in a 'stable region', as we identify for English, and find that results on English do not transfer to other languages.
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
Compared to the state-of-the-art unsupervised evaluation metrics, SUPERT correlates better with human ratings by 18-39%.
On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation
We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER.
PO-EMO: Conceptualization, Annotation, and Modeling of Aesthetic Emotions in German and English Poetry
Thus, we conceptualize a set of aesthetic emotions that are predictive of aesthetic appreciation in the reader, and allow the annotation of multiple labels per line to capture mixed emotions within their context.
Semantic Change and Emerging Tropes In a Large Corpus of New High German Poetry
Due to its semantic succinctness and novelty of expression, poetry is a great test bed for semantic change analysis.
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
A robust evaluation metric has a profound impact on the development of text generation systems.
Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications
Obstacles hindering the development of capsule networks for challenging NLP applications include poor scalability to large output spaces and less reliable routing processes.
Ranked #1 on
Text Classification
on RCV1
(P@1 metric)
Pitfalls in the Evaluation of Sentence Embeddings
Our motivation is to challenge the current evaluation of sentence embeddings and to provide an easy-to-access reference for future research.
Does My Rebuttal Matter? Insights from a Major NLP Conference
We then focus on the role of the rebuttal phase, and propose a novel task to predict after-rebuttal (i. e., final) scores from initial reviews and author responses.
Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems
Visual modifications to text are often used to obfuscate offensive comments in social media (e. g., "! d10t") or as a writing style ("1337" in "leet speak"), among other scenarios.
Predicting Research Trends From Arxiv
By extrapolation, we predict that these topics will remain lead problems/approaches in their fields in the short- and mid-term.
Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks
Activation functions play a crucial role in neural networks because they are the nonlinearities which have been attributed to the success story of deep learning.
PD3: Better Low-Resource Cross-Lingual Transfer By Combining Direct Transfer and Annotation Projection
We consider unsupervised cross-lingual transfer on two tasks, viz., sentence-level argumentation mining and standard POS tagging.
One Size Fits All? A simple LSTM for non-literal token and construction-level classification
In this paper, we tackle four different tasks of non-literal language classification: token and construction level metaphor detection, classification of idiomatic use of infinitive-verb compounds, and classification of non-literal particle verbs.
Killing Four Birds with Two Stones: Multi-Task Learning for Non-Literal Language Detection
In this paper we investigate multi-task learning for related non-literal language phenomena.
Cross-lingual Argumentation Mining: Machine Translation (and a bit of Projection) is All You Need!
Argumentation mining (AM) requires the identification of complex discourse structures and has lately been applied with success monolingually.
ArgumenText: Searching for Arguments in Heterogeneous Sources
Argument mining is a core technology for enabling argument search in large corpora.
Multi-Task Learning for Argumentation Mining in Low-Resource Settings
We investigate whether and where multi-task learning (MTL) can improve performance on NLP problems related to argumentation mining (AM), in particular argument component identification.
Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
Here, we generalize the concept of average word embeddings to power mean word embeddings.
What is the Essence of a Claim? Cross-Domain Claim Identification
Argument mining has become a popular research area in NLP.
Neural End-to-End Learning for Computational Argumentation Mining
Contrary to models that operate on the argument component level, we find that framing AM as dependency parsing leads to subpar performance results.
On the Linearity of Semantic Change: Investigating Meaning Variation via Dynamic Graph Models
We consider two graph models of semantic change.
EELECTION at SemEval-2017 Task 10: Ensemble of nEural Learners for kEyphrase ClassificaTION
From these approaches, we created an ensemble of differently hyper-parameterized systems, achieving a micro-F1-score of 0. 63 on the test data.
Still not there? Comparing Traditional Sequence-to-Sequence Models to Encoder-Decoder Neural Networks on Monotone String Translation Tasks
We analyze the performance of encoder-decoder neural models and compare them with well-known established methods.
Language classification from bilingual word embedding graphs
We study the role of the second language in bilingual word embeddings in monolingual semantic evaluation tasks.
Lemmatization and Morphological Tagging in German and Latin: A Comparison and a Survey of the State-of-the-art
This paper relates to the challenge of morphological tagging and lemmatization in morphologically rich languages by example of German and Latin.
Complex Decomposition of the Negative Distance kernel
Our evaluation shows that the power kernel produces F-scores that are comparable to the reference kernels, but is -- except for the linear kernel -- faster to compute.
On the Number of Many-to-Many Alignments of Multiple Sequences
We provide a new asymptotic formula for the case $S=\{(s_1,\ldots, s_N) \:|\: 1\le s_i\le 2\}$.
