Formality is one of the most important dimensions of writing style variation.
In this study we conducted an inter-rater reliability experiment for assessing
sentence formality on a five-point Likert scale, and obtained good agreement
results as well as different rating distributions for different sentence
categories. We also performed a difficulty analysis to identify the bottlenecks
of our rating procedure. Our main objective is to design an automatic scoring
mechanism for sentence-level formality, and this study is important for that