Dreem Open Datasets: Multi-Scored Sleep Datasets to compare Human and Automated sleep staging

31 Oct 2019  ·  Antoine Guillot, Fabien Sauvet, Emmanuel H. During, Valentin Thorey ·

Sleep stage classification constitutes an important element of sleep disorder diagnosis. It relies on the visual inspection of polysomnography records by trained sleep technologists. Automated approaches have been designed to alleviate this resource-intensive task. However, such approaches are usually compared to a single human scorer annotation despite an inter-rater agreement of about 85 % only. The present study introduces two publicly-available datasets, DOD-H including 25 healthy volunteers and DOD-O including 55 patients suffering from obstructive sleep apnea (OSA). Both datasets have been scored by 5 sleep technologists from different sleep centers. We developed a framework to compare automated approaches to a consensus of multiple human scorers. Using this framework, we benchmarked and compared the main literature approaches. We also developed and benchmarked a new deep learning method, SimpleSleepNet, inspired by current state-of-the-art. We demonstrated that many methods can reach human-level performance on both datasets. SimpleSleepNet achieved an F1 of 89.9 % vs 86.8 % on average for human scorers on DOD-H, and an F1 of 88.3 % vs 84.8 % on DOD-O. Our study highlights that using state-of-the-art automated sleep staging outperforms human scorers performance for healthy volunteers and patients suffering from OSA. Consideration could be made to use automated approaches in the clinical setting.

PDF Abstract

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Sleep Stage Detection DODH SimpleSleepNet Accuracy 89.9 # 1
Kappa 84.6 # 1
Sleep Stage Detection DODH DeepSleepNet Accuracy 89.6 # 2
Kappa 84.3 # 2
Sleep Stage Detection DODO SeqSleepNet Accuracy 85.5 # 3
Kappa 77.2 # 3
Sleep Stage Detection DODO DeepSleepNet Accuracy 87.5 # 2
Kappa 80.4 # 2
Sleep Stage Detection DODO SimpleSleepNet Accuracy 88.7 # 1
Kappa 82.3 # 1
Sleep Stage Detection MASS SS3 Deep Sleep Net Accuracy 89.1% # 1
Sleep Stage Detection MASS SS3 Simple Sleep Net Accuracy 88.8% # 2