RailSem19: A Dataset for Semantic Rail Scene Understanding

Solving tasks for autonomous road vehicles using com-puter vision is a dynamic and active research field. How-ever, one aspect of autonomous transportation has receivedlittle contributions: the rail domain. In this paper, we intro-duce the first public dataset for semantic scene understand-ing for trains and trams: RailSem19. This dataset consistsof 8500 annotated short sequences from the ego-perspectiveof trains, including over 1000 examples with railway cross-ings and 1200 tram scenes. Since it is the first image datasettargeting the rail domain, a novel label policy has been de-signed from scratch. It focuses on rail-specific labels notcovered by any other datasets. In addition to manual an-notations in the form of geometric shapes, we also supplydense pixel-wise semantic labeling. The dense labeling isa semantic-aware combination of (a) the geometric shapesand (b) weakly supervised annotations generated by exist-ing semantic segmentation networks from the road domain.Finally, multiple experiments give a first impression on howthe new dataset can be used to improve semantic sceneunderstanding in the rail environment. We present proto-types for the image-based classification of trains, switches,switch states, platforms, buffer stops, rail traffic signs andrail traffic lights. Applying transfer learning, we presentan early prototype for pixel-wise semantic segmentation onrail scenes. The resulting predictions show that this newdata also significantly improves scene understanding in sit-uations where cars and trains interact