We present a new simulated dataset for pedestrian action anticipation collected using the CARLA simulator. To generate this dataset, we place a camera sensor on the ego-vehicle in the Carla environment and set the parameters to those of the camera used to record the PIE dataset (i.e., 1920x1080, 110° FOV). Then, we compute bounding boxes for each pedestrian interacting with the ego vehicle as seen through the camera's field of view. We generated the data in two urban environments available in the CARLA simulator: Town02 and Town03.

The total number of simulated pedestrians is nearly 55k, equivalent to 14M bounding boxes samples. The critical point for each pedestrian is their first point of crossing the street (in case they will eventually cross) or the last bounding box coordinates of their path in the opposite case. The crossing behavior represents 25% of the total pedestrians. We balanced the training split of the dataset to obtain labeled sequences crossing/non-crossing in equal parts. We used sequence-flipping to augment the minority class (i.e., crossing behavior in our case) and then undersampled the rest of the dataset. The result is a total of nearly 50k pedestrian sequences.

Next, the pedestrian trajectory sequences were transformed into observation sequences of equal length (i.e., 0.5 seconds) with a 60% overlap for the training splits. The TTE length is between 30 and 60 frames. It resulted in a total of nearly 220k observation sequences.