BoxVIS: Video Instance Segmentation with Box Annotations

26 Mar 2023  ·  Minghan Li, Lei Zhang ·

It is expensive and labour-extensive to label the pixel-wise object masks in a video. As a result, the amount of pixel-wise annotations in existing video instance segmentation (VIS) datasets is small, limiting the generalization capability of trained VIS models. An alternative but much cheaper solution is to use bounding boxes to label instances in videos. Inspired by the recent success of box-supervised image instance segmentation, we adapt the state-of-the-art pixel-supervised VIS models to a box-supervised VIS (BoxVIS) baseline, and observe slight performance degradation. We consequently propose to improve the BoxVIS performance from two aspects. First, we propose a box-center guided spatial-temporal pairwise affinity (STPA) loss to predict instance masks for better spatial and temporal consistency. Second, we collect a larger scale box-annotated VIS dataset (BVISD) by consolidating the videos from current VIS benchmarks and converting images from the COCO dataset to short pseudo video clips. With the proposed BVISD and the STPA loss, our trained BoxVIS model achieves 43.2\% and 29.0\% mask AP on the YouTube-VIS 2021 and OVIS valid sets, respectively. It exhibits comparable instance mask prediction performance and better generalization ability than state-of-the-art pixel-supervised VIS models by using only 16\% of their annotation time and cost. Codes and data can be found at \url{}.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Instance Segmentation OVIS validation BoxVIS(Swin-L & Box-sup) mask AP 40.6 # 18
AP50 68.4 # 11
AP75 39.9 # 18
APso 59.4 # 4
APmo 45.8 # 5
APho 20.9 # 5
Video Instance Segmentation YouTube-VIS 2021 BoxVIS(Swin-L & Box-sup) mask AP 53.9 # 16
AP50 76.4 # 16
AP75 59.6 # 16
AR10 61.0 # 12
AR1 44.8 # 15


No methods listed for this paper. Add relevant methods here