BATMAN: Bilateral Attention Transformer in Motion-Appearance Neighboring Space for Video Object Segmentation

1 Aug 2022  ·  Ye Yu, Jialin Yuan, Gaurav Mittal, Li Fuxin, Mei Chen ·

Video Object Segmentation (VOS) is fundamental to video understanding. Transformer-based methods show significant performance improvement on semi-supervised VOS. However, existing work faces challenges segmenting visually similar objects in close proximity of each other. In this paper, we propose a novel Bilateral Attention Transformer in Motion-Appearance Neighboring space (BATMAN) for semi-supervised VOS. It captures object motion in the video via a novel optical flow calibration module that fuses the segmentation mask with optical flow estimation to improve within-object optical flow smoothness and reduce noise at object boundaries. This calibrated optical flow is then employed in our novel bilateral attention, which computes the correspondence between the query and reference frames in the neighboring bilateral space considering both motion and appearance. Extensive experiments validate the effectiveness of BATMAN architecture by outperforming all existing state-of-the-art on all four popular VOS benchmarks: Youtube-VOS 2019 (85.0%), Youtube-VOS 2018 (85.3%), DAVIS 2017Val/Testdev (86.2%/82.2%), and DAVIS 2016 (92.5%).

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Object Segmentation DAVIS 2016 CFBI (val) Jaccard (Mean) 88.3 # 14
Compare
F-Score 90.5 # 12
Compare
J&F 89.4 # 11
Compare
Video Object Segmentation DAVIS 2016 KMN (val) Jaccard (Mean) 89.5 # 8
Compare
F-Score 91.5 # 8
Compare
J&F 90.5 # 7
Compare
Video Object Segmentation DAVIS 2016 CFBI+ (val) Jaccard (Mean) 88.7 # 11
Compare
F-Score 91.1 # 11
Compare
J&F 89.9 # 9
Compare
Video Object Segmentation DAVIS 2016 RMN (val) Jaccard (Mean) 88.9 # 10
Compare
F-Score 88.7 # 15
Compare
J&F 88.8 # 12
Compare
Video Object Segmentation DAVIS 2016 LCM (val) Jaccard (Mean) 89.9 # 6
Compare
F-Score 91.4 # 9
Compare
Mean Jaccard & F-Measure 90.7 # 1
Compare
Video Object Segmentation DAVIS 2016 BATMAN (val) Jaccard (Mean) 90.7 # 3
Compare
F-Score 94.2 # 3
Compare
J&F 92.5 # 2
Compare
Video Object Segmentation DAVIS 2016 AOT (val) Jaccard (Mean) 90.1 # 5
Compare
F-Score 92.1 # 7
Compare
J&F 91.1 # 5
Compare
Video Object Segmentation DAVIS 2016 STCN (val) Jaccard (Mean) 90.8 # 2
Compare
F-Score 92.5 # 6
Compare
J&F 91.6 # 3
Compare
Video Object Segmentation DAVIS 2016 TransVOS (val) Jaccard (Mean) 89.8 # 7
Compare
F-Score 91.2 # 10
Compare
J&F 90.5 # 7
Compare
Video Object Segmentation DAVIS 2016 RPCMVOS (val) Jaccard (Mean) 87.1 # 15
Compare
F-Score 94 # 4
Compare
J&F 90.6 # 6
Compare
Video Object Segmentation DAVIS 2016 STM (val) Jaccard (Mean) 88.7 # 11
Compare
F-Score 89.9 # 14
Compare
Video Object Segmentation DAVIS 2017 (test-dev) RMN Jaccard 71.9 # 8
Compare
F-measure 78.1 # 9
Compare
Video Object Segmentation DAVIS 2017 (test-dev) LCM Jaccard 74.4 # 4
Compare
F-measure 81.8 # 4
Compare
Mean Jaccard & F-Measure 78.1 # 4
Compare
Video Object Segmentation DAVIS 2017 (test-dev) RPCMVOS Jaccard 75.8 # 3
Compare
F-measure 82.6 # 3
Compare
Mean Jaccard & F-Measure 79.2 # 3
Compare
Video Object Segmentation DAVIS 2017 (test-dev) TransVOS Jaccard 73 # 6
Compare
F-measure 80.9 # 5
Compare
Mean Jaccard & F-Measure 76.9 # 6
Compare
Video Object Segmentation DAVIS 2017 (test-dev) AOT Jaccard 75.9 # 2
Compare
F-measure 83.3 # 2
Compare
Mean Jaccard & F-Measure 79.6 # 2
Compare
Video Object Segmentation DAVIS 2017 (test-dev) BATMAN Jaccard 78.4 # 1
Compare
F-measure 86.1 # 1
Compare
Mean Jaccard & F-Measure 82.2 # 1
Compare
Video Object Segmentation DAVIS 2017 (test-dev) CFBI Jaccard 71.4 # 10
Compare
F-measure 78.7 # 8
Compare
Mean Jaccard & F-Measure 75 # 9
Compare
Video Object Segmentation DAVIS 2017 (test-dev) KMN Jaccard 74.1 # 5
Compare
F-measure 80.3 # 6
Compare
Mean Jaccard & F-Measure 77.2 # 5
Compare
Video Object Segmentation DAVIS 2017 (test-dev) STCN Jaccard 72.7 # 7
Compare
F-measure 79.6 # 7
Compare
Mean Jaccard & F-Measure 76.1 # 7
Compare
Video Object Segmentation DAVIS 2017 (test-dev) CFBI+ Jaccard 71.6 # 9
Compare
Mean Jaccard & F-Measure 75.6 # 8
Compare
Video Object Segmentation DAVIS 2017 (val) RPCMVOS Mean Jaccard & F-Measure 83.7 # 7
Compare
Jaccard 81.3 # 6
Compare
Video Object Segmentation DAVIS 2017 (val) AFB-URR Mean Jaccard & F-Measure 74.6 # 14
Compare
Jaccard 73 # 15
Compare
F-measure 76.1 # 15
Compare
Video Object Segmentation DAVIS 2017 (val) LWL Mean Jaccard & F-Measure 81.6 # 13
Compare
Jaccard 79.1 # 14
Compare
F-measure 84.1 # 14
Compare
Video Object Segmentation DAVIS 2017 (val) STM Jaccard 79.2 # 13
Compare
F-measure 84.3 # 13
Compare
Video Object Segmentation DAVIS 2017 (val) CFBI Mean Jaccard & F-Measure 81.9 # 12
Compare
Jaccard 79.3 # 12
Compare
F-measure 84.5 # 12
Compare
Video Object Segmentation DAVIS 2017 (val) RMN Mean Jaccard & F-Measure 83.5 # 8
Compare
Jaccard 81 # 7
Compare
F-measure 86 # 8
Compare
Video Object Segmentation DAVIS 2017 (val) SST Mean Jaccard & F-Measure 82.5 # 11
Compare
Jaccard 79.9 # 11
Compare
F-measure 85.1 # 11
Compare
Video Object Segmentation DAVIS 2017 (val) KMN Mean Jaccard & F-Measure 82.8 # 10
Compare
Jaccard 80 # 10
Compare
F-measure 85.6 # 10
Compare
Video Object Segmentation DAVIS 2017 (val) LCM Jaccard 80.5 # 8
Compare
F-measure 86.5 # 6
Compare
Video Object Segmentation DAVIS 2017 (val) BATMAN Mean Jaccard & F-Measure 86.2 # 2
Compare
F-measure 89.3 # 3
Compare
Video Object Segmentation DAVIS 2017 (val) CFBI+ Mean Jaccard & F-Measure 82.9 # 9
Compare
Jaccard 80.1 # 9
Compare
F-measure 85.7 # 9
Compare
Video Object Segmentation DAVIS 2017 (val) STCN Mean Jaccard & F-Measure 85.4 # 4
Compare
Jaccard 82.2 # 4
Compare
F-measure 88.6 # 4
Compare
Video Object Segmentation DAVIS 2017 (val) AOT Mean Jaccard & F-Measure 84.9 # 5
Compare
Jaccard 82.3 # 3
Compare
F-measure 87.5 # 5
Compare
Video Object Segmentation DAVIS 2017 (val) TransVOS Mean Jaccard & F-Measure 83.9 # 6
Compare
Jaccard 81.4 # 5
Compare
F-measure 86.4 # 7
Compare
Video Object Segmentation YouTube-VOS 2018 STM Jaccard (Seen) 79.7 # 16
Compare
Jaccard (Unseen) 72.8 # 15
Compare
F-Measure (Seen) 84.2 # 13
Compare
F-Measure (Unseen) 80.9 # 13
Compare
Mean Jaccard & F-Measure 79.4 # 14
Compare
Video Object Segmentation YouTube-VOS 2018 RMN Jaccard (Seen) 82.1 # 7
Compare
Jaccard (Unseen) 75.7 # 11
Compare
F-Measure (Seen) 85.7 # 10
Compare
F-Measure (Unseen) 82.4 # 12
Compare
Video Object Segmentation YouTube-VOS 2018 SST Jaccard (Seen) 81.2 # 12
Compare
Jaccard (Unseen) 76 # 10
Compare
Mean Jaccard & F-Measure 81.7 # 10
Compare
Video Object Segmentation YouTube-VOS 2018 STCN Jaccard (Seen) 81.9 # 9
Compare
Jaccard (Unseen) 77.9 # 6
Compare
F-Measure (Seen) 86.5 # 8
Compare
F-Measure (Unseen) 85.7 # 6
Compare
Mean Jaccard & F-Measure 83 # 6
Compare
Video Object Segmentation YouTube-VOS 2018 CFBI+ Jaccard (Seen) 81.8 # 10
Compare
Jaccard (Unseen) 77.1 # 7
Compare
F-Measure (Seen) 86.6 # 7
Compare
F-Measure (Unseen) 85.6 # 7
Compare
Mean Jaccard & F-Measure 82.8 # 7
Compare
Video Object Segmentation YouTube-VOS 2018 RPCMVOS Jaccard (Seen) 83.1 # 5
Compare
Jaccard (Unseen) 78.5 # 4
Compare
F-Measure (Seen) 87.7 # 5
Compare
F-Measure (Unseen) 86.7 # 4
Compare
Mean Jaccard & F-Measure 84 # 5
Compare
Video Object Segmentation YouTube-VOS 2018 AOT Jaccard (Seen) 83.7 # 4
Compare
Jaccard (Unseen) 78.1 # 5
Compare
F-Measure (Seen) 88.5 # 4
Compare
F-Measure (Unseen) 86.1 # 5
Compare
Mean Jaccard & F-Measure 84.1 # 4
Compare
Video Object Segmentation YouTube-VOS 2018 LCM Jaccard (Seen) 82.2 # 6
Compare
Mean Jaccard & F-Measure 82 # 8
Compare
Video Object Segmentation YouTube-VOS 2018 KMN Jaccard (Seen) 81.4 # 11
Compare
Jaccard (Unseen) 75.3 # 12
Compare
F-Measure (Seen) 85.6 # 11
Compare
F-Measure (Unseen) 83.3 # 10
Compare
Mean Jaccard & F-Measure 81.4 # 12
Compare
Video Object Segmentation YouTube-VOS 2018 AFB-URR Jaccard (Seen) 78.8 # 17
Compare
Jaccard (Unseen) 74.1 # 14
Compare
F-Measure (Seen) 83.1 # 14
Compare
F-Measure (Unseen) 82.6 # 11
Compare
Mean Jaccard & F-Measure 79.6 # 13
Compare
Video Object Segmentation YouTube-VOS 2018 LWL Jaccard (Seen) 80.4 # 15
Compare
Jaccard (Unseen) 76.4 # 9
Compare
F-Measure (Seen) 84.9 # 12
Compare
F-Measure (Unseen) 84.4 # 8
Compare
Mean Jaccard & F-Measure 81.5 # 11
Compare
Video Object Segmentation YouTube-VOS 2018 CFBI Jaccard (Seen) 81.1 # 13
Compare
F-Measure (Seen) 85.8 # 9
Compare
Visual Object Tracking YouTube-VOS 2018 KMN Jaccard (Unseen) 75.3 # 2
Compare
Visual Object Tracking YouTube-VOS 2018 CFBI F-Measure (Unseen) 83.4 # 1
Compare
Visual Object Tracking YouTube-VOS 2018 RMN Jaccard (Unseen) 75.7 # 1
Compare
Visual Object Tracking YouTube-VOS 2018 TransVOS F-Measure (Seen) 86.7 # 1
Compare
F-Measure (Unseen) 83.4 # 1
Compare
Video Object Segmentation YouTube-VOS 2018 TransVOS Jaccard (Seen) 82 # 8
Compare
Jaccard (Unseen) 75 # 13
Compare
F-Measure (Seen) 86.7 # 6
Compare
F-Measure (Unseen) 83.4 # 9
Compare
Mean Jaccard & F-Measure 81.8 # 9
Compare
Video Object Segmentation YouTube-VOS 2018 BATMAN Jaccard (Seen) 84.7 # 2
Compare
Jaccard (Unseen) 79.2 # 3
Compare
F-Measure (Seen) 89.8 # 2
Compare
F-Measure (Unseen) 87.4 # 3
Compare
Mean Jaccard & F-Measure 85.3 # 3
Compare
Video Object Segmentation YouTube-VOS 2019 SST Mean Jaccard & F-Measure 81.8 # 8
Compare
Jaccard (Seen) 80.9 # 8
Compare
Jaccard (Unseen) 76.6 # 8
Compare
Video Object Segmentation YouTube-VOS 2019 CFBI+ Mean Jaccard & F-Measure 82.6 # 7
Compare
Jaccard (Seen) 81.7 # 6
Compare
Jaccard (Unseen) 77.1 # 7
Compare
F-Measure (Seen) 86.2 # 6
Compare
F-Measure (Unseen) 85.2 # 7
Compare
Video Object Segmentation YouTube-VOS 2019 STCN Mean Jaccard & F-Measure 82.7 # 6
Compare
Jaccard (Seen) 81.1 # 7
Compare
Jaccard (Unseen) 78.2 # 6
Compare
F-Measure (Seen) 85.4 # 7
Compare
F-Measure (Unseen) 85.9 # 6
Compare
Video Object Segmentation YouTube-VOS 2019 RPCMVOS Mean Jaccard & F-Measure 83.9 # 5
Compare
Jaccard (Seen) 82.6 # 5
Compare
Jaccard (Unseen) 79.1 # 3
Compare
F-Measure (Seen) 86.9 # 5
Compare
F-Measure (Unseen) 87.1 # 4
Compare
Video Object Segmentation YouTube-VOS 2019 AOT Mean Jaccard & F-Measure 84.1 # 4
Compare
Jaccard (Seen) 83.5 # 4
Compare
Jaccard (Unseen) 78.4 # 5
Compare
F-Measure (Seen) 88.1 # 4
Compare
F-Measure (Unseen) 86.3 # 5
Compare
Video Object Segmentation YouTube-VOS 2019 BATMAN Mean Jaccard & F-Measure 85 # 3
Compare
Jaccard (Seen) 84.5 # 2
Compare
Jaccard (Unseen) 79 # 4
Compare
F-Measure (Seen) 89.3 # 2
Compare
F-Measure (Unseen) 87.2 # 3
Compare
Video Object Segmentation YouTube-VOS 2019 CFBI Mean Jaccard & F-Measure 81 # 9
Compare
Jaccard (Seen) 80.6 # 9
Compare
Jaccard (Unseen) 75.2 # 9