SAM 2: Segment Anything in Images and Videos
We present Segment Anything Model 2 (SAM 2), a foundation model towards solving promptable visual segmentation in images and videos. We build a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. Our model is a simple transformer architecture with streaming memory for real-time video processing. SAM 2 trained on our data provides strong performance across a wide range of tasks. In video segmentation, we observe better accuracy, using 3x fewer interactions than prior approaches. In image segmentation, our model is more accurate and 6x faster than the Segment Anything Model (SAM). We believe that our data, model, and insights will serve as a significant milestone for video segmentation and related perception tasks. We are releasing our main model, dataset, as well as code for model training and our demo.
PDF AbstractCode
Colab
Cityscapes
LVIS
DAVIS 2017
YouTube-VOS 2018
SA-1B
FBMS
GTEA
Hypersim
Referring Expressions for DAVIS 2016 & 2017
OVIS
WoodScape
Virtual KITTI 2
MOSE
UVO
Cityscapes-VPS
LVOS
TrashCan
DiDi
VOST
EgoHOS
NDD20
PIDray
iShape
Lindenthal Camera Traps
Results from the Paper
Ranked #1 on
Semi-Supervised Video Object Segmentation
on MOSE
(using extra training data)
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Semi-Supervised Video Object Segmentation | DAVIS 2017 (val) | SAM2 | J&F | 90.7 | # 1 | ||
| Params(M) | 224.4 | # 23 | |||||
| Visual Object Tracking | DiDi | SAM2.1 | Tracking quality | 0.649 | # 3 | ||
| Semi-Supervised Video Object Segmentation | MOSE | SAM2 | J&F | 77.9 | # 1 | ||
| Visual Object Tracking | VOT2022 | SAM2.1 | EAO | 0.692 | # 2 |
