Segment Anything (SAM): Promptable Segmentation and the SA-1B Dataset

Segmentation is the computer vision task of marking exactly which pixels belong to an object, rather than drawing a box around it. Meta AI's Segment Anything project, described in the paper Segment Anything submitted on 5 April 2023 by Alexander Kirillov, Eric Mintun, Nikhila Ravi and colleagues, reframed it as a promptable task: give the model a hint about what you want, and it returns a mask.

What "promptable" means

The Segment Anything Model (SAM) accepts several kinds of prompt:

• Points - a click on the object, optionally with clicks marking background to exclude.
• Boxes - a rough rectangle around the object.
• Masks - a coarse earlier mask to refine.
• Text - explored in the paper as a proof of concept rather than a released feature.

A single click is often ambiguous. A click on a shirt could mean the shirt, the person wearing it, or a pocket. SAM handles this by returning three candidate masks at different levels of granularity, each with a predicted quality score, rather than guessing one.

Three parts, split for speed

SAM is built from three components:

1. Image encoder - a large Vision Transformer pretrained with masked autoencoding. It runs once per image and produces an embedding.
2. Prompt encoder - turns points, boxes and masks into embeddings.
3. Mask decoder - a lightweight Transformer that combines the image embedding and prompt embeddings into masks.

The split is the design decision that matters in practice. The expensive encoder runs once; after that, each new prompt only runs the small decoder, which the paper reports takes about 50 milliseconds in a web browser. That makes interactive use - click, see a mask, click again - possible.

SA-1B: the dataset built alongside the model

The paper's dataset, SA-1B, contains "over 1 billion masks on 11M licensed and privacy respecting images" - by the authors' account far more masks than any earlier segmentation dataset. It was built with a three-stage "data engine" in which the model and the data improved each other:

• Assisted-manual - annotators labelled masks using an early SAM as an interactive tool.
• Semi-automatic - SAM proposed masks for confident objects, and annotators labelled what was missed.
• Fully automatic - SAM was prompted with a regular grid of points on each image and generated masks with no annotator involved. Nearly all of SA-1B's masks come from this stage.

The authors evaluated SAM zero-shot - on data it was not trained for - across 23 segmentation datasets, and report its single-point masks were often competitive with, or better than, earlier fully supervised results.

SAM 2: from images to video

SAM 2: Segment Anything in Images and Videos, first submitted on 1 August 2024, extended the same promptable approach to video. Its architecture adds a streaming memory so that an object selected in one frame can be tracked through later frames in real time. The paper reports that on video it reaches better accuracy than prior approaches while needing 3x fewer user interactions, and that on images it is more accurate than the original SAM and 6x faster. Meta released the model, the training code, a demo and a new video segmentation dataset alongside it.

Where it fits

Segmentation sits between classification ("what is in this image") and detection ("where is it, roughly") in the vision task ladder covered in Computer Vision Explained. Promptable models like SAM are typically used as a building block - for annotation tools, image editing, and as a first step that hands precise object masks to another model.