Segment Anything论文阅读笔记

Abstract

Segment Anything (SA) project: a new task, model, and dataset for image segmentation.

we built the largest segmentation dataset to date (by far:迄今为止), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks.The Segment Anything Model (SAM) and corresponding dataset (SA-1B) releasing at SA to foster research into foundation models for computer vision.

Introduction

Large language models pre-trained on web-scale datasets are revolutionizing NLP （彻底改变）with strong zero-shot and few-shot generalization. These “foundation models” can generalize to tasks and data distributions beyond those seen during training. （zero-shot and few-shot generalization零样本和少样本泛化）

Foundation models have also been explored in computer vision ，albeit to a lesser extent. （尽管程度较小）

Our goal is to build a foundation model for image segmentation. That is, we seek to develop a promptable model and pre-train it on a broad dataset using a task that enables powerful generalization. With this model, we aim to solve a range of downstream segmentation problems on new data distributions using prompt engineering.

The success of this plan hinges on（取决于）three components: task, model, and data. To develop them, we address the following questions about image segmentation:

1. What task will enable zero-shot generalization?

2. What is the corresponding model architecture?

3. What data can power this task and model?

These questions are entangled and require a comprehen- sive solution.（错综复杂需要一个综合的解决方案。）

Surprisingly, we find that a simple design satisfies all three constraints: a powerful image encoder computes an image embedding, a prompt encoder embeds prompts, and then the two information sources are combined in a lightweight mask decoder that predicts segmentation masks. We refer to this model as the Segment Anything Model, or SAM .

data engine has three stages:

assisted-manual
semi-automatic
and fully automatic

2. Segment Anything Task