When to use instance, and when to use semantic segmentation?

Hi everyone,

I know the theoretical difference between instance and semantic segmentation. However, I how to decide which one to use which on practical projects? Does anyone have experience with this?

This question is somewhat difficult to answer and slightly philosophical in nature.

The intention for “semantically” segmenting an image is to label every pixel in the image according to a set of defined semantic classes (in Hasty you don’t label the “background”, but we assume everything left unlabeled in the image is background class).
Whereas for instance segmentation - as the name implies - is for differentiating between every instance of a class as well.

So it depends on what the requirements for your use case are? Do you need to differentiate between multiple instances of something (person a, person b) or you don’t care about the instances (person)?
Does your project contain categories that are hard to define as “objects”? like the sky - there is only ever one instance of it, or the road where there can multiple instances of a road in an image but it can be philosophically hard to argue what comprises an “instance” of a road.

TL;DR: depends on your use case. Do you care about the counts/instances of a class or not?


1 Like

Thanks for the answer! Should the type of hardware I use for deployment have any impact on the decision?