Skip to content

ocl.cli.eval_cluster_metrics

Evaluate a trained model for object discovery by clustering object representations.

Given a set of images, each with a set of ground truth masks and a set of object masks and representations, we perform the following steps: 1) Assign each object a cluster id by clustering the corresponding representations over all images. 2) Merge object masks with the same cluster id on the same image to form a semantic mask. 3) Compute IoU between masks of predicted clusters and ground truth classes over all images. 4) Assign clusters to classes based on the IoU and a matching strategy.

EvaluationClusteringConfig dataclass

Configuration for evaluation.

Source code in ocl/cli/eval_cluster_metrics.py 
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
@dataclasses.dataclass
class EvaluationClusteringConfig:
    """Configuration for evaluation."""

    # Path to training configuration file or configuration dir. If dir, train_config_name
    # needs to be set as well.
    train_config_path: str

    # Number of classes. Note that on COCO, this should be one larger than the maximum class ID that
    # can appear, which does not correspond to the real number of classes.
    n_classes: int
    # Clustering methods to get cluster ID per object by clustering representations
    # This only supports clustering metrics.
    clusterings: Optional[Dict[str, Any]] = None
    # Paths for model outputs to get cluster ID per object
    model_clusterings: Optional[Dict[str, str]] = None

    train_config_overrides: Optional[List[str]] = None
    train_config_name: Optional[str] = None
    checkpoint_path: Optional[str] = None
    output_dir: Optional[str] = None
    report_filename: str = "clustering_metrics.json"

    batch_size: int = 25
    class_discovery_threshold: float = 0.02
    use_mask_threshold: bool = False
    mask_threshold: float = 0.5
    ignore_background: bool = False
    use_unmatched_as_background: bool = False
    use_ignore_masks: bool = False
    n_min_mask_pixels: int = 1  # Minimum number of pixels a mask must occupy to be considered valid
    n_min_max_mask_values: float = 1e-4  # Mask must have at least one value above threshold

    # Type of representation to use for clustering.
    representation_type: RepresentationType = RepresentationType.SLOTS

    # Setting this allows to add modules to the model that are executed during evaluation
    modules: Optional[Dict[str, Any]] = None
    # Setting this allows to evaluate on a different dataset than the model was trained on
    dataset: Optional[Any] = None

    # Path to slot representations
    slots_path: str = "perceptual_grouping.objects"
    # Path to feature representations
    features_path: str = "feature_extractor.features"
    # Path to slot masks, image shaped
    masks_path: str = "object_decoder.masks_as_image"
    # Path to slot masks, but flattened to match the size of features
    masks_flat_path: str = "object_decoder.masks"
    # Path to reference masks
    target_masks_path: str = "input.segmentation_mask"
    # Path to ignore masks
    ignore_masks_path: str = "input.ignore_mask"
    # Path under which representation to cluster is stored
    cluster_representation_path: str = "representation"
    # Path under which empty slot mask is stored
    empty_slots_path: str = "empty_slots"

    class_name_by_category_id: Optional[Dict[int, str]] = None