Evaluation clustering config

Configuration to evaluate a trained model for object discovery by clustering object representations.

Given a set of images, each with a set of ground truth masks and a set of object masks and representations, we perform the following steps:

1. Assign each object a cluster id by clustering the corresponding representations over all images.
2. Merge object masks with the same cluster id on the same image to form a semantic mask.
3. Compute IoU between masks of predicted clusters and ground truth classes over all images.
4. Assign clusters to classes based on the IoU and a matching strategy.

In contrast to most other configurations, the clustering evaluation configuration is defined programmatically in python. The definition can be found below:

ocl/cli/eval_cluster_metrics.py:EvaluationClusteringConfig

@dataclasses.dataclass
class EvaluationClusteringConfig:
    """Configuration for evaluation."""

    # Path to training configuration file or configuration dir. If dir, train_config_name
    # needs to be set as well.
    train_config_path: str

    # Number of classes. Note that on COCO, this should be one larger than the maximum class ID that
    # can appear, which does not correspond to the real number of classes.
    n_classes: int
    # Clustering methods to get cluster ID per object by clustering representations
    # This only supports clustering metrics.
    clusterings: Optional[Dict[str, Any]] = None
    # Paths for model outputs to get cluster ID per object
    model_clusterings: Optional[Dict[str, str]] = None

    train_config_overrides: Optional[List[str]] = None
    train_config_name: Optional[str] = None
    checkpoint_path: Optional[str] = None
    output_dir: Optional[str] = None
    report_filename: str = "clustering_metrics.json"

    batch_size: int = 25
    class_discovery_threshold: float = 0.02
    use_mask_threshold: bool = False
    mask_threshold: float = 0.5
    ignore_background: bool = False
    use_unmatched_as_background: bool = False
    use_ignore_masks: bool = False
    n_min_mask_pixels: int = 1  # Minimum number of pixels a mask must occupy to be considered valid
    n_min_max_mask_values: float = 1e-4  # Mask must have at least one value above threshold

    # Type of representation to use for clustering.
    representation_type: RepresentationType = RepresentationType.SLOTS

    # Setting this allows to add modules to the model that are executed during evaluation
    modules: Optional[Dict[str, Any]] = None
    # Setting this allows to evaluate on a different dataset than the model was trained on
    dataset: Optional[Any] = None

    # Path to slot representations
    slots_path: str = "perceptual_grouping.objects"
    # Path to feature representations
    features_path: str = "feature_extractor.features"
    # Path to slot masks, image shaped
    masks_path: str = "object_decoder.masks_as_image"
    # Path to slot masks, but flattened to match the size of features
    masks_flat_path: str = "object_decoder.masks"
    # Path to reference masks
    target_masks_path: str = "input.segmentation_mask"
    # Path to ignore masks
    ignore_masks_path: str = "input.ignore_mask"
    # Path under which representation to cluster is stored
    cluster_representation_path: str = "representation"
    # Path under which empty slot mask is stored
    empty_slots_path: str = "empty_slots"

    class_name_by_category_id: Optional[Dict[int, str]] = None