Spatio-Temporal Failure Mode Analysis

The Spatio-Temporal Failure Mode Analysis Test is designed to analyze the model's performance on the spatio-temporal dataset.

Execute Test:

test_session = TestSession(
    project_name="Cropin",
    run_name=run_name,
    access_key=ACCESS_KEY,
    secret_key=SECRET_KEY,
    host=HOST
)

rules = FMARules()

rules.add(metric='F1Score',metric_threshold=0.2,conf_threshold=0.4,iou_threshold=0.8,background_label= "background",include_background=True,type="label",clazz=["ALL"])
rules.add(metric="PixelAccuracy",metric_threshold=0.7,conf_threshold=0.5,iou_threshold=0.5, clazz=["ALL"], type="label", background_label="Unlabelled", include_background=True)
rules.add(metric = "Precision",  metric_threshold = 0.3,conf_threshold=0.5,iou_threshold=0.5, clazz=["ALL"], type="label", background_label="background", include_background=True)
rules.add(metric="wIoU", metric_threshold=0.1,conf_threshold=0.5,iou_threshold=0.5, weights={"road": 100, "unlabelled": 1}, type="label", clazz=["ALL"])
rules.add(metric="Recall", metric_threshold=0.2,conf_threshold=0.5,iou_threshold=0.5, clazz=["ALL"], type="label", background_label="Unlabelled", include_background=True)

cls_default = clustering(test_session=test_session,
                         dataset_name=dataset_name,
                         method="k-means",
                         embedding_col="ImageEmbedding",
                         level="image",
                         args={"numOfClusters": 5}
                         )

edge_case_detection = failure_mode_analysis_geospatial(test_session=test_session,
                                            dataset_name=dataset_name,
                                            test_name="fma_semantic_geospatial_1",
                                            modelColumnName="ModelInferences",
                                            gtColumnName="LabelPath",
                                            rules=rules,
                                            outputType="semantic_segmentation",
                                            embeddingColumnName="ImageEmbedding",
                                            clustering=cls_default,
                                            labelMapping={
                                                },
                                            type="fma",
                                            primary_metadata= "place")

test_session.add(edge_case_detection)

test_session.run()

Use the FMARules() to configure various metrics such as F1Score, Pixel Accuracy, Precision, Weighted Intersection over Union (wIoU), and Recall. These metrics are instrumental in assessing different aspects of model performance on labeled geospatial data.

Execute Test:

• Implement the failure_mode_analysis_geospatial() function with the necessary parameters to conduct an in-depth analysis of the geospatial segments:

  • test_session: The session managing tests.

  • dataset_name: Dataset under analysis.

  • modelColumnName and gtColumnName: Columns for model inferences and ground truth labels.

Add Test to Session:

• Register the test with the session using test_session.add().

Run Test:

• Execute the session's tests with test_session.run().

By following these steps, you have successfully set up and executed a Spatio-Temporal Failure Mode Analysis Test on the RagaAI Testing Platform.

Post-execution, review the results to identify and remove or handle duplicates as necessary.

Analyzing Test Results:

Understanding Clustering:

• Cluster Analysis: Users can quickly pinpoint clusters that perform poorly and evaluate their impact on the overall model.

• Identifying Underperforming Clusters: Users can delve deeper into particular clusters or spatio-temporal data points to understand the underlying reasons for underperformance.

Data Analysis:

• Switch to Analysis Tab: To get a detailed performance report, go to the Analysis tab.

• View Performance Metrics: Evaluate metrics like label-wise performance and temporal graphs.

• Confusion Matrix: The class-based confusion matrix in Spatio Temporal Failure Mode Analysis provides a detailed breakdown of performance for each class. Users can view the confusion matrix in three ways:

  • Absolute: Shows the absolute number of pixels.

  • Normalised (Ground Truth): Normalises values with respect to the ground truth.

  • Normalised (Model): Normalises values with respect to model inference.

By following these steps, users can efficiently leverage the Spatio Temporal Failure Mode Analysis test to obtain a thorough insight into their model's performance, pip-point key areas for improvement, and make data-driven decisions to enhance model accuracy and reliability.