# Spatio-Temporal Failure Mode Analysis

### Execute Test:

```
test_session = TestSession(
    project_name="Cropin",
    run_name=run_name,
    access_key=ACCESS_KEY,
    secret_key=SECRET_KEY,
    host=HOST
)

rules = FMARules()

rules.add(metric='F1Score',metric_threshold=0.2,conf_threshold=0.4,iou_threshold=0.8,background_label= "background",include_background=True,type="label",clazz=["ALL"])
rules.add(metric="PixelAccuracy",metric_threshold=0.7,conf_threshold=0.5,iou_threshold=0.5, clazz=["ALL"], type="label", background_label="Unlabelled", include_background=True)
rules.add(metric = "Precision",  metric_threshold = 0.3,conf_threshold=0.5,iou_threshold=0.5, clazz=["ALL"], type="label", background_label="background", include_background=True)
rules.add(metric="wIoU", metric_threshold=0.1,conf_threshold=0.5,iou_threshold=0.5, weights={"road": 100, "unlabelled": 1}, type="label", clazz=["ALL"])
rules.add(metric="Recall", metric_threshold=0.2,conf_threshold=0.5,iou_threshold=0.5, clazz=["ALL"], type="label", background_label="Unlabelled", include_background=True)

cls_default = clustering(test_session=test_session,
                         dataset_name=dataset_name,
                         method="k-means",
                         embedding_col="ImageEmbedding",
                         level="image",
                         args={"numOfClusters": 5}
                         )

edge_case_detection = failure_mode_analysis_geospatial(test_session=test_session,
                                            dataset_name=dataset_name,
                                            test_name="fma_semantic_geospatial_1",
                                            modelColumnName="ModelInferences",
                                            gtColumnName="LabelPath",
                                            rules=rules,
                                            outputType="semantic_segmentation",
                                            embeddingColumnName="ImageEmbedding",
                                            clustering=cls_default,
                                            labelMapping={
                                                },
                                            type="fma",
                                            primary_metadata= "place")

test_session.add(edge_case_detection)

test_session.run()
```

Use the `FMARules()` to configure various metrics such as F1Score, Pixel Accuracy, Precision, Weighted Intersection over Union (wIoU), and Recall. These metrics are instrumental in assessing different aspects of model performance on labeled geospatial data.

**Execute Test:**

* Implement the `failure_mode_analysis_geospatial()` function with the necessary parameters to conduct an in-depth analysis of the geospatial segments:
  * `test_session`: The session managing tests.
  * `dataset_name`: Dataset under analysis.
  * `modelColumnName` and `gtColumnName`: Columns for model inferences and ground truth labels.

**Add Test to Session:**

* Register the test with the session using `test_session.add()`.

**Run Test:**

* Execute the session's tests with `test_session.run()`.

By following these steps, you have successfully set up and executed a Spatio-Temporal Failure Mode Analysis Test on the RagaAI Testing Platform.

Post-execution, review the results to identify and remove or handle duplicates as necessary.

### **Analyzing Test Results:**

<figure><img src="/files/QmfDieVx16smP923mtFQ" alt=""><figcaption></figcaption></figure>

#### Understanding Clustering:

* **Cluster Analysis:** Users can quickly pinpoint clusters that perform poorly and evaluate their impact on the overall model.
* **Identifying Underperforming Clusters:** Users can delve deeper into particular clusters or spatio-temporal data points to understand the underlying reasons for underperformance.

<figure><img src="/files/x1nfrYNmKFDX1lY52fSz" alt=""><figcaption></figcaption></figure>

#### Data Analysis:

* **Switch to Analysis Tab**: To get a detailed performance report, go to the Analysis tab.
* **View Performance Metrics**: Evaluate metrics like label-wise performance and temporal graphs.
* **Confusion Matrix**: The class-based confusion matrix in Spatio Temporal Failure Mode Analysis provides a detailed breakdown of performance for each class. Users can view the confusion matrix in three ways:
  * Absolute: Shows the absolute number of pixels.
  * Normalised (Ground Truth): Normalises values with respect to the ground truth.
  * Normalised (Model): Normalises values with respect to model inference.

<figure><img src="/files/pZ9DiZdSs2v3ebimi5gY" alt=""><figcaption></figcaption></figure>

<figure><img src="/files/qjUoDoCmdyJtvL9AOdGl" alt=""><figcaption></figcaption></figure>

By following these steps, users can efficiently leverage the Spatio Temporal Failure Mode Analysis test to obtain a thorough insight into their model's performance, pip-point key areas for improvement, and make data-driven decisions to enhance model accuracy and reliability.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/spatio-temporal-failure-mode-analysis.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.