Failure Mode Analysis

Failure Mode Analysis is a test that allows users to deeply analyse their model's performance.

Failure Mode Analysis enables you to define rules on metrics by setting thresholds. Based on rules, get a sorted list of all clusters that are breaching the threshold and clusters that are within the threshold. RagaAI will create clusters using the embeddings of the Dataset. Clusters that underperform and breaches the threshold will be highlighted on the Issue Stats screen.

Execute Test

The following code snippet is designed to perform a Failure Mode Analysis test on a specified dataset within the RagaAI environment.

First, you define the rules that will be used to evaluate the model's performance. These rules are based on metrics such as BLEU, Cosine Similarity, METEOR, and ROUGE (Refer Metric Glossary).

rules = FMARules()
rules.add(metric = "F1Score", metric_threshold = 0.86, label = "ALL", type="label", background_label="Unlabelled", include_background=False)
rules.add(metric = "Precision", metric_threshold = 0.86, label = "ALL", type="label", background_label="Unlabelled", include_background=True)
rules.add(metric = "wIoU", metric_threshold = 0.5, weights={"Road": 100, "Unlabelled": 1}, type="label", label = "ALL")

cls_default = clustering(method="k-means", embedding_col="ImageVectorsM1", level="image", args={"numOfClusters": 7})

edge_case_detection = failure_mode_analysis(test_session=test_session,
                                            dataset_name = "satellite_image_dataset",
                                            test_name = "Test",
                                            model = "ModelB",
                                            gt = "ModelA",
                                            rules = rules,
                                            output_type="semantic_segmentation",
                                            type="embedding",
                                            clustering=cls_default
                                            )
                                            
test_session.add(edge_case_detection)

test_session.run()

• FMARules(): Initialises the rules for the FMA test.

• rules.add(): Adds a new rule with specific parameters:

  • metric: The performance metric to evaluate (e.g., F1Score, Precision).

  • metric_threshold: The minimum acceptable value for the metric.

  • label: Specifies the label(s) these metrics apply to. "ALL" means all labels.

  • type: The type of rule, which is "label" in this case.

  • background_label: Indicates the label used for background or default class.

  • include_background: Determines whether to include the background label in the analysis.

  • weights: In the case of wIoU, sets the importance of each label in the metric.

Next, you define the clustering parameters to group similar failure modes.

• clustering(): Sets the clustering method for grouping failure modes.

• method: The clustering technique used, "k-means" in this case.

• embedding_col: The column containing embedding vectors to be used for clustering.

• level: The level at which clustering is applied, "image" here indicates that each image is treated as an individual data point.

• args: Additional arguments for the clustering method, such as the number of clusters.

test_session.add(): Registers the failure mode analysis test with the test session.

test_session.run(): Starts the execution of all the tests added to the session, including the FMA.

Following this guide, you've successfully set up and initiated a Failure Mode Analysis test within RagaAI.

Analysing Test Results

Clustering (Embedding View)

1. Navigate to Embedding View: Find the interactive visualisation interface in RagaAI.

2. Visualise Clusters: Utilise the tool to visualise complex clustering, uncovering hidden patterns and structures in the data.

3. Select Data Points: Use the lasso tool to select specific data points of interest and analyse their results.

Visualising Data

1. Grid View: Access the grid view to see data points within the selected clusters.

2. Data Filtering: Use this feature to focus on specific subsets of your dataset that meet certain conditions, helping to extract meaningful patterns and trends.

Understanding Clustering and Threshold Breaches

• Cluster Analysis: RagaAI creates clusters using the embeddings of the dataset. These clusters group similar data points together.

• Identifying Underperforming Clusters: Clusters that underperform (breaching the threshold) will be highlighted on the Issue Stats screen.

Navigating and Interpreting Results

• Directly Look at Problematic Clusters: Users can quickly identify clusters responsible for underperformance and assess their impact on the overall model.

• In-Depth Analysis: Dive deeper into specific clusters or data points to understand the root causes of underperformance.

Data Analysis

1. Switch to Analysis Tab: To get a detailed performance report, go to the Analysis tab.

2. View Performance Metrics: Examine metrics like label-wise performance and temporal graphs.

3. Confusion Matrix: The class-based confusion matrix in Failure Mode Analysis provides a detailed breakdown of performance for each class. Users can view the confusion matrix in three ways:

   • Absolute: Shows the absolute number of pixels.

   • Normalised (Ground Truth): Normalises values with respect to the ground truth.

   • Normalised (Model): Normalises values with respect to model inference.

Practical Tips

• Set Realistic Thresholds: Choose thresholds that reflect the expected performance of your model.

• Leverage Visual Tools: Make full use of RagaAI’s visualisation capabilities to gain insights that might not be apparent from raw data alone.

By following these steps, users can efficiently leverage the Failure Mode Analysis test to gain a comprehensive understanding of their model's performance, identify key areas for improvement, and make data-driven decisions to enhance model accuracy and reliability.