Failure Mode Analysis

Failure Mode Analysis is a test that allows users to deeply analyse their model's performance.

Failure Mode Analysis enables you to define rules on metrics by setting thresholds. Based on rules, get a sorted list of all clusters that are breaching the threshold and clusters that are within the threshold. RagaAI will create clusters using the embeddings of the Dataset. Clusters that underperform and breaches the threshold will be highlighted on the Issue Stats screen.

Execute Test

The following code snippet is designed to perform a Failure Mode Analysis test on a specified dataset within the RagaAI environment.

First, you define the rules that will be used to evaluate the model's performance. These rules are based on metrics such as F1Score, Precision, Recall (Refer Metric Glossary).

rules = FMARules()
rules.add(metric='F1Score',  metric_threshold=0.5, label='ALL', conf_threshold=0.5, iou_threshold=0.5)
rules.add(metric='Recall',  metric_threshold=0.5,  label='ALL', conf_threshold=0.5, iou_threshold=0.5)
rules.add(metric='Precision',  metric_threshold=0.5, label='ALL', conf_threshold=0.5, iou_threshold=0.5)

dataset_name = "validation_dataset"

cls_default = clustering(test_session=test_session,
                         dataset_name=dataset_name,
                         method="k-means",
                         embedding_col="Embedding",
                         level="image",
                         args={"numOfClusters": 7}
                         )


edge_case_detection = failure_mode_analysis(test_session=test_session,
                                            dataset_name=dataset_name,
                                            test_name=run_name,
                                            model="RagaModel",
                                            gt="GT",
                                            rules=rules,
                                            type="fma",
                                            output_type="instance_segmentation",
                                            clustering=cls_default,
                                            embedding_col_name="Embedding")


test_session.add(edge_case_detection)
test_session.run()

• FMARules(): Initialises the rules for the FMA test.

• rules.add(): Adds a new rule with specific parameters:

  • metric: The performance metric to evaluate (e.g., F1Score, Precision).

  • metric_threshold: The minimum acceptable value for the metric.

  • label: Specifies the label(s) these metrics apply to. Use "ALL" to apply to all labels.

  • conf_threshold: Threshold for confidence score.

  • iou_threshold: Threshold for intersection over union (IOU) for geometry mapping.

Next, you define the clustering parameters to group similar failure modes.

• clustering(): Sets the clustering method for grouping failure modes.

• method: The clustering technique used, "k-means" in this case.

• embedding_col: The column containing embedding vectors to be used for clustering.

• level: The level at which clustering is applied, "image" here indicates that each image is treated as an individual data point.

• args: Additional arguments for the clustering method, such as the number of clusters.

Use the failure_mode_analysis() function to execute the FMA test with the following parameters:

• test_session: The session object managing tests.

• dataset_name: Name of the dataset to be tested.

• test_name: Name of the test run.

• model: Name of the machine learning model to be evaluated.

• gt: Ground truth data column name.

• rules: Predefined rules for the FMA.

• type: Type of test, which should be set to "fma".

• output_type: Type of output expected from the model.

• clustering: Clustering object obtained earlier.

• embedding_col_name: Name of the column containing embeddings.

test_session.add(): Registers the failure mode analysis test with the test session.

test_session.run(): Starts the execution of all the tests added to the session, including the FMA.

Following this guide, you've successfully set up and initiated a Failure Mode Analysis test within RagaAI.

Analysing Test Results

Clustering (Embedding View)

1. Navigate to Embedding View: Find the interactive visualisation interface in RagaAI.

2. Visualise Clusters: Utilise the tool to visualise complex clustering, uncovering hidden patterns and structures in the data.

3. Select Data Points: Use the lasso tool to select specific data points of interest and analyse their results.

Visualising Data

1. Grid View: Access the grid view to see data points within the selected clusters.

2. Data Filtering: Use this feature to focus on specific subsets of your dataset that meet certain conditions, helping to extract meaningful patterns and trends.

Understanding Clustering and Threshold Breaches

• Cluster Analysis: RagaAI creates clusters using the embeddings of the dataset. These clusters group similar data points together.

• Identifying Underperforming Clusters: Clusters that underperform (breaching the threshold) will be highlighted on the Issue Stats screen.

Navigating and Interpreting Results

• Directly Look at Problematic Clusters: Users can quickly identify clusters responsible for underperformance and assess their impact on the overall model.

• In-Depth Analysis: Dive deeper into specific clusters or data points to understand the root causes of underperformance.

Data Analysis

1. Switch to Analysis Tab: To get a detailed performance report, go to the Analysis tab.

2. View Performance Metrics: Examine metrics like label-wise performance and temporal graphs.

3. Confusion Matrix: The class-based confusion matrix in Failure Mode Analysis provides a detailed breakdown of performance for each class. Users can view the confusion matrix in three ways:

   • Absolute: Shows the absolute number of pixels.

   • Normalised (Ground Truth): Normalises values with respect to the ground truth.

   • Normalised (Model): Normalises values with respect to model inference.

Practical Tips

• Set Realistic Thresholds: Choose thresholds that reflect the expected performance of your model.

• Leverage Visual Tools: Make full use of RagaAI’s visualisation capabilities to gain insights that might not be apparent from raw data alone.

By following these steps, users can efficiently leverage the Failure Mode Analysis test to gain a comprehensive understanding of their model's performance, identify key areas for improvement, and make data-driven decisions to enhance model accuracy and reliability.