Failure Mode Analysis

Failure Mode Analysis is a test that allows users to deeply analyse their model's performance.

Failure Mode Analysis enables you to define rules on metrics by setting thresholds. Based on rules, get a sorted list of all clusters that are breaching the threshold and clusters that are within the threshold. RagaAI will create clusters using the embeddings of the Dataset. Clusters that underperform and breaches the threshold will be highlighted on the Issue Stats screen.

Execute Test

The following code snippet is designed to perform a Failure Mode Analysis test on a specified dataset within the RagaAI environment.

First, you define the rules that will be used to evaluate the model's performance. These rules are based on metrics such as F1 Score, Precision, and Recall (Refer Metric Glossary).

rules = SDRules()
rules.add(metric="Accuracy", label=["All"], metric_threshold=0.80)

cls_default = clustering(test_session=test_session,
                         dataset_name=dataset_name,
                         method="k-means",
                         embedding_col="ImageEmbedding",
                         level="image",
                         args={"numOfClusters": 9}, 
                         force=True)

edge_case_detection = fma_structured_data(test_session=test_session,
                                            dataset_name = "default_pred",
                                            test_name = "FMA SD",
                                            type = "fma",
                                            output_type="structured_data",
                                            embedding= "ImageEmbedding",
                                            model = "model_pred",
                                            gt = "repay_fail",
                                            rules = rules,
                                            clustering = cls_default)
test_session.add(edge_case_detection)

test_session.run()

• SDRules(): Initialises the rules for the Failure Mode Analysis test.

  • rules.add(): Adds a new rule with specific parameters:

    • metric: The performance metric to evaluate (e.g., F1Score, Precision, Accuracy, Recall).

    • label: Specifies the label(s) these metrics apply to. "ALL" means all labels.

    • metric_threshold: The minimum acceptable value for the metric.

Next, you define the clustering parameters to group similar failure modes.

• clustering(): Sets the clustering method for grouping failure modes.

  • method: The clustering technique used, "k-means" in this case.

  • embedding_col: The column containing embedding vectors to be used for clustering.

  • level: The level at which clustering is applied, "image" here indicates that each image is treated as an individual data point.

  • args: Additional arguments for the clustering method, such as the number of clusters.

  • force: set as "True" if you wish to rerun clustering on the existing dataset rather than using cached clustering results.

  Note: The clustering parameters are not required if the test is being run on metadata.

• fma_structured_data(): Configures the failure mode analysis test with the following parameters:

  • test_session: Defines the test session created by the user with the project name, access key, secret key and the host token.

  • dataset_name: Specifies the dataset to be used by the user for the test.

  • test_name: Identifies with the test the user is running.

  • type: Specifies the level (embedding for cluster level and metadata for metadata level) on which the test is being run.

  • output_type: Contains the usecase the test is being run on. For example: Object Detection, Semantic Segmentation and many more.

  • embedding: Specified the name of the embedding column in the dataset.

  • model: Contains the model inferences which are supposed to be evaluated.

  • gt: The groundtruth against which the model inferences are going to be compared across the diverse scenarios.

  • rules: The previously defined rules for failure mode analysis test

  • clustering: Specifies the clustering method defined by the user earlier. This is only required incase failure mode analysis is being run at a cluster level.

• test_session.add(): Registers the failure mode analysis test with the test session.

• test_session.run(): Starts the execution of all the tests added to the session, including the FMA.

Following this guide, you've successfully set up and initiated a Failure Mode Analysis test on the RagaAI Testing Platform.

Analysing Test Results

Clustering (Embedding View)

1. Navigate to Embedding View: Find the interactive visualisation interface in RagaAI.

2. Visualise Clusters: Utilise the tool to visualise complex clustering, uncovering hidden patterns and structures in the data.

3. Select Data Points: Use the lasso tool to select specific data points of interest and analyse their results.

Visualising Data

Data Filtering: Use this feature to focus on specific subsets of your dataset that meet certain conditions, helping to extract meaningful patterns and trends.

Understanding Clustering and Threshold Breaches

• Cluster Analysis: RagaAI creates clusters using the embeddings of the dataset. These clusters group similar data points together.

• Identifying Underperforming Clusters: Clusters that underperform (breaching the threshold) will be highlighted on the Issue Stats screen.

Navigating and Interpreting Results

• Directly Look at Problematic Clusters: Users can quickly identify clusters responsible for underperformance and assess their impact on the overall model.

• In-Depth Analysis: Dive deeper into specific clusters or data points to understand the root causes of underperformance.

• Feature Level Analysis: Understand the divergence between the datapoints selected in the embeddings and the overall datapoints of the datasets.

Data Analysis

1. Switch to Analysis Tab: To get a detailed performance report, go to the Analysis tab.

2. View Performance Metrics: Examine metrics like label-wise performance and temporal graphs.

3. Confusion Matrix: The class-based confusion matrix in Failure Mode Analysis provides a detailed breakdown of performance for each class.

Practical Tips

• Set Realistic Thresholds: Choose thresholds that reflect the expected performance of your model.

• Leverage Visual Tools: Make full use of RagaAI’s visualisation capabilities to gain insights that might not be apparent from raw data alone.

• Delve Deeper Into Features: Do a deeper root case analysis at a feature level and gain insights on model performance over datapoints across different features.

• Data Level View: Get results at individual datapoint level

By following these steps, users can efficiently leverage the Failure Mode Analysis test to gain a comprehensive understanding of their model's performance, identify key areas for improvement, and make data-driven decisions to enhance model accuracy and reliability.