Failure Mode Analysis

Failure Mode Analysis is a test that allows users to deeply analyse the pipeline performance.

Failure Mode Analysis enables you to define rules on metrics by setting thresholds. Based on rules, get a sorted list of all clusters that are breaching the threshold and clusters that are within the threshold. RagaAI will create clusters using the embeddings of the Dataset. Clusters that underperform and breaches the threshold will be highlighted on the Issue Stats screen.

Execute Test

The following code snippet is designed to perform a Failure Mode Analysis test on a specified dataset within the RagaAI environment.

First, you define the rules that will be used to evaluate the model's performance. These rules are based on metrics such as F1 Score, Precision, and Recall (Refer Metric Glossary).

rules = FMARules()
rules.add(metric="Precision", conf_threshold=0.2, metric_threshold=0.5, iou_threshold=0.5, frame_overlap_threshold=0.5, label="ALL")
rules.add(metric="Recall", conf_threshold=0.2, metric_threshold=0.5, iou_threshold=0.5, frame_overlap_threshold=0.5, label="ALL")
rules.add(metric="F1Score", conf_threshold=0.2, metric_threshold=0.5, iou_threshold=0.5, frame_overlap_threshold=0.5, label="ALL")

edge_case_detection = failure_mode_analysis(test_session=test_session,
                                            dataset_name = "bdd_video_test_2",
                                            test_name = f"FMAEventMD-{datetime.datetime.now().strftime('%Y%m%d%H%M%S')}",
                                            gt="GT",
                                            model="Red_Light_V2",
                                            object_detection_gt="GT",
                                            object_detection_model="Red_Light_V2",
                                            rules = rules,
                                            output_type="event_detection",
                                            type="metadata",
                                            aggregation_level=["weather","time_of_day"])

test_session.add(edge_case_detection)
test_session.run()

• FMARules(): Initialises the rules for the FMA test.

  • rules.add(): Adds a new rule with specific parameters:

    • metric: The performance metric to evaluate (e.g., F1Score, Precision).

    • metric_threshold: The minimum acceptable value for the metric.

    • conf_threshold: The minimum acceptable confidence value for a detection to be true.

    • iou_threshold: The measure of overlap between predicted and ground truth bounding boxes in object detection, with a higher IoU indicating greater accuracy.

    • frame_overlap_threshold: This sets the required degree of overlap between consecutive frames' bounding boxes, influencing the sensitivity of object tracking in applications.

    • label: Specifies the label(s) these metrics apply to. "ALL" means all labels.

• failure_mode_analysis(): Configures the failure mode analysis test with the following parameters:

  • test_session: Defines the test session created by the user with the project name, access key, secret key and the host token.

  • dataset_name: Specifies the dataset to be used by the user for the test.

  • test_name: Identifies with the test the user is running.

  • gt: The groundtruth pipeline against which the pipeline detections are going to be compared across the diverse scenarios.

  • model: Contains the pipeline detections which are supposed to be evaluated.

  • object_detection_gt: This is the intermediate groundtruth object detection bounding box rendering for the videos.

  • object_detection_model: These are the intermediate model object detection bounding box rendering for the videos.

  • rules: The previously defined rules for failure mode analysis test.

  • output_type: Contains the usecase the test is being run on. For example: objecy_detection, event_detection and many more.

  • type: Specifies the level (embedding for cluster level and metadata for metadata level) on which the test is being run.

  • aggregation_level: Specifies the scenarios across which the test results are generated. This is only required incase failure mode analysis is being run at a metadata level.

• test_session.add(): Registers the failure mode analysis test with the test session.

• test_session.run(): Starts the execution of all the tests added to the session, including the FMA.

Following this guide, you've successfully set up and initiated a Failure Mode Analysis test on the RagaAI Testing Platform.

Analysing Test Results

Navigating and Interpreting Results

• Directly Look at Problematic Scenarios: Users can quickly identify scenarios responsible for underperformance and assess their impact on the overall model.

• In-Depth Analysis: Dive deeper into specific scenarios or data points to understand the root causes of underperformance.

Data Analysis

1. Switch to Analysis Tab: To get a detailed performance report, go to the Analysis tab.

2. View Performance Metrics: Examine metrics like label-wise performance and temporal graphs.

3. Confusion Matrix: The class-based confusion matrix in Failure Mode Analysis provides a detailed breakdown of performance for each class.

Practical Tips

• Set Realistic Thresholds: Choose thresholds that reflect the expected performance of your model.

• Leverage Visual Tools: Make full use of RagaAI’s visualisation capabilities to gain insights that might not be apparent from raw data alone.

By following these steps, users can efficiently leverage the Failure Mode Analysis test to gain a comprehensive understanding of their model's performance, identify key areas for improvement, and make data-driven decisions to enhance model accuracy and reliability.