Label Drift

Detect shifts in labels over time. Ensure annotation consistency in evolving datasets.

Execute Test:

The code executes the Class Imbalance Test using two different metrics, namely Jensen-Shannon Divergence and Chi-Squared Test, to evaluate the distribution of classes within a training dataset.

rules = LDTRules()
rules.add(metric="js_divergence", label=["ALL"],  metric_threshold=0.10)
rules.add(metric="chi_squared_test", label=["ALL"], metric_threshold=0.10)

run_name = f"Label Drift v1"

test_session = TestSession(
            project_name="Instance Segmentation",run_name=run_name,access_key="8Sxdx2ELb70quckrkklZ",secret_key="UeIWErIbh8sAFVxpLqtfJA0dMW7QsaiApuRmOYz8",host="https://backend.platform.raga.ai")
      
ref_dataset_name = "training_dataset"
eval_dataset_name = "validation_dataset"
distribution_test = label_drift_test(test_session=test_session,
                                     referenceDataset=ref_dataset_name,
                                     evalDataset=eval_dataset_name,
                                     test_name=run_name,
                                     type="label_drift",
                                     output_type="semantic_segmentation",
                                     gt="GT",
                                     rules=rules)


test_session.add(distribution_test)
test_session.run()

1. Initialise Label Drift Rules:

   • Use the LDTRules() function to initialize the rules for the test.

2. Add Rules:

   • Use the rules.add() function to add specific rules with the following parameters:

     • metric: The metric used to evaluate label distribution (e.g., js_divergence, chi_squared_test).

     • metric_threshold: The threshold for the metric, indicating when the label distribution is considered drifted.

     • label: Specifies the label(s) to which the rule applies. Use ["ALL"] to apply to all labels.

3. Configure Test Run:

   • Define the test run configuration, including the project name, test name, and session credentials.

4. Execute Label Drift Test:

   • Use the label_drift_test() function to execute the test with the following parameters:

     • test_session: The session object managing tests.

     • referenceDataset: Name of the reference dataset.

     • evalDataset: Name of the evaluation dataset.

     • test_name: Name of the test run.

     • type: Type of test, which should be set to "label_drift".

     • output_type: Type of output expected from the model.

     • gt: Ground truth data column name.

     • rules: Predefined rules for the test.

5. Add Test to Session:

   • Use the test_session.add() function to register the test with the test session.

6. Run Test:

   • Use the test_session.run() function to start the execution of all tests added to the session, including the Label Drift Test.

By following these steps, you can effectively detect label drift between datasets using the Label Drift Test.

Interpreting Test Results for Label Drift

The Label Drift Test results provide insights into the distribution of labels between a reference dataset and an evaluation dataset. The results are presented through visualisations and data grids, facilitating the identification of label drift.

Bar Chart Comparison

• The bar chart compares the distribution of labels between the reference dataset and the evaluation dataset.

• This visualisation highlights any discrepancies or shifts in label distribution between the two datasets.

Compare label distributions to identify overall trends in label drift between datasets.

Data Grid Views

• Two side-by-side data grid views are provided to visualise the reference dataset and evaluation dataset.

• Each data grid displays annotations with images sorted by label drift scores.

• Users can compare the label distributions and identify specific images with notable label drift.

Image View

• The Image View allows users to delve into detailed analyses of label drift for individual images.

• Interactive annotation rendering and original image viewing functionalities are available to facilitate examination.

By leveraging these features, users can effectively identify and address label drift between datasets, ensuring the integrity and relevance of their machine learning models over time.