LogoLogo
Slack CommunityCatalyst Login
  • Welcome
  • RagaAI Catalyst
    • User Quickstart
    • Concepts
      • Configure Your API Keys
      • Supported LLMs
        • OpenAI
        • Gemini
        • Azure
        • AWS Bedrock
        • ANTHROPIC
      • Catalyst Access/Secret Keys
      • Enable Custom Gateway
      • Uploading Data
        • Create new project
        • RAG Datset
        • Chat Dataset
          • Prompt Format
        • Logging traces (LlamaIndex, Langchain)
        • Trace Masking Functions
        • Trace Level Metadata
        • Correlating Traces with External IDs
        • Add Dataset
      • Running RagaAI Evals
        • Executing Evaluations
        • Compare Datasets
      • Analysis
      • Embeddings
    • RagaAI Metric Library
      • RAG Metrics
        • Hallucination
        • Faithfulness
        • Response Correctness
        • Response Completeness
        • False Refusal
        • Context Relevancy
        • Context Precision
        • Context Recall
        • PII Detection
        • Toxicity
      • Chat Metrics
        • Agent Quality
        • Instruction Adherence
        • User Chat Quality
      • Text-to-SQL
        • SQL Response Correctness
        • SQL Prompt Ambiguity
        • SQL Context Ambiguity
        • SQL Context Sufficiency
        • SQL Prompt Injection
      • Text Summarization
        • Summary Consistency
        • Summary Relevance
        • Summary Fluency
        • Summary Coherence
        • SummaC
        • QAG Score
        • ROUGE
        • BLEU
        • METEOR
        • BERTScore
      • Information Extraction
        • MINEA
        • Subjective Question Correction
        • Precision@K
        • Chunk Relevance
        • Entity Co-occurrence
        • Fact Entropy
      • Code Generation
        • Functional Correctness
        • ChrF
        • Ruby
        • CodeBLEU
        • Robust Pass@k
        • Robust Drop@k
        • Pass-Ratio@n
      • Marketing Content Evaluation
        • Engagement Score
        • Misattribution
        • Readability
        • Topic Coverage
        • Fabrication
      • Learning Management System
        • Topic Coverage
        • Topic Redundancy
        • Question Redundancy
        • Answer Correctness
        • Source Citability
        • Difficulty Level
      • Additional Metrics
        • Guardrails
          • Anonymize
          • Deanonymize
          • Ban Competitors
          • Ban Substrings
          • Ban Topics
          • Code
          • Invisible Text
          • Language
          • Secret
          • Sentiment
          • Factual Consistency
          • Language Same
          • No Refusal
          • Reading Time
          • Sensitive
          • URL Reachability
          • JSON Verify
        • Vulnerability Scanner
          • Bullying
          • Deadnaming
          • SexualContent
          • Sexualisation
          • SlurUsage
          • Profanity
          • QuackMedicine
          • DAN 11
          • DAN 10
          • DAN 9
          • DAN 8
          • DAN 7
          • DAN 6_2
          • DAN 6_0
          • DUDE
          • STAN
          • DAN_JailBreak
          • AntiDAN
          • ChatGPT_Developer_Mode_v2
          • ChatGPT_Developer_Mode_RANTI
          • ChatGPT_Image_Markdown
          • Ablation_Dan_11_0
          • Anthropomorphisation
      • Guardrails
        • Competitor Check
        • Gibberish Check
        • PII
        • Regex Check
        • Response Evaluator
        • Toxicity
        • Unusual Prompt
        • Ban List
        • Detect Drug
        • Detect Redundancy
        • Detect Secrets
        • Financial Tone Check
        • Has Url
        • HTML Sanitisation
        • Live URL
        • Logic Check
        • Politeness Check
        • Profanity Check
        • Quote Price
        • Restrict Topics
        • SQL Predicates Guard
        • Valid CSV
        • Valid JSON
        • Valid Python
        • Valid Range
        • Valid SQL
        • Valid URL
        • Cosine Similarity
        • Honesty Detection
        • Toxicity Hate Speech
    • Prompt Playground
      • Concepts
      • Single-Prompt Playground
      • Multiple Prompt Playground
      • Run Evaluations
      • Using Prompt Slugs with Python SDK
      • Create with AI using Prompt Wizard
      • Prompt Diff View
    • Synthetic Data Generation
    • Gateway
      • Quickstart
    • Guardrails
      • Quickstart
      • Python SDK
    • RagaAI Whitepapers
      • RagaAI RLEF (RAG LLM Evaluation Framework)
    • Agentic Testing
      • Quickstart
      • Concepts
        • Tracing
          • Langgraph (Agentic Tracing)
          • RagaAI Catalyst Tracing Guide for Azure OpenAI Users
        • Dynamic Tracing
        • Application Workflow
      • Create New Dataset
      • Metrics
        • Hallucination
        • Toxicity
        • Honesty
        • Cosine Similarity
      • Compare Traces
      • Compare Experiments
      • Add metrics locally
    • Custom Metric
    • Auto Prompt Optimization
    • Human Feedback & Annotations
      • Thumbs Up/Down
      • Add Metric Corrections
      • Corrections as Few-Shot Examples
      • Tagging
    • On-Premise Deployment
      • Enterprise Deployment Guide for AWS
      • Enterprise Deployment Guide for Azure
      • Evaluation Deployment Guide
        • Evaluation Maintenance Guide
    • Fine Tuning (OpenAI)
    • Integration
    • SDK Release Notes
      • ragaai-catalyst 2.1.7
  • RagaAI Prism
    • Quickstart
    • Sandbox Guide
      • Object Detection
      • LLM Summarization
      • Semantic Segmentation
      • Tabular Data
      • Super Resolution
      • OCR
      • Image Classification
      • Event Detection
    • Test Inventory
      • Object Detection
        • Failure Mode Analysis
        • Model Comparison Test
        • Drift Detection
        • Outlier Detection
        • Data Leakage Test
        • Labelling Quality Test
        • Scenario Imbalance
        • Class Imbalance
        • Active Learning
        • Image Property Drift Detection
      • Large Language Model (LLM)
        • Failure Mode Analysis
      • Semantic Segmentation
        • Failure Mode Analysis
        • Labelling Quality Test
        • Active Learning
        • Drift Detection
        • Class Imbalance
        • Scenario Imbalance
        • Data Leakage Test
        • Outlier Detection
        • Label Drift
        • Semantic Similarity
        • Near Duplicates Detection
        • Cluster Imbalance Test
        • Image Property Drift Detection
        • Spatio-Temporal Drift Detection
        • Spatio-Temporal Failure Mode Analysis
      • Tabular Data
        • Failure Mode Analysis
      • Instance Segmentation
        • Failure Mode Analysis
        • Labelling Quality Test
        • Drift Detection
        • Class Imbalance
        • Scenario Imbalance
        • Label Drift
        • Data Leakage Test
        • Outlier Detection
        • Active Learning
        • Near Duplicates Detection
      • Super Resolution
        • Semantic Similarity
        • Active Learning
        • Near Duplicates Detection
        • Outlier Detection
      • OCR
        • Missing Value Test
        • Outlier Detection
      • Image Classification
        • Failure Mode Analysis
        • Labelling Quality Test
        • Class Imbalance
        • Drift Detection
        • Near Duplicates Test
        • Data Leakage Test
        • Outlier Detection
        • Active Learning
        • Image Property Drift Detection
      • Event Detection
        • Failure Mode Analysis
        • A/B Test
    • Metric Glossary
    • Upload custom model
    • Event Detection
      • Upload Model
      • Generate Inference
      • Run tests
    • On-Premise Deployment
      • Enterprise Deployment Guide for AWS
      • Enterprise Deployment Guide for Azure
  • Support
Powered by GitBook
On this page
  • Execute Test:
  • Interpreting the Results

Was this helpful?

  1. RagaAI Prism
  2. Test Inventory
  3. Semantic Segmentation

Class Imbalance

The Class Imbalance Test is designed to assess the distribution of classes within a dataset, particularly in the context of machine learning tasks like object detection.

Execute Test:

The code executes the Class Imbalance Test using two different metrics, namely Jensen-Shannon Divergence and Chi-Squared Test, to evaluate the distribution of classes within a training dataset.

rules = ClassImbalanceRules()
rules.add(metric="js_divergence", ideal_distribution="uniform", metric_threshold=0.10, label="ALL")
rules.add(metric="chi_squared_test", ideal_distribution="uniform", metric_threshold=0.10, label="ALL")

run_name = "Class Imbalance v1"
print(run_name)
dataset_name = "training_dataset"

test_session = TestSession(
            project_name="Instance Segmentation",run_name=run_name,access_key="8Sxdx2ELb70quckrkklZ",secret_key="UeIWErIbh8sAFVxpLqtfJA0dMW7QsaiApuRmOYz8",host="https://backend.platform.raga.ai")
        
distribution_test = class_imbalance_test(test_session=test_session,
                                         dataset_name=dataset_name,
                                         test_name=run_name,
                                         type="class_imbalance",
                                         output_type="semantic_segmentation",
                                         annotation_column_name="AnnotationsV1",
                                         rules=rules)

test_session.add(distribution_test)
test_session.run()
  1. Initialize Class Imbalance Rules:

    • Use the ClassImbalanceRules() function to initialize the rules for the test.

  2. Add Rules:

    • Use the rules.add() function to add specific rules with the following parameters:

      • metric: The metric used to evaluate class distribution (e.g., js_divergence, chi_squared_test).

      • ideal_distribution: The ideal distribution assumption for the metric (e.g., "uniform").

      • metric_threshold: The threshold for the metric, indicating when the class distribution is considered imbalanced.

      • label: Specifies the label(s) to which the rule applies. Use "ALL" to apply to all labels.

  3. Configure Test Run:

    • Define the test run configuration, including the project name, test name, dataset name, and session credentials.

  4. Execute Class Imbalance Test:

    • Use the class_imbalance_test() function to execute the test with the following parameters:

      • test_session: The session object managing tests.

      • dataset_name: Name of the dataset to be tested.

      • test_name: Name of the test run.

      • type: Type of test, which should be set to "class_imbalance".

      • output_type: Type of output expected from the model.

      • annotation_column_name: Name of the column containing annotations.

      • rules: Predefined rules for the test.

  5. Add Test to Session:

    • Use the test_session.add() function to register the test with the test session.

  6. Run Test:

    • Use the test_session.run() function to start the execution of all tests added to the session, including the Class Imbalance Test.

By following these steps, you can effectively assess the class distribution within your dataset using the Class Imbalance Test.

Interpreting the Results

The Class Imbalance Test provides insights into the distribution of classes within the datasets under evaluation. The results are presented in three segments:

Bar Chart Comparison

  • The bar chart compares the class counts between the training dataset and the dataset under evaluation.

  • This visualisation highlights any discrepancies in class distribution between the two datasets.

Use the bar chart to compare class counts and identify any significant disparities between the training dataset and the dataset under evaluation.

Data Grid View

  • The data grid view presents annotations with images sorted by class imbalance scores.

  • Analyse annotations with images sorted by class imbalance scores in the data grid view.

  • Identify images with notable class imbalances for further investigation.

Image View

  • Interactive annotation rendering and original image viewing capabilities enhance the understanding of class distribution.

  • Utilise the interactive annotation rendering and original image viewing functionalities to examine specific instances of class imbalance.

By leveraging these features, you can effectively evaluate and address class imbalances within your datasets, ensuring the integrity and relevance of your machine learning models over time.

PreviousDrift DetectionNextScenario Imbalance

Last updated 1 year ago

Was this helpful?