Quickstart

1. Create a Project

• Create a new project using the button. Each represents a unique AI application / use-case. Select any such project to perform a deep dive.

• Explore a Run: Every Run represents the execution of the test suite on a set of datasets and model. Select a Run to view all tests performed.

2. Explore Test Results

Access Test Results: On the summary page, click view iterations and view results to explore test results for a specific dataset and model.

Note: An in-depth companion piece to every project and every test run is available on the sandbox guide.

3. Execute your own Test!

Step 1: Start the Test Run

• Navigate to the RagaAI dashboard and open your project.

• Click on the 'Initiate Run' button located towards the right side of the dashboard.

Step 2: Access Keys Popup

• A popup will appear displaying the Access Key and Secret Key.

• These keys are crucial as they link the Google Colab file to your RagaAI account and the specific project you are working on.

Step 3: Launch the Colab Notebook

• Click on the 'Try It' button within the popup.

• This action will open a new tab with a Google Colab notebook preloaded with the necessary code cells for executing the test.

Step 4: Enter Your Access Keys

• In the Colab notebook, locate the cell titled 'Put your ACCESS KEY & SECRET KEY'.

• Replace the placeholder text with your actual Access Key and Secret Key provided in the popup.

Step 5: Install the dependencies

• The first cell typically contains the command to install the RagaAI testing platform library using pip. Run this cell to install the required library in your Colab environment.

• After setting your Access Key and Secret Key, proceed to the cell labeled 'Import All raga lib from raga module' and run it to import the necessary modules.

Step 6: Upload a Dataset

test_ds = CocoDataset(
            test_session=test_session,
            name="test-data",
            type="object_detection",  # Specify the type of task (e.g., object detection, classification)
            json_file_path="path/to/annotation_file.json",  # Path to the JSON annotation file
            image_folder_path='path/to/image_folder',  # Path to the folder containing the images
            s3_bucket_name="product-raga",  # Name of the S3 bucket where the dataset is stored
            s3_key="coco-images",  # S3 key pointing to the dataset images
            aws_raga_access_key=AWS_ACCESS_KEY_ID,  # Your AWS access key
            aws_raga_secret_key=AWS_SECRET_ACCESS_KEY,  # Your AWS secret key
            bucket_region= bucket_region  # The region of your S3 bucket
        )

Step 7: Generate Embeddings

In some cases, the tests you want to run may require the generation of embeddings for your dataset. Embeddings are numerical representations of your data that are crucial for certain types of AI models and evaluations:

dataset = Dataset(name=dataset_name, test_session=test_session, init=False)
dataset.generate_embeddings(model_name="dinov2-model",  # Model used for generating embeddings
                            model_version="v1",  # Version of the model
                            embed_col_name="dinov2-model:v1",  # Column name for storing embeddings
                            output_type="object_detection",  # Type of task
                            input_func=input_function,  # Custom function for input processing
                            output_func=output_function)  # Custom function for output processing

you can add your existing embeddings as well, by using:

test_ds.add_embeddings(userDataFrame=df, model="dna", col_name="embedding_column_name")

Step 8: Generate Inferences

If your dataset does not have model inferences, then you can use ragaAI to generate inferences [we also support using of custom models]

dataset = Dataset(name=dataset_name, test_session=test_session, init=False)
dataset.generate_inferences(model_name="rcnn-inference-model",
                            model_version="v1",
                            inference_col_name="rcnn-inference-model:v2",
                            output_type="object_detection",
                            input_func=input_function,
                            output_func=output_function)

User can add their own inferences as well, by using:

test_ds.add_inference(inferences_file_path="/content/updated_dataset_with_embeddings.csv",
                      format="YOLOv5",
                      model_name="anything",
                      customer_column_name="annotations")

Step 9: Setup a run

• Set up a Run: This cell contains the code to initialise a test session with parameters like project name, run name, and your keys. Running this cell will start the test run using the onboarded dataset.

run_name = "your_run_name"
test_session = TestSession(project_name=project_name, run_name = run_name, access_key=ACCESS_KEY, secret_key=SECRET_KEY, host=HOST)

Step 10: Execute test

• Run Tests: These cells contains the code to execute the different available tests. Alter the parameters like threshold, metric, etc and execute the cell

Explore the Test Inventory

Visualising Results

Once the test is complete, you can easily visualise the results. Navigate back to our platform's user interface (UI). Inside the Runs tab, you'll find a new run containing results for the tests executed by you.