Upload dataset via CSV

Once your project is created, you can upload datasets to it for evaluation.

Via UI:

  1. Open your Project from the Project list.

  2. It will take you to the Dataset tab.

  3. From the options to create new dataset, select "Upload via CSV" method

  4. Click on the upload area and browse/drag and drop your local CSV file. Ensure the file size does not exceed 1GB.

  5. Enter a suitable name and description [optional] for your dataset.

  6. Click Next to proceed.

Next, you will be directed to map your dataset schema with Catalyst's inbuilt schema, so that your column headings don't require editing.

Here is a list of Catalyst's inbuilt schema elements (definitions are for reference purposes and may vary slightly based on your use case):

Via SDK:

This guide provides a step-by-step explanation on how to use the RagaAI Python SDK to upload data to your project. The example demonstrates how to manage datasets and upload a CSV file into the platform. The following sections will cover initialisation, listing existing datasets, mapping schema, and uploading the CSV data.

1. Prerequisites

  • Ensure you have the RagaAI Python SDK installed. If not, you can install it using:

    pip install ragaai-catalyst
  • You need secret key, access key and project name, which you can get by navigating to settings/authenticate on UI.

2. Importing Required Modules

Import the Dataset module from the ragaai_catalyst library to handle the dataset operations.

from ragaai_catalyst import Dataset
import pandas as pd

3. Initialise Dataset Management

Initialise the dataset manager for a specific project. This will allow you to interact with the datasets in that project.

# Initialize Dataset management for a specific project
dataset_manager = Dataset(project_name="demo_project")

Replace "demo_project" with your actual project name.

4. List Existing Datasets

You can list all the existing datasets within your project to check what data is already available.

# List existing datasets
datasets = dataset_manager.list_datasets()
print("Existing Datasets:", datasets)

This prints a list of existing datasets available in your project.

5. Get the Schema Elements

Retrieve the supported schema elements from the project. This will help you understand how to map your CSV columns to the dataset schema.

# Get the schema elements
schemaElements = dataset_manager.get_csv_schema()['data']['schemaElements']
print('Supported column names: ', schemaElements)

This step returns the available schema elements that can be used for mapping your CSV columns.

6. Create the Schema Mapping

Create a dictionary to map your CSV column names to the schema elements supported by RagaAI. For example:

pythonCopy code #Create the schema mapping accordingly
schema_mapping = {'sql_context': 'context', 'sql_prompt': 'prompt'}

In this case, the column 'sql_context' in the CSV is mapped to 'context' in the dataset, and 'sql_prompt' is mapped to 'prompt'.

7. Upload the Dataset from CSV

Finally, use the create_from_csv function to upload the CSV data into the platform. Specify the CSV path, dataset name, and the schema mapping.

# Create a dataset from CSV
dataset_manager.create_from_csv(
    csv_path='/content/synthetic_text_to_sql_gpt_4o_mini.csv',
    dataset_name='csv_upload31',
    schema_mapping=schema_mapping
)

Replace the csv_path and dataset_name with your CSV file path and desired dataset name, respectively.

8. Verifying the Upload

After uploading, you can verify the upload by listing the datasets again or checking the project dashboard.

# List datasets to verify the upload
datasets = dataset_manager.list_datasets()
print("Updated Datasets:", datasets)

9. Verifying the Upload

Navigate to Dataset tab inside your project to explore your dataset and run evals

Last updated