User Quickstart

This guide will help you get started with RagaAI Catalyst using any sample dataset, straight from your Python environment.

The above video demonstrates all the steps required to perform your first evaluation on the Catalyst software. All commands shown in the video have been documented below.

Click here to open the companion notebook in Google Colab

1. Sign Up and Authentication

1.1 Sign Up

• Start by signing up for an account @ https://catalyst.raga.ai/

1.2 Install Python Package

pip install ragaai_catalyst -U -q

1.3 Authenticate Using Keys

• Authenticate using your access key and secret key:

from ragaai_catalyst import RagaAICatalyst, Experiment, Tracer, Dataset

catalyst = RagaAICatalyst(
    access_key="ACCESS_KEY",
    secret_key="SECRET_KEY",
    base_url="https://catalyst.raga.ai/api"
)

• You can find Catalyst Access & Secret Keys in the UI by navigating to Settings -> Authenticate and use the "Generate New Key" button to create a fresh pair:

Finding RagaAI Access & Secret Keys

• A valid API key will be required to run evaluations on the public sandbox. Your API keys can be added on the UI by navigating to Settings -> API Keys as follows:

Adding API Keys on Catalyst UI

2. Create New Project

• Create a new project for testing your LLM Application:

new_project = catalyst.create_project(project_name="PROJECT-NAME", description="PROJECT-DESCRIPTION")

• User can also create the project via UI by clicking the "Create New Project" button on the platform homepage:

• List your projects:

projects = catalyst.list_projects(num_projects=2)
print(f"Projects: {projects}")

• Note: 'num_projects' is optional and is used to list the latest 'n' projects (here, 2). Passing list_projects() without any arguments will list all the projects available.

3. Upload Dataset

RagaAI Catalyst enables users to ingest data in two broad ways: real-time tracing and static uploads of data (CSV, Pandas, etc.).

3.1 Upload Dataset via CSV

Once your project is created, you can upload datasets to it. Here's how you can upload a dataset via a CSV file:

Create New Dataset in a Project

1. Select your newly created project from the project list.

2. Navigate to the Dataset tab within the project.

3. Click on the Create New Dataset button.

4. In the upload window, select the Upload CSV tab.

5. Click on the upload area and browse/drag and drop your local CSV file. Ensure the file size does not exceed 1GB.

6. Enter a suitable name for your dataset.

7. Click Next to proceed.

Next, you will be directed to map your dataset schema with Catalyst's inbuilt schema, so that your column headings don't require editing:

Set Dataset Schema

Here is a list of Catalyst's inbuilt schema elements (definitions are for reference purposes and may vary slightly based on your use case):

Schema Element	Definition
traceId	Unique ID associated with a trace
metadata	Any additional data not falling into a defined bucket
cost	Expense associated with generating a particular inference
expected_context	Context documents expected to be retrieved for a query
latency	Time taken for an inference to be returned
system_prompt	Predefined instruction provided to an LLM to shape its behaviour during interactions
traceUri	Unique identifier used to trace and log the sequence of operations during an LLM inference process
pipeline	Sequence of processes or stages that an input passes through before producing an output in LLM systems
response	Output generated by an LLM after processing a given prompt or query
context	Surrounding information or history provided to an LLM to inform and influence its responses
sanitized_response	A version of the LLM's output that has been modified to remove sensitive, inappropriate, or undesirable content
prompt	Input or query provided to an LLM that triggers the generation of a response
expected_response	Anticipated or ideal output that an LLM should produce in response to a given prompt
timestamp	Specific date and time at which an LLM action, such as an inference or a response, occurs

3.2 Upload Dataset Via Tracing

• Set up a tracer to monitor your application:

from ragaai_catalyst import Tracer

tracer = Tracer(
    project_name="PROJECT-NAME",
    metadata={"key1": "value1", "key2": "value2"},
    tracer_type="langchain",
    pipeline={
        "llm_model": "gpt-4o-mini",
        "vector_store": "faiss",
        "embed_model": "text-embedding-ada-002",
    },
)

tracer.start()

• Once the tracer has successfully started, execute your RAG Langchain code. You can find a sample RAG code in this Colab project. Your OpenAI API key will be required to run this example.

• Once traces have been logged using your code, stop the tracer and check its status:

tracer.stop()
tracer.get_upload_status()

• If successful, you can view the logged dataset by navigating to Project_Name -> Logs.

• In order to run metrics on traced logs, Catalyst requires users to create a sub-dataset as follows:

dataset = Dataset(project_name="your_project_name")

# Sample filter to create sub-dataset from comprehensive list of traced logs; change values if needed
filter_list = [{
        "name":"llm_model",
        "values":["gpt-4o-mini"]
    },
    {
        "name":"prompt_length",
        "lte": 10000,
        "gte": 0
    }]
    
dataset_name ='your_dataset_name'
dataset.create_dataset(dataset_name, filter_list)

dataset.list_datasets()

• You can also do this on the Catalyst UI by navigating to Project_Name -> Logs and clicking on the "Create Dataset" button.

• In the dialog box that pops up, users can create a sub-dataset from the traced logs by filtering on metadata, human feedback, prompt length, and time as shown:

Creating Sub-Dataset from Traced Logs

4. Create Experiments and Run Metrics

• Set up an experiment and add metrics for evaluation:

from ragaai_catalyst import Experiment

experiment_manager = Experiment(
    project_name="your_project_name",
    experiment_name="Your Experiment Name",
    experiment_description="Experiment description",
    dataset_name="your_dataset_name"
)

response = experiment_manager.add_metrics(
    metrics=[
        {"name": "Hallucination", "config": {"model": "gpt-4o-mini", "reason": True, "provider": "OpenAI"}},
        {"name": "Hallucination", "config": {"model": "azure/azure-gpt-4o-mini", "reason": True, "provider": "Azure"}}
    ]
)

print("Metric Response:", response)

• Check out the list of Supported metrics.

• Check out Test Execution documentation to learn about Metric Configuration

• Once the experiment is executed, get the status of ypur experiment execution by Navigating to Job Status from the navigation bar on the User Interface. Alternatively, run the following command to print the status on your notebook:

status = experiment_manager.get_status()
print("Experiment Status:", status)

• You can view the experiment on the platform. Alternatively, run the following command to print the results on your notebook:

results = experiment_manager.get_results()
print("Experiment Results:")
results

Experiment View:

Refer docs to learn more about analysing new results [Learn More]