Concepts

RagaAI Catalyst offers a cutting-edge evaluation and observability suite for GenAI applications. The complete user workflow of the application rests on a few basic concepts, which are highlighted below:

Projects:

A Catalyst Project is the central hub where all datasets, experiments, evaluations, analyses, and comparisons are organised and stored. Projects can be likened to enterprise use-cases, meaning all evaluations and experiments related to different use-cases can have their separate space within the application.

Datasets:

Catalyst Datasets can be thought of like spreadsheets. These allow for a row-and-column view of your uploaded/traced data. Typically, these are structured with rows containing individual prompts, and each column containing related data, for e.g, context, response, metrics computed, metadata, etc. As with spreadsheets, users can add rows (more data) and columns (new evaluations) to the dataset dynamically.

Traces:

Traces refer to the real-time logs of a deployed GenAI application. RagaAI Catalyst allows SDK-based tracing of LLM inferences in real-time via SDK commands (for more details, refer this page).

Metrics/Evals:

Metrics are quantitative measures used to assess various aspects of application performance. They can include Hallucination, Response Completeness, Context Precision, Toxicity, etc. Metrics help you understand how well your applications are performing and where improvements may be needed.