Concepts

RagaAI Catalyst evaluation and observability stack. It is helpful to learn the specific concepts of RagaAI in order to leverage RagaAI successfully. In this section, the concepts of RagaAI are featured. These include Project, Experiments, Usage Monitors, Metrics,Logs and datasets , each of which is important to better understand how RagaAI Catalyst works.

Project:

The project is the central hub where all related experiments, data, and configurations are organized. It serves as a comprehensive workspace, allowing you to manage and coordinate various aspects of your LLM-powered applications in a cohesive manner.

Experiments:

Experiments are structured tests designed to evaluate the performance and behavior of LLM-powered applications. They allow you to test different configurations, prompts, and model variations, pipelines to determine the most effective approach for your specific use case. Experiments can be run in development, CI/CD, or production environments.

Usage Monitors:

Usage monitors are tools for tracking the real-time usage and performance of your applications. They provide insights into how your applications are being used, no. of metrics metrics run on different models and total evals calculated in a dataset. This real-time data is crucial for ensuring the reliability and efficiency of your applications and keeping track of usage summary across projects.

Metrics/Evals:

Metrics are quantitative measures used to assess various aspects of application performance. They can include response time, accuracy, throughput, and other relevant performance indicators. Metrics help you understand how well your applications are performing and where improvements may be needed.

Logs:

Logs provide a detailed view of your project's activity in real-time. In the Logs tab, you can monitor traces, which are detailed records of the interactions and processes within your project.

Datasets:

Datasets are collections of data used for training, testing, and validating LLM-powered applications. Proper dataset management is crucial for ensuring the quality and relevance of the data used in your experiments and evaluations.