Concepts

Concepts in RagaAI Catalyst

This section introduces the key concepts and functionalities in RagaAI Catalyst, including tracing, decorators, dataset structure, metrics, and code versioning.

---

1. Tracing

Tracing is the process of capturing and analyzing interactions within an Agentic application. It records essential details such as input, output, and metadata during execution. Tracing is crucial for debugging, optimization, and evaluation.

---

2. Decorators

Decorators in RagaAI Catalyst enhance functions by adding tracing capabilities. They enable logging, instrumentation, or tracing without modifying the original code.

Types of Decorators and Their Usage

1. trace_llm:

   • Purpose: Traces interactions with Large Language Models (LLMs).

   • Traces:

     • Input prompts.

     • Generated responses.

     • Metadata.

     • Exceptions during processing.

2. trace_tool:

   • Purpose: Traces the usage of tools in the framework.

   • Traces:

     • Input data.

     • Tool-generated results.

     • Exceptions during processing.

3. trace_agent:

   • Purpose: Traces actions of agents (e.g., virtual assistants).

   • Traces:

     • Actions performed by the agent.

     • Outcomes of actions.

     • State changes.

     • Exceptions during processing.

     • Note: Only agents can have children, which may include LLMs, tools, or other agents.

---

3. Trace Components

LLM

• Captures LLM inputs, outputs, and token usage.

• Enables evaluation on spans such as LLM, tool, agent, or the entire trace.

Tracing Metadata and Components

• Metadata includes contextual details such as token usage, error status, model information, and execution context.

• Each trace component can capture unique attributes to aid in detailed evaluations.

---

4. Dataset

Dataset Column Names

• TraceID: Unique identifier for the trace.

• Timestamp: Execution timestamp.

• Trace URL: URL to access trace details.

• Feedback: User-provided feedback on trace.

• Response: Captured LLM response.

• Metadata: Key contextual details logged.

• Metric Score: Evaluation score.

• Completion Token: Tokens generated by the model.

• Prompt Token: Tokens used in the prompt.

• Cost: Resource usage cost.

---

5. Metrics

Metric Evaluation Levels

1. LLM: Evaluate spans with type llm. Runs on the last occurrence of the selected span.

2. Tool: Evaluate spans with type tool. Runs on the last occurrence of the selected span.

3. Agent: Evaluate spans with type conversation. Runs on the last occurrence of the selected span.

4. Trace: Evaluate the entire trace.json.

5. Conversation: Evaluate all input-output interactions in an Agentic application.

---

Metric Parameter Mappings

1. LLM:

   • data.input: Input prompt or query.

   • data.output: LLM response.

   • data.function_call: Called function name.

   • data.function_parameters: Function parameters.

   • info.model: Model information.

   • info.token_usage: Token usage statistics.

   • data.gt: Ground truth.

2. Tool:

   • data.input: Input to the tool.

   • data.output: Tool output.

   • data.error: Tool error details (if any).

   • networkcalls.status_code: HTTP status code.

   • networkcalls.url: Request URL.

   • networkcalls.method: HTTP method.

   • data.gt: Ground truth.

3. Agent:

   • data.input: Input prompt or query.

   • data.output: Agent response.

   • data.function_call: Called function name.

   • data.function_parameters: Function parameters.

   • info.model: Model information.

   • info.token_usage: Token usage statistics.

   • data.gt: Ground truth.

---

6 . Custom Metrics

• RagaAI Catalyst supports custom metrics for advanced evaluations.

• Refer to the documentation for setup and configuration of custom metrics.

---

7. Code Versioning

• RagaAI Catalyst automatically versions the code whenever changes are made to trace instrumentation.

• Users can view the code diffs to track modifications and compare versions.

• This feature ensures traceability and consistency in trace evaluations.

By understanding these concepts, users can effectively leverage RagaAI Catalyst's features to analyze, evaluate, and optimize their Agentic applications.