Concepts
Concepts in RagaAI Catalyst
This section introduces the key concepts and functionalities in RagaAI Catalyst, including tracing, decorators, dataset structure, metrics, and code versioning.
1. Tracing
Tracing is the process of capturing and analyzing interactions within an Agentic application. It records essential details such as input, output, and metadata during execution. Tracing is crucial for debugging, optimization, and evaluation.
2. Decorators
Decorators in RagaAI Catalyst enhance functions by adding tracing capabilities. They enable logging, instrumentation, or tracing without modifying the original code.
Types of Decorators and Their Usage
trace_llm:Purpose: Traces interactions with Large Language Models (LLMs).
Traces:
Input prompts.
Generated responses.
Metadata.
Exceptions during processing.
trace_tool:Purpose: Traces the usage of tools in the framework.
Traces:
Input data.
Tool-generated results.
Exceptions during processing.
trace_agent:Purpose: Traces actions of agents (e.g., virtual assistants).
Traces:
Actions performed by the agent.
Outcomes of actions.
State changes.
Exceptions during processing.
Note: Only agents can have children, which may include LLMs, tools, or other agents.
3. Trace Components
LLM
Captures LLM inputs, outputs, and token usage.
Enables evaluation on spans such as LLM, tool, agent, or the entire trace.
Tracing Metadata and Components
Metadata includes contextual details such as token usage, error status, model information, and execution context.
Each trace component can capture unique attributes to aid in detailed evaluations.
4. Dataset
Dataset Column Names
TraceID: Unique identifier for the trace.
Timestamp: Execution timestamp.
Trace URL: URL to access trace details.
Feedback: User-provided feedback on trace.
Response: Captured LLM response.
Metadata: Key contextual details logged.
Metric Score: Evaluation score.
Completion Token: Tokens generated by the model.
Prompt Token: Tokens used in the prompt.
Cost: Resource usage cost.
5. Metrics
Metric Evaluation Levels
LLM: Evaluate spans with type
llm. Runs on the last occurrence of the selected span.Tool: Evaluate spans with type
tool. Runs on the last occurrence of the selected span.Agent: Evaluate spans with type
conversation. Runs on the last occurrence of the selected span.Trace: Evaluate the entire
trace.json.Conversation: Evaluate all input-output interactions in an Agentic application.
Metric Parameter Mappings
LLM:
data.input: Input prompt or query.data.output: LLM response.data.function_call: Called function name.data.function_parameters: Function parameters.info.model: Model information.info.token_usage: Token usage statistics.data.gt: Ground truth.
Tool:
data.input: Input to the tool.data.output: Tool output.data.error: Tool error details (if any).networkcalls.status_code: HTTP status code.networkcalls.url: Request URL.networkcalls.method: HTTP method.data.gt: Ground truth.
Agent:
data.input: Input prompt or query.data.output: Agent response.data.function_call: Called function name.data.function_parameters: Function parameters.info.model: Model information.info.token_usage: Token usage statistics.data.gt: Ground truth.
6 . Custom Metrics
RagaAI Catalyst supports custom metrics for advanced evaluations.
Refer to the documentation for setup and configuration of custom metrics.
7. Code Versioning
RagaAI Catalyst automatically versions the code whenever changes are made to trace instrumentation.
Users can view the code diffs to track modifications and compare versions.
This feature ensures traceability and consistency in trace evaluations.
By understanding these concepts, users can effectively leverage RagaAI Catalyst's features to analyze, evaluate, and optimize their Agentic applications.
Last updated