Tracing

Tracing is a core capability of RagaAI Catalyst designed to help you debug, evaluate, and analyse the behavior of agentic applications by capturing detailed experiment-level traces. These traces document the inputs, outputs, decisions, and intermediate steps taken by your application, enabling a deep understanding of its performance and identifying areas for improvement.

Note: this works on Python :3.10,3.11,3.12

---

What is Tracing for Agentic Applications?

In agentic applications, tracing refers to the process of monitoring and recording the sequence of operations executed by the system. This includes:

• Interactions between agents, tools, and external systems.

• Decision-making processes of agents across variable inputs.

• Key performance metrics and error patterns.

Tracing provides insights into complex workflows, enabling you to:

• Debug issues effectively.

• Validate application logic.

• Optimize the overall performance.

---

How to Capture Traces Using RagaAI Catalyst

Follow the steps below to start tracing your agentic application using RagaAI Catalyst.

---

Quickstart Guide

Use this Sample Colab Notebook to get started quickly.

---

Step-by-Step Instructions

1. Install RagaAI Catalyst

   !pip install ragaai-catalyst

2. Import Required Modules

   from ragaai_catalyst import RagaAICatalyst
   from ragaai_catalyst import Tracer

3. Initialize RagaAI Catalyst

   catalyst = RagaAICatalyst(
       access_key="access_key",
       secret_key="secret_key",
   )

4. Create a Project and Dataset Define your project and dataset for structured trace management.

   project_name = "Project_Name" # create a project on the UI
   tracer_dataset_name = "dataset_name"

5. Create a Tracer Object

   tracer = Tracer(
       project_name=project_name,
       dataset_name=tracer_dataset_name,
       tracer_type="agentic_tracing",
   )
   init_tracing(catalyst=catalyst, tracer=tracer)

6. Wrap Functions with RagaAI Catalyst Decorators Use decorators to trace specific functionalities in your application.

   from ragaai_catalyst import trace_llm, trace_tool, trace_agent
   @trace_llm(name="agent_name")
   async def my_llm_function():
    # Your LLM call here
    pass
   
   @trace_tool("my_tool")
   def my_tool_function():
    # Your tool logic here
   pass
   
   @trace_agent("my_agent")
   def my_agent_function():
    # Your agent logic here
    pass

Additional Resources

For detailed explanations of each decorator and their use cases, refer to this guide.

Note: when you change anything inside your code to run the trace, it will create a new code version

Note: this works on Python :3.10,3.11,3.12

How to add Ground Truth?

To include ground truth data in a trace, follow these steps:

1. Define an input parameter: In the traced element, create an input with the parameter name gt and assign it a default value.

2. Pass the ground truth: When invoking the function during the agent run, include the ground truth data as the argument for the gt parameter.

By doing this, the ground truth data will automatically be recorded in the trace.

from ragaai_catalyst import trace_llm, trace_tool, trace_agent
@trace_llm(name="agent_name")
async def my_llm_function():
 # Your LLM call here
 pass

@trace_tool("my_tool")
def my_tool_function():
 # Your tool logic here
pass

@trace_agent("my_agent")
def my_agent_function():
 # Your agent logic here
 pass


with tracer:
    my_llm("Hello, world!", gt='some gt value')
    my_tool_function(gt='some gt value')
    my_agent_function(gt='some gt value')

RagaAI Tracing Supported Attributes :

1. User Input

2. Agent Output

3. Network Calls

4. File Read/ Write

5. Tool Calls

6. LLM Calls

7. Custom Calls

Defining Trace and Span

Understanding the core concepts of trace and span is essential for effectively using RagaAI Catalyst’s tracing features.

---

What is a Trace?

A trace represents the complete lifecycle of an operation or experiment in your agentic application. It includes:

• The series of actions and decisions made by agents.

• Interactions between components like LLMs, tools, and agents.

• Metadata capturing contextual information such as timestamps, inputs, outputs, and errors.

Think of a trace as the "big picture" view of how your application executes an operation from start to finish.

Example

If you are testing an application to handle a financial query, the trace might include:

1. The user's input.

2. The decision-making process of the financial agent.

3. API calls to external services.

4. The generated response and any intermediate computations.

---

What is a Span?

A span is a single unit of work within a trace. It represents a specific operation, such as:

• A call to an external API.

• A task performed by a tool or an agent.

• An execution of an LLM prompt.

Each span contains:

• Operation Name: The task being performed (e.g., "API Call", "LLM Response").

• Start and End Time: Duration of the operation.

• Attributes: Additional metadata specific to the operation, such as parameters or error codes.

Spans are the building blocks of a trace, providing granular details about each step within the larger operation.

---

Relation Between Trace and Span

• A trace is composed of multiple spans.

• Each span contributes to the trace by capturing detailed information about a specific task or operation.

---