Tracing

Note: this works on python>=3.11

Tracing is a core capability of RagaAI Catalyst designed to help you debug, evaluate, and analyse the behavior of agentic applications by capturing detailed experiment-level traces. These traces document the inputs, outputs, decisions, and intermediate steps taken by your application, enabling a deep understanding of its performance and identifying areas for improvement.

---

What is Tracing for Agentic Applications?

In agentic applications, tracing refers to the process of monitoring and recording the sequence of operations executed by the system. This includes:

• Interactions between agents, tools, and external systems.

• Decision-making processes of agents across variable inputs.

• Key performance metrics and error patterns.

Tracing provides insights into complex workflows, enabling you to:

• Debug issues effectively.

• Validate application logic.

• Optimize the overall performance.

---

How to Capture Traces Using RagaAI Catalyst

Follow the steps below to start tracing your agentic application using RagaAI Catalyst.

---

Quickstart Guide

Use this Sample Colab Notebook to get started quickly.

---

Step-by-Step Instructions

1. Install RagaAI Catalyst

   !pip install ragaai-catalyst

2. Import Required Modules

   from ragaai_catalyst import RagaAICatalyst
   from ragaai_catalyst import Tracer

3. Initialize RagaAI Catalyst

   catalyst = RagaAICatalyst(
       access_key="access_key",
       secret_key="secret_key",
   )

4. Create a Project and Dataset Define your project and dataset for structured trace management.

   project_name = "Project_Name" # create a project on the UI
   tracer_dataset_name = "dataset_name"

5. Create a Tracer Object

   tracer = Tracer(
       project_name=project_name,
       dataset_name=tracer_dataset_name,
       metadata={"key1": "value1", "key2": "value2"},
       tracer_type="agentic_tracing",
       pipeline={"key": "value"},
   )

6. Start the Tracer

   tracer.start()

7. Wrap Functions with RagaAI Catalyst Decorators Use decorators to trace specific functionalities in your application.

   @tracer.trace_llm("my_llm_call")
   async def my_llm_function():
       # Your LLM call here
       pass

   @tracer.trace_tool("my_tool")
   def my_tool_function():
       # Your tool logic here
       pass

   @tracer.trace_agent("my_agent")
   def my_agent_function():
       # Your agent logic here
       pass

Additional Resources

For detailed explanations of each decorator and their use cases, refer to this guide.

1. Stop the Tracer After tracing, ensure you stop the tracer to finalise the session.

tracer.stop()

Note: when you change anything inside your code to run the trace, it will create a new code version

Note: this works on python>=3.11

How to add Ground Truth?

To include ground truth data in a trace, follow these steps:

1. Define an input parameter: In the traced element, create an input with the parameter name gt and assign it a default value.

2. Pass the ground truth: When invoking the function during the agent run, include the ground truth data as the argument for the gt parameter.

By doing this, the ground truth data will automatically be recorded in the trace.

@tracer.trace_llm("my_llm")
async def my_llm(prompt, gt=None):
    # Your LLM logic here
    pass

@tracer.trace_tool("my_tool")
async def my_tool_function(gt=None):
    # Your tool logic here
    pass

@tracer.trace_agent("my_agent")
async def my_agent_function(gt=None):
    # Your agent logic here
    pass


with tracer:
    my_llm("Hello, world!", gt='some gt value')
    my_tool_function(gt='some gt value')
    my_agent_function(gt='some gt value')

Defining Trace and Span

Understanding the core concepts of trace and span is essential for effectively using RagaAI Catalyst’s tracing features.

---

What is a Trace?

A trace represents the complete lifecycle of an operation or experiment in your agentic application. It includes:

• The series of actions and decisions made by agents.

• Interactions between components like LLMs, tools, and agents.

• Metadata capturing contextual information such as timestamps, inputs, outputs, and errors.

Think of a trace as the "big picture" view of how your application executes an operation from start to finish.

Example

If you are testing an application to handle a financial query, the trace might include:

1. The user's input.

2. The decision-making process of the financial agent.

3. API calls to external services.

4. The generated response and any intermediate computations.

---

What is a Span?

A span is a single unit of work within a trace. It represents a specific operation, such as:

• A call to an external API.

• A task performed by a tool or an agent.

• An execution of an LLM prompt.

Each span contains:

• Operation Name: The task being performed (e.g., "API Call", "LLM Response").

• Start and End Time: Duration of the operation.

• Attributes: Additional metadata specific to the operation, such as parameters or error codes.

Spans are the building blocks of a trace, providing granular details about each step within the larger operation.

---

Relation Between Trace and Span

• A trace is composed of multiple spans.

• Each span contributes to the trace by capturing detailed information about a specific task or operation.

---