RagaAI Catalyst Tracing Guide for Azure OpenAI Users

Learn how to set up tracing for Azure OpenAI models in RagaAI. Ensure detailed monitoring and evaluation.

This guide helps Azure OpenAI users implement RagaAI Catalyst's tracing capabilities to monitor, debug, and analyze their agentic applications.

Prerequisites

• Python 3.10 or higher

• Azure OpenAI API access

• RagaAI Catalyst access credentials

Quick Setup Guide

1. Install RagaAI Catalyst

pip install ragaai-catalyst

2. Set Up Tracing with Azure OpenAI

from ragaai_catalyst import RagaAICatalyst, Tracer, init_tracing
from llama_index.llms.azure_openai import AzureOpenAI

# Initialize RagaAI Catalyst
catalyst = RagaAICatalyst(
    access_key="your_access_key",
    secret_key="your_secret_key",
)

# Create a tracer
tracer = Tracer(
    project_name="YourProjectName",  # Create this project in the RagaAI UI first
    dataset_name="YourDatasetName",
    tracer_type="agentic_tracing",  # Use appropriate tracer type ("agentic/llamaindex" for llamaindex examples)
)

# Initialize tracing
init_tracing(catalyst=catalyst, tracer=tracer)

# Create your Azure OpenAI instance
azure_llm = AzureOpenAI(
    azure_endpoint="https://your-resource-name.openai.azure.com/",
    model="your-deployment-name",  # e.g., "gpt-4o-mini"
    api_key="your-azure-openai-api-key",
    api_version="2024-05-01-preview",  # Use appropriate API version
    engine="your-deployment-name"  # Same as your deployment name
)

# Optional: Set model cost information for billing analysis
tracer.set_model_cost({
    "model_name": "gpt-4o-mini",
    "input_cost_per_million_token": 10_000_000,
    "output_cost_per_million_token": 20_000_000
})

Tracing Methods

Using Decorators: Use decorators to trace specific functions in your application.

from ragaai_catalyst import trace_llm, trace_tool, trace_agent

# Trace LLM calls
@trace_llm(name="azure_completion")
async def get_completion(prompt):
    response = await azure_llm.achat([ChatMessage(role="user", content=prompt)])
    return response.message.content

# Trace tool usage
@trace_tool(name="calculator_tool")
def calculate(x, y, operation):
    if operation == "add":
        return x + y
    elif operation == "multiply":
        return x * y
    # Add more operations as needed

# Trace agent behavior
@trace_agent(name="financial_advisor")
def advisor_agent(query, gt=None):  # Note: gt parameter for ground truth
    # Agent implementation
    return response

Supported Trace Attributes

RagaAI Catalyst captures various attributes in your traces:

• User Input

• Agent Output

• Network Calls

• File Read/Write Operations

• Tool Calls

• LLM Calls (including Azure OpenAI)

• Custom Calls

Accessing Trace Results

View your traces in the RagaAI Catalyst UI:

1. Log in to your RagaAI account

2. Navigate to your project

3. Select the dataset you specified in your code

4. View detailed traces showing each span (operation) within your application