Add metrics locally

Extend Catalyst by adding local metrics. Tailor AI evaluation to your organization’s specific needs.

RagaAI allows users to define and upload custom metrics locally, catering to specific requirements that involve specialized libraries or unique metric definitions to an user. This feature is particularly useful for metrics that are not universally accessible or are highly specific to the user's needs. Users can upload these metrics at both the Trace Level and Span Level, and these metrics will be added during trace execution.

---

How to Upload Metrics Locally?

1. Span Level Metric

Span-level metrics are added to specific spans within a trace. This is done by defining the tracer decorator and using the add_metrics() method.

Example Code:

@tracer.trace_tool(name="span_name")
def function():
    tracer.span('span_name').add_metrics(
        name='metric_name', 
        score=score, 
        reasoning=reasoning, 
        cost=cost, 
        latency=latency
    )
    pass

Key Parameters:

• name: The name of the metric. This will appear as a column in the dataset.

• score: The metric score for the specific input.

• reasoning: (Optional) Explanation or reasoning for the metric score.

• cost: (Optional) The cost associated with the metric.

• latency: (Optional) The latency value for the metric.

Points to Note:

• Metric names are case-insensitive.

• No two spans can have the same metric name. If duplicate names are used:

  • The SDK will treat the first occurrence as the default name.

  • Subsequent metrics with the same name will have a suffix in the format {metric_name_(n-1)}.

---

2. Trace Level Metric

Trace-level metrics apply to the entire trace and can be defined at a higher level than spans. The process to add these metrics follows similar principles but applies them globally across the trace.

User can add the metric anywhere between tracer.start() and tracer.stop() using the method:

    tracer.add_metrics(
        name='metric_name', 
        score=score, 
        reasoning=reasoning, #optional
        cost=cost, #optional
        latency=latency #optional
    )

---