Tool Selection Accuracy

Assess if the tool selections in your agentic pipeline is optimal

Tool Selection Accuracy

Objective

This metric evaluates the appropriateness and effectiveness of an AI system's tool selection when responding to various tasks. It assesses how well the AI selects the most suitable tools from a set of available options, considering factors such as relevance, efficiency, and potential impact on the task's success.

Interpretation

A score closer to 1.0 indicates that the AI consistently selects the most appropriate and effective tools for the given tasks, leading to successful outcomes. A score closer to 0.0 suggests that the AI frequently selects suboptimal or inappropriate tools, which could negatively impact task performance.

Syntax

# Tool Selection Accuracy
response = experiment_manager.execute(
    metrics=[
        {"name": "tool_selection_accuracy", "config": {"model": "gpt-4o-mini"}}
    ]
)