False Refusal RAG Metric

Identify the instances where LLM improperly refuses to answer a question despite having sufficient information to do so. Understand how to fine-tune overly cautious or misaligned refusal behaviors.

Objective: This metric identifies instances where an LLM incorrectly declines to provide a response, despite the available context containing sufficient information to respond accurately.

Required Parameters: Prompt, Response, Context

Interpretation: 1 corresponds to a response being falsely refused.

Metric Execution via UI:

Code Execution:

metrics=[
    {"name": "False Refusal", "config": {"model": "gpt-4o-mini", "provider": "openai"}, "column_name": "your-column-identifier", "schema_mapping": schema_mapping}
]

The "schema_mapping" variable needs to be defined first and is a pre-requisite for evaluation runs. Learn how to set this variable here.

Example:

• Prompt: Can you summarise the book 'Pride and Prejudice'?

• Context: Pride and Prejudice is a novel by Jane Austen that follows the character development of Elizabeth Bennet, the dynamic protagonist of the book. Set in the early 19th century, the novel deals with themes of love, reputation, and class distinctions.

• Response: Sorry, I cannot provide a summary of the book.

• Metric Output: {‘score’: 1, ‘reason’: ‘information provided should enable the LLM to generate a summary’}