Faithfulness

Objective: This metric determines the proportion of facts in the response that originate from the context information. The generated answer is considered faithful if all the claims made can be inferred from the provided context.

Required Parameters: Response, Context

Interpretation:

• Lower faithfulness score indicates the model is not able to focus on the correct context document.

• Lower faithfulness score indicates the model is hallucinating and generating information not present in the context documents.

• Lower faithfulness score indicates the Knowledge Base has contradicting information regarding the topic referred to in the prompt.

Code Execution:

experiment_manager = Experiment(project_name="project_name",
                                experiment_name="experiment_name",
                                dataset_name="dataset_name")

response = experiment_manager.add_metrics(
    metrics=[
        {"name":"Faithfulness", "config": {"reason": True, "model": "gpt-4o-mini", "batch_size" : 5, "provider": "OpenAI"}}
    ]
)

Refer Executing tests page to learn about Metric Configurations

Example:

• Prompt: Who discovered penicillin?

• Context: Penicillin is one of the most important discoveries in medical science, marking the beginning of the antibiotic era. It was discovered in 1928 by Alexander Fleming, a Scottish bacteriologist.

• Response: Alexander Dumas discovered penicillin.

• Metric Output: {‘score’: 0, ‘reason’: ‘As per context penicillin was discovered by Alexander Fleming’}