Hallucination

Objective: This metric evaluates the overlap of facts between the Response and the Context. It penalises any fabricated, incorrect or contradictory facts mentioned in the Response that are not found in the Context.

Required Parameters: Response, Context

Interpretation: A higher score indicates the model response was hallucinated.

Code Execution:

experiment_manager = Experiment(project_name="project_name",
                                experiment_name="experiment_name",
                                dataset_name="dataset_name")

# Hallucination Test
response =  experiment_manager.add_metrics(
    metrics=[
        {"name":"Hallucination", "config": {"reason": True, "model": "gpt-4o-mini", "batch_size" : 5, "provider": "OpenAI"}}
    ]
)

Refer Executing tests page to learn about Metric Configurations

Example:

• Prompt: What is the capital of Brazil?

• Context: Brazil is the largest country in South America, known for its diverse culture and the Amazon rainforest. Its official language is Portuguese and its capital is Brasília.

• Response: The capital of Brazil is Rio de Janeiro, which is famous for its Copacabana beach, Christ the Redeemer statue, and vibrant carnival celebrations.

• Metric Output: {‘score’: 1, ‘reason’: ‘The capital of Brazil is Brasília, not Rio de Janeiro’}