Faithfulness RAG Metric

Measure how well an LLM’s answer stays true to the source material. Ensure factual accuracy in RAG by avoiding distortions or errors.

Objective: This metric determines the proportion of facts in the response that originate from the context information. The generated answer is considered faithful if all the claims made can be inferred from the provided context.

Required Parameters: Prompt, Response, Context

Interpretation:

• Lower faithfulness score indicates the model is not able to focus on the correct context document.

• Lower faithfulness score indicates the model is hallucinating and generating information not present in the context documents.

• Lower faithfulness score indicates the Knowledge Base has contradicting information regarding the topic referred to in the prompt.

Code Execution:

metrics=[
    {"name": "Faithfulness", "config": {"model": "gpt-4o-mini", "provider": "openai"}, "column_name": "your-column-identifier", "schema_mapping": schema_mapping}
]

The "schema_mapping" variable needs to be defined first and is a pre-requisite for evaluation runs. Learn how to set this variable here.

Example:

• Prompt: Who discovered penicillin?

• Context: Penicillin is one of the most important discoveries in medical science, marking the beginning of the antibiotic era. It was discovered in 1928 by Alexander Fleming, a Scottish bacteriologist.

• Response: Alexander Dumas discovered penicillin.

• Metric Output: {‘score’: 0, ‘reason’: ‘As per context penicillin was discovered by Alexander Fleming’}