Context Relevancy RAG Metric

Evaluate how relevant the retrieved context is to the user’s query and the LLM’s answer, ensuring the model uses appropriate and useful information.

The Context Relevancy metric evaluates the quality of the retriever used in the RAG pipeline. This metric is vital to ensure that the the documents retrieved by the retriever is relevant for answering the prompt and the retriever mechanism in the RAG pipeline is working as expected.

Required Parameters: Prompt, Context

Interpretation:

Lower metric score indicates one of these:

• The retrieval mechanism is not working poorly.

• The Knowledge Base doesn't have sufficient data to supply documents to the prompt.

Metric Execution via UI:

Code Execution

metrics=[
    {"name": "Context Relevancy", "config": {"model": "gpt-4o-mini", "provider": "openai"}, "column_name": "your-column-identifier", "schema_mapping": schema_mapping}
]

The "schema_mapping" variable needs to be defined first and is a pre-requisite for evaluation runs. Learn how to set this variable here.