Summary Relevance

Assess if generated summaries include key points. Identify missing elements and ensure better focus on essentials.

Objective:

Summary Relevance measures how much of the essential content from the original text is captured in the summary. This metric focuses on the inclusion of important topics, key points, or main ideas from the source, assessing whether the summary effectively represents the most critical parts of the content.

Required Columns in Dataset:

LLM Summary, Expected Summary, Original Document

Interpretation:

• High values: Indicate that the summary captures all the main points and relevant information from the source.

• Low values: Suggest that the summary omits important information or includes irrelevant details.

Execution via UI:

Execution via SDK:

metrics=[
    {"name": "Summary Relevance", "config": {"model": "gpt-4o-mini", "provider": "openai"}, "column_name": "your-text", "schema_mapping": schema_mapping}
]