Summary Coherence

Check if summaries are logically connected and structured. Spot gaps in flow and enhance overall quality.

Objective:

Summary Coherence measures the logical flow and structural integrity of the summary. It examines whether ideas are presented in a logically connected manner, ensuring that the summary follows a clear progression of thoughts. This is especially important in multi-sentence summaries, where disjointed or fragmented ideas can disrupt understanding.

Required Columns in Dataset:

LLM Summary, Prompt

Interpretation:

• High values: Indicate that the summary presents information in a well-organized, logically connected sequence.

• Low values: Suggest that the summary is disorganized or has disconnected thoughts, making it harder to follow.

Execution via UI:

Execution via SDK:

metrics=[
    {"name": "Summary Coherence", "config": {"model": "gpt-4o-mini", "provider": "openai"}, "column_name": "your-text", "schema_mapping": schema_mapping}
]