QAG Score

Apply QAG Score to measure summary reliability. Detect factual gaps and improve question-answer consistency.

Objective:

QAG Score evaluates the quality of a generated summary or response by formulating questions based on the generated output and then checking if answers to those questions match the original text. This method leverages question generation and answering tasks to verify factual alignment and information consistency, particularly useful in summarization and knowledge extraction tasks. It focuses on the factual relevance of the generated content and its ability to respond accurately to pertinent questions.

Required Columns in Dataset:

LLM Summary, Original Document

Interpretation:

• High QAG Score: Suggests that the generated output answers key questions accurately, meaning it is factually aligned with the original source.

• Low QAG Score: Implies that the generated content may fail to answer or misrepresent important information, reflecting factual inconsistencies.

Execution via UI:

Execution via SDK:

metrics=[
    {"name": "QAGScore", "config": {"model": "gpt-4o-mini", "provider": "openai"}, "column_name": "your-text", "schema_mapping": schema_mapping}
]