Fact Entropy

Measure uncertainty in LLM facts. Use entropy scoring to identify unreliable answers and boost accuracy.

Objective:

Fact Entropy measures the degree of uncertainty or randomness in the factual consistency of a model’s output. It quantifies how confidently a model presents facts, with higher entropy indicating more variability or less certainty in factual assertions. This metric is critical for assessing factual accuracy in generative models, particularly in tasks where consistent, reliable factual grounding is required.

Required Columns in Dataset:

Generated Facts, Expected Facts

Interpretation:

• High Fact Entropy: Implies greater uncertainty or inconsistency in the model's factual assertions, suggesting unreliable or volatile outputs.

• Low Fact Entropy: Reflects more confident and consistent fact generation, indicating stronger factual accuracy and stability in the output.

Execution via UI:

Execution via SDK:

metrics=[
    {"name": "Fact Entropy", "config": {"model": "gpt-4o-mini", "provider": "openai"}, "column_name": "your-text", "schema_mapping": schema_mapping}
]