Factual Consistency

Ensure AI outputs stay factually consistent. Detect contradictions and improve trustworthiness.

This scanner checks if the provided content disagrees with a specific statement or question, this test ensures language model results are accurate and logical.

Parameters:

data:

• prompt (str): Prompt given to the model

• response (str): Reponse given by the model, which will be checked for factual consistency

arguments:

• threshold (float, optional): Threshold to determine if the response is factually correct. Default is 0.5.

• use_onnx (bool, optional): Whether to use onnx model to check for factual consistency.

Interpretation:

Higher score represents more logical and factually correct model response. The test passes if the model response is correct.

Code Example:

prompt = "Angela Merkel is a politician in Germany and leader of the CDU."
response = "This text is about economics"

evaluator.add_test(
    test_names=["factual_consistency_guardrail"],
    data={'prompt':prompt,
          'response': response,
    },
    arguments={'model': 'gpt-4', 'threshold': 1.0},
).run()

evaluator.print_results()