Factual Consistency
This scanner checks if the provided content disagrees with a specific statement or question, this test ensures language model results are accurate and logical.
Parameters:
data:
prompt
(str): Prompt given to the modelresponse
(str): Reponse given by the model, which will be checked for factual consistency
arguments:
threshold
(float, optional): Threshold to determine if the response is factually correct. Default is 0.5.use_onnx
(bool, optional): Whether to use onnx model to check for factual consistency.
Interpretation:
Higher score represents more logical and factually correct model response. The test passes if the model response is correct.
Code Example:
prompt = "Angela Merkel is a politician in Germany and leader of the CDU."
response = "This text is about economics"
evaluator.add_test(
test_names=["factual_consistency_guardrail"],
data={'prompt':prompt,
'response': response,
},
arguments={'model': 'gpt-4', 'threshold': 1.0},
).run()
evaluator.print_results()
Last updated
Was this helpful?