Consistency
Objective: The test is intended to check the consistency of the model i.e if the model can generate similar answers on multiple runs w.r.t the prompt and response provided
Required Parameters: Prompt, Response
By default, 5 responses are generated
Interpretation: A higher score signifies model is more consistent in its response to the given prompt.
# Consistency Test
prompt = "Who is Issac Newton?"
consistent_response = "Sir Isaac Newton FRS was an English polymath active as a mathematician, physicist, astronomer, alchemist, theologian, and author who was described in his time as a natural philosopher"
inconsistent_response = "Issac Newton was an English poet of the second generation of Romantic poets, along with Lord Byron and Percy Bysshe Shelly."
evaluator.add_test(
test_names=["consistency_test"],
data={"prompt": prompt, "response": consistent_response},
arguments={"threshold": 0.5},
).add_test(
test_names=["consistency_test"],
data={"prompt": prompt, "response": inconsistent_response},
arguments={"threshold": 0.5},
).run()
evaluator.print_results()
Last updated
Was this helpful?