Consistency

Objective: The test is intended to check the consistency of the model i.e if the model can generate similar answers on multiple runs w.r.t the prompt and response provided

Required Parameters: Prompt, Response

• By default, 5 responses are generated

Interpretation: A higher score signifies model is more consistent in its response to the given prompt.

# Consistency Test
prompt = "Who is Issac Newton?"
consistent_response = "Sir Isaac Newton FRS was an English polymath active as a mathematician, physicist, astronomer, alchemist, theologian, and author who was described in his time as a natural philosopher"
inconsistent_response = "Issac Newton was an English poet of the second generation of Romantic poets, along with Lord Byron and Percy Bysshe Shelly."

evaluator.add_test(
    test_names=["consistency_test"],
    data={"prompt": prompt, "response": consistent_response},
    arguments={"threshold": 0.5},
).add_test(
    test_names=["consistency_test"],
    data={"prompt": prompt, "response": inconsistent_response},
    arguments={"threshold": 0.5},
).run()

evaluator.print_results()