Coherence

Objective: The test is intended to check the coherence of the model response i.e if the model is able to generate ideas and arguments in a logical manner

Parameters:

data:

• prompt (str): The prompt for the response.

• response (str): The response to be evaluated.

• context (str, optional): The context for the response (default is None).

arguments:

• strictness (int, optional): The number of times response is evaluated (default is 1).

• model (str, optional): The model to be used for evaluation (default is "gpt-4").

Interpretation: Passed result signifies model is more coherent in its response to the given prompt. Failed result signifies model is less coherent in its response, giving unorganized or incorrect response.

# Add tests with custom data
evaluator.add_test(
    test_names=["coherence_test"],
    data={
        "prompt": ["Can you explain the process of photosynthesis in detail?"],
        "response": ["Photosynthesis is a complex biochemical process in which plants convert light energy into chemical energy, utilizing water, carbon dioxide, chlorophyll, and enzymes."]
    },
    arguments={"model": "gpt-4", "threshold": 0.6},
).run()

evaluator.print_results()