Overall
Objective: The test is intended to check overall model performance in generating text based on a given concept set
Required Parameters:
Response: The sentence generated by the model using the words in concept_set.
Expected_Response: The sentence generated by the model you want to compare or the ground truth.
Concept_set: A list of words in their root form, each tagged with its corresponding part of speech (e.g., "_V" for verb, "_N" for noun, "_A" for adjective).
Optional Parameters:
model (str, optional): The name of the language model to be used for task of evaluating responses. Defaults to "gpt-3.5-turbo".
temperature (float,optional): This parameter allows you to adjust the randomness of the response generated by the specified model.
max_tokens (int,optional): This parameter allows you to specify the maximum length of the generated response.
Interpretation: 1 indicates model response is better than ground truth/other model response.
"response" : "The quick brown fox jumps over the lazy dog.",
"expected_response" : "The quick brown fox beats the dog.",
"concept_set" : ["quick_A", "brown_A", "fox_N", "jump_V", "lazy_A", "dog_N"]
Here, winner_test will return score 1 as response is better than expected_response. Pos_test and Cover_test will return 1 as all concepts are covered and all PoS used in response are correct. Hence final output, 1*1*1 = 1
# Example with higher score.
# Make sure you pass all words in concept_set in their root form.
evaluator.add_test(
test_names=["overall_test"],
data={
"response" : "The quick brown fox jumps over the lazy dog.",
"expected_response" : "The quick brown fox beats the dog.",
"concept_set" : ["quick_A", "brown_A", "fox_N", "jump_V", "lazy_A", "dog_N"]
},
arguments={"model": "gpt-4"},
).run()
evaluator.print_results()
Last updated
Was this helpful?