Cover

Objective: The test is intended to check what proportion of the words provided was covered by the model in the response

Required Parameters:

• Response: The sentence generated by the model.

• Concept_set: A list of words in their root form, each tagged with its corresponding part of speech (e.g., "_V" for verb, "_N" for noun, "_A" for adjective).

Interpretation: A higher score indicates that the model's response contains all concepts as specified in the concept set.

• Example with Higher Score:

  • Concept set: ["cat_N", "sleep_V", "windowsill_N"]

  • Response: "A cat is sleeping on the windowsill." (All concepts are used in model response.)

• Example with Lower Score:

  • Concept set: ["sky_N", "sun_N", "windowsill_N"]

  • Response: "The sky is blue and the sun is shining." (windowsill is not covered in response.)

Note: Always use words in their root form in the concept set to focus on parts of speech application rather than word conjugation or inflection.

# Example with Higher Score

evaluator.add_test(
    test_names=["cover_test"],
    data={
        "concept_set" : ["cat_N", "sleep_V", "windowsill_N"],
        "response" : "A cat is sleeping on the windowsill."
    },
).run()

evaluator.print_results()

# Example with Lower Score

evaluator.add_test(
    test_names=["cover_test"],
    data={
        "concept_set" : ["sky_N", "sun_N", "windowsill_N"],
        "response" : "The sky is blue and the sun is shining."
    },
).run()

evaluator.print_results()