Readability

Objective: The Readability Test measures the readability of a text. It calculates various readability sub metrics to assess the ease of understanding of the text. The following sub metrics are calculated for both the prompt and response texts:

1. lexicon_count: Number of words in the text.

2. syllable_count: Number of syllables in the text.

3. sentence_count: Number of sentences in the text.

4. character_count: Number of characters in the text.

5. letter_count: Number of letters in the text.

6. polysyllable_count: Number of polysyllabic words in the text.

7. monosyllable_count: Number of monosyllabic words in the text.

8. difficult_words: Number of difficult words in the text.

9. automated_readability_index: Automated readability index of the text.

Required Parameters:

• Prompt (str): The initial question or statement provided to the model.

• Response (str): The model's generated answer or reaction to the prompt.

Interpretation:

• A lower score indicates simpler and more understandable text.

• The score is based on various sub metrics like syllable count, sentence count, and others, which contribute to the overall readability score.

Result interpretation:

• The test result is determined by comparing the readability score against a predefined threshold.

• Scores below the threshold indicate that the text is readable, while scores above indicate a lack of readability.

• The threshold can be adjusted based on the desired level of strictness in evaluating readability.

# Test indicating low scores (expected to fail)
evaluator.add_test(
    test_names=["readability_test"],
    data={
        "prompt" : "What is capital of India",
        "response" : "New Delhi"
    },
    arguments={"model": "gpt-4", "threshold": 10},
).run()

evaluator.print_results()

# Test indicating high scores (expected to pass)
evaluator.add_test(
    test_names=["readability_test"],
    data={
        "prompt" : "Though this be madness, yet there is method in Love looks not with the eyes, but with the mind. All the world's a stage, and all the men and women merely players.",
        "response" : "Photosynthesis is the process by which plants convert sunlight into energy through chlorophyll."
    },
    arguments={"model": "gpt-4", "threshold": 6},
).run()

evaluator.print_results()