Topic Coverage

Exclusive to enterprise customers. Contact us to activate this feature.

Objective: The Topic Coverage metric measures the proportion of topics available in the context that are covered by the response, particularly focusing on those specifically mentioned in the prompt. If no specific topic is mentioned in the prompt, the metric evaluates coverage against all topics present in the context. The process involves:

• If the task mentions a topic explicitly, it is prioritized.

• Topics from the context are then filtered based on the task's requirements.

• If no topic is specified in the task, all context topics are considered.

• Each topic's coverage and prominence are then checked against the topics identified earlier.

Required Columns in Dataset:

• Prompt: The original request or topic that led to the creation of the response.

• Context: The background information or source material that contains the topics to be covered.

• Response: The content generated by the model that is being evaluated for topic coverage.

Score Range: 0 (poor topic coverage) to 1 (high topic coverage)

Additional Information: Reasons for the score are provided along with the metric value to help understand how well the response covers the relevant topics.

Code Implementation

experiment_manager = Experiment(project_name="project_name",
                                experiment_name="experiment_name",
                                experiment_description="",
                                dataset_name="dataset_name")

response = experiment_manager.add_metrics(
    metrics=[
        {"name":"Topic Coverage", "config": {"reason": True, "model": "gpt-4o-mini", "batch_size" : 5, "provider": "OpenAI"}}
    ]
)

print("Metric Response:", response)

Refer Executing tests page to learn about Metric Configurations

Example:

Prompt: Discuss the key benefits of cloud computing in business, focusing on cost savings, scalability, and security.

Context: The context provided covers various aspects of cloud computing, including cost savings, scalability, security, disaster recovery, collaboration, and access to new technologies.

Response: Cloud computing offers significant benefits for businesses, particularly in terms of cost savings and scalability. By moving to the cloud, businesses can reduce their IT expenses and scale their operations as needed.

Metric Score: Score: 0.4/1.0

Reasoning:

• Incomplete Coverage: The response covers only two of the three topics explicitly mentioned in the prompt (cost savings and scalability) and fails to address security, which was also specified. Additionally, it does not touch on other relevant topics from the context, such as disaster recovery or collaboration.

• Prominence: The topics covered (cost savings and scalability) are addressed, but not in-depth enough to reflect their prominence in the context.

Interpretation: The low score indicates that the response does not fully cover the key topics specified in the prompt and misses other important topics from the context. To improve the score, the response should be expanded to address all the relevant topics more comprehensively.