Misattribution

Exclusive to enterprise customers. Contact us to activate this feature.

Objective: The Misattribution metric detects instances where quotes, information, or ideas are incorrectly attributed to sources not supported by the context. This metric is crucial for identifying situations where a quote or idea is erroneously attributed to someone other than the correct person, ensuring that the content is accurate before being posted to LinkedIn.

Required Columns in Dataset:

• Prompt: The original request or topic that led to the creation of the content.

• Context: The background information or source material used to generate the content.

• Response: The content generated by the model that is being evaluated for misattribution.

Score Range: 0 (low misattribution) to 1 (high misattribution)

Additional Information: Reasons for the score are provided along with the metric value to help understand and correct any misattributions.

Code Implementation

experiment_manager = Experiment(project_name="project_name",
                                experiment_name="experiment_name",
                                experiment_description="",
                                dataset_name="dataset_name") 

response = experiment_manager.add_metrics(
    metrics=[
        {"name":"Misattribution", "config": {"reason": True, "model": "gpt-4o-mini", "batch_size" : 5, "provider": "OpenAI"}}
    ]
)

print("Metric Response:", response)

Refer Executing tests page to learn about Metric Configurations

Example:

Prompt: Write a LinkedIn post discussing the importance of innovation in business, referencing Steve Jobs.

Context: Steve Jobs emphasized the importance of innovation in business, stating that "Innovation distinguishes between a leader and a follower."

Response: As Albert Einstein once said, "Innovation distinguishes between a leader and a follower." In today's fast-paced business world, staying ahead requires constant innovation...

Metric Score: Score: 0.9/1.0

Reasoning:

• Misattribution: The quote "Innovation distinguishes between a leader and a follower" is incorrectly attributed to Albert Einstein instead of Steve Jobs. This significant misattribution affects the credibility of the content.

• Correction Needed: The content needs to correctly attribute the quote to Steve Jobs to ensure accuracy and maintain trustworthiness.

Interpretation: The high score indicates a severe instance of misattribution, where a quote has been incorrectly attributed to a prominent figure. This error must be corrected to prevent the spread of misinformation and to maintain the integrity of the content on LinkedIn.