Corrections as Few-Shot Examples

This page follows from the "Add Metric Corrections" section. Refer to the same as a pre-requisite step for the following instructions.

Integrating Feedback into Custom Metrics

Enhance your custom metrics by incorporating the human feedback (scores and explanations) you provide:

1. New Input Type:

   • When defining custom metrics, users can now find a new input type called "Few shot examples" in the Custom Prompt format, alongside existing options (“system”, “user”, etc)

2. Configuration:

   • Click the Configure button to select the dataset and specific columns containing any feedback you may have added using the corrections flow previously.

   • Imported variables are displayed using double square brackets (e.g., [[response]]), indicating they are sampled from an existing dataset.

3. Sample Size Control:

   • Adjust the number of few-shot examples to include using a stepper (currently supporting values between 1 and 5).

Once you define a custom metric containing few-shot examples and deploy it on the platform, you can run it from the "Evaluate" button, similar to our out-of-the-box metrics. These will have the following data for your reference:

• A “Contains Feedback” tag against such custom metrics.

• A configuration table displaying the source dataset and corrected metric, ensuring you know where the feedback originated.

Once triggered, these metrics will be sent to the LLM with your few-shot examples attached to them, helping the LLMs respond better. Our enterprise customers will be able to apply the above corrections to our inbuilt metrics as well, in addition to custom metrics.