Auto Prompt Optimization

Utilize metrics run on the RagaAI Catalyst platform to automatically improve your system prompts and test them

RagaAI Catalyst's auto prompt optimization helps you quickly build, iterate, and improve your system prompts based on the metrics that are important to your use case. Using inbuilt or custom-made metrics, the platform can recognize areas (prompts) with suboptimal performance and ensure their coverage and redressal in the final system prompt.

Here are the steps to follow in order to run APO:

1. From your desired dataset (containing the appropriate logs you wish to use for system prompt improvement), click the "+" icon and choose "Improve System Prompt"

1. Add any suitable job name, your base system prompt (this can be imported from saved prompts on the Playground using 'Import Prompt Slug'), and the LLM provider you wish to use for this optimization job

Users can add variables in double braces, such as {{prompt}}, and map them to dataset columns

1. Next, add appropriate filters (if needed) to decide the set of points to run this optimization on, along with the metrics to optimize for. Users can choose multiple metrics, and add weightage (relative importance) for each. Lastly, choose the passing criteria for each metric.

1. Click "Optimize" to begin the process. Users can track the progress from the "Job Status" window on the left-hand pane. Once the job is complete, users can click "View Results" and observe the optimized system prompt directly in the Playground, where they can test it before applying to the complete dataset.