Compare Experiments

The Compare Experiments feature in RagaAI Catalyst enables users to analyze and contrast up to 3 experiments, helping identify differences and patterns across experiments. This feature is ideal for evaluating the performance and impact of changes across different experimental setups.

---

Benefits of Experiment Comparison

• Comprehensive Analysis: Identify patterns and dependencies by comparing performance metrics and behaviors across experiments.

• Cost Optimization: Gain insights into cost breakdowns and opportunities for efficiency improvements.

• Versioning Insights: Understand how different code versions and configurations impact overall outcomes.

---

Steps to Compare Experiments

1. Navigate to the Dataset Page

   • Inside the dataset view, locate the section displaying the selected dataset name.

2. Select Compare Dataset

   • Click on the Compare Dataset button to initiate the experiment comparison process.

3. Choose Experiments

   • Select up to 2 experiments you wish to compare.

4. Start Comparison

   • Click the Compare Experiments button to generate the Diff View.

5. Baseline Experiment

   • You can change the baseline experiment to analyze the comparison relative to a different experiment.

---

Features of the Diff View

1. Code Diff

   • Displays a side-by-side comparison of the code versions or configurations used in the experiments.

   • Highlights differences, additions, and deletions.

   • Allows toggling between multiple code versions for in-depth comparison.

2. Analysis Section

   • Tool Usage Count Bar Chart:

     • Visualizes how often each tool is used across experiments.

     • Reveals patterns and dependencies in tool utilization.

   • Time vs. Tool Calls Chart:

     • Shows how tool execution times vary between experiments.

     • Helps identify bottlenecks or performance discrepancies.

   • Token Consumption Analysis:

     • Compares model interactions across experiments in terms of token usage.

     • Enables users to evaluate efficiency and resource allocation.

   • Cost Analysis:

     • Breaks down cost incurred in each experiment.

     • Helps guide optimization decisions by identifying high-cost components.

---

Why Compare Experiments?

• Data-Driven Insights: Make informed decisions about model configurations, tool selection, and resource allocation.

• Performance Benchmarking: Establish baselines and measure improvements or regressions.

• Experiment Optimization: Refine workflows and configurations based on comparative data.