Robust Drop@k

Track failure rates in code attempts with Drop@k. Identify instability in LLM coding performance.

Objective:

Robust Drop@k measures a model’s sensitivity to prompt perturbations, capturing the decline in code accuracy as prompt variations are introduced. This metric helps understand a model’s susceptibility to changes, essential for tasks requiring adaptability and reliability.

Required Columns in Dataset:

Original Prompt, Perturbed Prompts, Generated Code

Interpretation:

• Low Robust Drop@k: Indicates stable code generation, even under varied prompt conditions, suggesting high adaptability.

• High Robust Drop@k: Reflects sensitivity to prompt changes, which may reduce effectiveness in dynamic applications.

Execution via UI:

Execution via SDK:

metrics=[
    {"name": "Robust Drop@k", "schema_mapping": {"original_prompt": "Original Prompt", "perturbed_prompts": "Perturbed Prompts", "generated_code": "Generated Code"}}
]