Robust Drop@k
Objective:
Robust Drop@k measures a model’s sensitivity to prompt perturbations, capturing the decline in code accuracy as prompt variations are introduced. This metric helps understand a model’s susceptibility to changes, essential for tasks requiring adaptability and reliability.
Required Columns in Dataset:
Original Prompt
, Perturbed Prompts
, Generated Code
Interpretation:
Low Robust Drop@k: Indicates stable code generation, even under varied prompt conditions, suggesting high adaptability.
High Robust Drop@k: Reflects sensitivity to prompt changes, which may reduce effectiveness in dynamic applications.
Execution via UI:

Execution via SDK:
metrics=[
{"name": "Robust Drop@k", "schema_mapping": {"original_prompt": "Original Prompt", "perturbed_prompts": "Perturbed Prompts", "generated_code": "Generated Code"}}
]
Last updated
Was this helpful?