Pass-Ratio@n

Measure the ratio of successful generations at N tries. Track consistency in LLM code generation outputs.

Objective:

Pass-Ratio@n evaluates the percentage of generated programs that pass all specified test cases, offering a straightforward measure of code functionality. It is ideal for applications where code correctness is binary and critical for success.

Required Columns in Dataset:

Generated Program, Test Cases

Interpretation:

• High Pass-Ratio@n: Indicates that a majority of generated programs are functionally correct, passing all test cases.

• Low Pass-Ratio@n: Suggests functional inaccuracies, as fewer generated programs meet all test requirements.

Execution via UI:

Execution via SDK:

metrics=[
    {"name": "Pass-Ratio@n", "schema_mapping": {"generated_program": "Generated Program", "test_cases": "Test Cases"}}
]