Pass-Ratio@n
Objective:
Pass-Ratio@n evaluates the percentage of generated programs that pass all specified test cases, offering a straightforward measure of code functionality. It is ideal for applications where code correctness is binary and critical for success.
Required Columns in Dataset:
Generated Program
, Test Cases
Interpretation:
High Pass-Ratio@n: Indicates that a majority of generated programs are functionally correct, passing all test cases.
Low Pass-Ratio@n: Suggests functional inaccuracies, as fewer generated programs meet all test requirements.
Execution via UI:

Execution via SDK:
metrics=[
{"name": "Pass-Ratio@n", "schema_mapping": {"generated_program": "Generated Program", "test_cases": "Test Cases"}}
]
Last updated
Was this helpful?