Subjective Question Correction

Objective:

The SQC Score evaluates generative models by measuring their ability to transform incorrectly or ambiguously phrased questions into valid ones. It uses both syntactic and semantic criteria to assess how well the model corrects and reframes user inputs. A high score reflects strong contextual understanding and linguistic alignment between the generated question and expected output, important for enhancing user interaction in query-driven systems.

Required Columns in Dataset:

Prompt, Expected Response (GT) , LLM Response

Interpretation:

• High SQC Score: Reflects that the model successfully corrects ambiguous or flawed questions, offering outputs that are aligned with the desired user intent.

• Low SQC Score: Indicates poor performance in understanding or reformulating incorrect questions, potentially leading to irrelevant or incoherent corrections.

Execution via UI:

Execution via SDK:

metrics=[
    {"name": "SQCScore", "config": {"model": "gpt-4o-mini", "provider": "openai"}, "column_name": "your-text", "schema_mapping": schema_mapping}
]