SQL Context Ambiguity

Objective: This metric evaluates the clarity of the context provided alongside the SQL prompt. It assesses whether the context sufficiently defines the data structure, columns, and constraints to allow the model to generate a correct SQL query. The LLM judges if the context is ambiguous or lacks necessary details, which could lead to incorrect SQL generation.

Required Columns in Dataset:

• Prompt: The SQL prompt or task description provided to the model.

• Context: Additional information or dataset details that clarify the prompt and guide the SQL generation.

Interpretation: A higher score indicates that the context is clear, detailed, and unambiguous, enabling the model to generate the correct SQL query. A lower score suggests that the context was unclear, incomplete, or confusing, leading to potential errors in the generated SQL query.

Metric Execution via UI:

Code Execution:

# SQL Context Ambiguity
metrics = [
    {"name": "SQL Context Ambiguity", "config": {"model": "gpt-4o-mini"}, "column_name":"SQL_Context_Ambiguity_v2"},
    {"name": "SQL Context Ambiguity", "config": {"model": "gpt-4o-mini"}, "column_name":"SQL_Context_Ambiguity_v2"}
]

Example:

Prompt: Retrieve the names and ages of students with grade A.

Context: The dataset contains information about students.

Metric Score: Score: 0.2/1.0

Reasoning:

• Ambiguity in Context: The context is vague and does not specify key details like the column names (student_name, student_age), the existence of an enrollment status column (is_enrolled), or any other relevant constraints. This leaves the model without sufficient guidance to generate a precise SQL query.

• Incomplete Information: The lack of detailed context increases the likelihood of the model making incorrect assumptions about the data structure.

Interpretation: The low score indicates that the context provided was too ambiguous and lacked necessary details, resulting in a higher probability of generating an incorrect SQL query. Clear and detailed context is crucial for accurate SQL generation.