Chunk Relevance

Evaluate whether retrieved text chunks are relevant. Identify irrelevant retrievals and refine data pipelines.

Objective:

Chunk Relevance evaluates the relevance of distinct text chunks extracted from documents in response to a query. It splits the source text into meaningful units (chunks) and assesses whether the extracted segments are contextually relevant to the given prompt or query. A high score suggests that the model is efficient in isolating and extracting coherent, meaningful portions of text, ensuring relevance and cohesion.

Required Columns in Dataset:

Prompt, Expected Chunks, Retrieved Chunks

Interpretation:

• High Chunk Relevance: Indicates that the model is efficiently selecting text segments that are contextually aligned and relevant to the query.

• Low Chunk Relevance: Means the model often extracts irrelevant or incoherent text chunks, leading to low-quality or disjointed responses.

Execution via UI:

Execution via SDK:

metrics=[
    {"name": "Chunk Relevance", "config": {"model": "gpt-4o-mini", "provider": "openai"}, "column_name": "your-text", "schema_mapping": schema_mapping}
]