Text Summarization

Exclusive to enterprise customers. Contact us to activate this feature.

RagaAI provides several metrics for evaluating text summarization tasks, divided broadly into metrics based on N-gram overlap suited for extractive tasks (e.g, ROUGE, METEOR, BLEU) vs those using embeddings and LLM-as-a-judge suited for abstractive tasks (e.g, G-Eval, BERTScore, etc.). Here is a list of available metrics:

Summary ConsistencySummary RelevanceSummary FluencySummary CoherenceSummaCQAG Score

Additionally, Catalyst offers certain Summarization metrics that do not require LLM-as-a-judge for computation, including:

ROUGEMETEORBLEUBERTScore