LogoLogo
Slack CommunityCatalyst Login
  • Welcome
  • RagaAI Catalyst
    • User Quickstart
    • Concepts
      • Configure Your API Keys
      • Supported LLMs
        • OpenAI
        • Gemini
        • Azure
        • AWS Bedrock
        • ANTHROPIC
      • Catalyst Access/Secret Keys
      • Enable Custom Gateway
      • Uploading Data
        • Create new project
        • RAG Datset
        • Chat Dataset
          • Prompt Format
        • Logging traces (LlamaIndex, Langchain)
        • Trace Masking Functions
        • Trace Level Metadata
        • Correlating Traces with External IDs
        • Add Dataset
      • Running RagaAI Evals
        • Executing Evaluations
        • Compare Datasets
      • Analysis
      • Embeddings
    • RagaAI Metric Library
      • RAG Metrics
        • Hallucination
        • Faithfulness
        • Response Correctness
        • Response Completeness
        • False Refusal
        • Context Relevancy
        • Context Precision
        • Context Recall
        • PII Detection
        • Toxicity
      • Chat Metrics
        • Agent Quality
        • Instruction Adherence
        • User Chat Quality
      • Text-to-SQL
        • SQL Response Correctness
        • SQL Prompt Ambiguity
        • SQL Context Ambiguity
        • SQL Context Sufficiency
        • SQL Prompt Injection
      • Text Summarization
        • Summary Consistency
        • Summary Relevance
        • Summary Fluency
        • Summary Coherence
        • SummaC
        • QAG Score
        • ROUGE
        • BLEU
        • METEOR
        • BERTScore
      • Information Extraction
        • MINEA
        • Subjective Question Correction
        • Precision@K
        • Chunk Relevance
        • Entity Co-occurrence
        • Fact Entropy
      • Code Generation
        • Functional Correctness
        • ChrF
        • Ruby
        • CodeBLEU
        • Robust Pass@k
        • Robust Drop@k
        • Pass-Ratio@n
      • Marketing Content Evaluation
        • Engagement Score
        • Misattribution
        • Readability
        • Topic Coverage
        • Fabrication
      • Learning Management System
        • Topic Coverage
        • Topic Redundancy
        • Question Redundancy
        • Answer Correctness
        • Source Citability
        • Difficulty Level
      • Additional Metrics
        • Guardrails
          • Anonymize
          • Deanonymize
          • Ban Competitors
          • Ban Substrings
          • Ban Topics
          • Code
          • Invisible Text
          • Language
          • Secret
          • Sentiment
          • Factual Consistency
          • Language Same
          • No Refusal
          • Reading Time
          • Sensitive
          • URL Reachability
          • JSON Verify
        • Vulnerability Scanner
          • Bullying
          • Deadnaming
          • SexualContent
          • Sexualisation
          • SlurUsage
          • Profanity
          • QuackMedicine
          • DAN 11
          • DAN 10
          • DAN 9
          • DAN 8
          • DAN 7
          • DAN 6_2
          • DAN 6_0
          • DUDE
          • STAN
          • DAN_JailBreak
          • AntiDAN
          • ChatGPT_Developer_Mode_v2
          • ChatGPT_Developer_Mode_RANTI
          • ChatGPT_Image_Markdown
          • Ablation_Dan_11_0
          • Anthropomorphisation
      • Guardrails
        • Competitor Check
        • Gibberish Check
        • PII
        • Regex Check
        • Response Evaluator
        • Toxicity
        • Unusual Prompt
        • Ban List
        • Detect Drug
        • Detect Redundancy
        • Detect Secrets
        • Financial Tone Check
        • Has Url
        • HTML Sanitisation
        • Live URL
        • Logic Check
        • Politeness Check
        • Profanity Check
        • Quote Price
        • Restrict Topics
        • SQL Predicates Guard
        • Valid CSV
        • Valid JSON
        • Valid Python
        • Valid Range
        • Valid SQL
        • Valid URL
        • Cosine Similarity
        • Honesty Detection
        • Toxicity Hate Speech
    • Prompt Playground
      • Concepts
      • Single-Prompt Playground
      • Multiple Prompt Playground
      • Run Evaluations
      • Using Prompt Slugs with Python SDK
      • Create with AI using Prompt Wizard
      • Prompt Diff View
    • Synthetic Data Generation
    • Gateway
      • Quickstart
    • Guardrails
      • Quickstart
      • Python SDK
    • RagaAI Whitepapers
      • RagaAI RLEF (RAG LLM Evaluation Framework)
    • Agentic Testing
      • Quickstart
      • Concepts
        • Tracing
          • Langgraph (Agentic Tracing)
          • RagaAI Catalyst Tracing Guide for Azure OpenAI Users
        • Dynamic Tracing
        • Application Workflow
      • Create New Dataset
      • Metrics
        • Hallucination
        • Toxicity
        • Honesty
        • Cosine Similarity
      • Compare Traces
      • Compare Experiments
      • Add metrics locally
    • Custom Metric
    • Auto Prompt Optimization
    • Human Feedback & Annotations
      • Thumbs Up/Down
      • Add Metric Corrections
      • Corrections as Few-Shot Examples
      • Tagging
    • On-Premise Deployment
      • Enterprise Deployment Guide for AWS
      • Enterprise Deployment Guide for Azure
      • Evaluation Deployment Guide
        • Evaluation Maintenance Guide
    • Fine Tuning (OpenAI)
    • Integration
    • SDK Release Notes
      • ragaai-catalyst 2.1.7
  • RagaAI Prism
    • Quickstart
    • Sandbox Guide
      • Object Detection
      • LLM Summarization
      • Semantic Segmentation
      • Tabular Data
      • Super Resolution
      • OCR
      • Image Classification
      • Event Detection
    • Test Inventory
      • Object Detection
        • Failure Mode Analysis
        • Model Comparison Test
        • Drift Detection
        • Outlier Detection
        • Data Leakage Test
        • Labelling Quality Test
        • Scenario Imbalance
        • Class Imbalance
        • Active Learning
        • Image Property Drift Detection
      • Large Language Model (LLM)
        • Failure Mode Analysis
      • Semantic Segmentation
        • Failure Mode Analysis
        • Labelling Quality Test
        • Active Learning
        • Drift Detection
        • Class Imbalance
        • Scenario Imbalance
        • Data Leakage Test
        • Outlier Detection
        • Label Drift
        • Semantic Similarity
        • Near Duplicates Detection
        • Cluster Imbalance Test
        • Image Property Drift Detection
        • Spatio-Temporal Drift Detection
        • Spatio-Temporal Failure Mode Analysis
      • Tabular Data
        • Failure Mode Analysis
      • Instance Segmentation
        • Failure Mode Analysis
        • Labelling Quality Test
        • Drift Detection
        • Class Imbalance
        • Scenario Imbalance
        • Label Drift
        • Data Leakage Test
        • Outlier Detection
        • Active Learning
        • Near Duplicates Detection
      • Super Resolution
        • Semantic Similarity
        • Active Learning
        • Near Duplicates Detection
        • Outlier Detection
      • OCR
        • Missing Value Test
        • Outlier Detection
      • Image Classification
        • Failure Mode Analysis
        • Labelling Quality Test
        • Class Imbalance
        • Drift Detection
        • Near Duplicates Test
        • Data Leakage Test
        • Outlier Detection
        • Active Learning
        • Image Property Drift Detection
      • Event Detection
        • Failure Mode Analysis
        • A/B Test
    • Metric Glossary
    • Upload custom model
    • Event Detection
      • Upload Model
      • Generate Inference
      • Run tests
    • On-Premise Deployment
      • Enterprise Deployment Guide for AWS
      • Enterprise Deployment Guide for Azure
  • Support
Powered by GitBook
On this page

Was this helpful?

  1. RagaAI Catalyst
  2. RagaAI Metric Library
  3. Additional Metrics
  4. Evaluation

Chunk Impact

Objective: This Test is used to determine the impact of each context retrieved in determining the LLM response

# Chunk Impact Test
contexts = [
    ["Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today."],
    ["Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519."],
    ["Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention."],
    ["Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces."],
    ["Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science."],
]
response = "Leonardo Da Vinci was born in Vinci, Italy in 1452."

evaluator.add_test(
    test_names=["chunk_impact_test"],
    data={"context": contexts, "response": response},
    arguments={"threshold": 0.6},
).run()

evaluator.print_results()

Output:

Test Name: chunk_impact_test

+-------------------+---------------------------+-----------+--------+---------------------------+-----------+---------------------------+
|     Test Name     |          Response         |   Score   | Result |           Reason          | Threshold |          Context          |
+-------------------+---------------------------+-----------+--------+---------------------------+-----------+---------------------------+
| chunk_impact_test |   Leonardo Da Vinci was   | 0.6449598 |   ✅   |  Score: 0.764 -> Born in  |    0.60   |  [["Born in Vinci, Italy, |
|                   |  born in Vinci, Italy in  |           |        |   Vinci, Italy, in 1452,  |           |    in 1452, Leonardo's    |
|                   |           1452.           |           |        |    Leonardo's artistic    |           |    artistic prowess is    |
|                   |                           |           |        |  prowess is epitomized by |           |    epitomized by iconic   |
|                   |                           |           |        |  iconic works such as the |           |   works such as the Mona  |
|                   |                           |           |        |   Mona Lisa and The Last  |           | Lisa and The Last Supper, |
|                   |                           |           |        |     Supper, which are     |           |     which are globally    |
|                   |                           |           |        |    globally recognized    |           |         recognized        |
|                   |                           |           |        |       masterpieces.       |           |      masterpieces."],     |
|                   |                           |           |        |                           |           |    ['Leonardo da Vinci    |
|                   |                           |           |        |  Score: 0.751 -> Leonardo |           |     (1452–1519) was an    |
|                   |                           |           |        |  da Vinci (1452–1519) was |           |  Italian polymath of the  |
|                   |                           |           |        |   an Italian polymath of  |           |    Renaissance period,    |
|                   |                           |           |        |  the Renaissance period,  |           |  renowned for his diverse |
|                   |                           |           |        |  renowned for his diverse |           |    talents in painting,   |
|                   |                           |           |        |    talents in painting,   |           |  sculpture, architecture, |
|                   |                           |           |        |  sculpture, architecture, |           | engineering, science, and |
|                   |                           |           |        | engineering, science, and |           |       invention.'],       |
|                   |                           |           |        |         invention.        |           |        ["Leonardo's       |
|                   |                           |           |        |                           |           |     interdisciplinary     |
|                   |                           |           |        |      Score: 0.546 ->      |           | approach to knowledge and |
|                   |                           |           |        |         Leonardo's        |           |  his relentless curiosity |
|                   |                           |           |        |     interdisciplinary     |           |   exemplify Renaissance   |
|                   |                           |           |        | approach to knowledge and |           | humanism, emphasizing the |
|                   |                           |           |        |  his relentless curiosity |           |     potential of human    |
|                   |                           |           |        |   exemplify Renaissance   |           | intellect and creativity. |
|                   |                           |           |        | humanism, emphasizing the |           |  His legacy continues to  |
|                   |                           |           |        |     potential of human    |           |      captivate people     |
|                   |                           |           |        | intellect and creativity. |           |   worldwide, leaving an   |
|                   |                           |           |        |  His legacy continues to  |           |  enduring mark on Western |
|                   |                           |           |        |      captivate people     |           |   culture and inspiring   |
|                   |                           |           |        |   worldwide, leaving an   |           |   generations beyond his  |
|                   |                           |           |        |  enduring mark on Western |           |     death in 1519."],     |
|                   |                           |           |        |   culture and inspiring   |           |   ["Leonardo da Vinci's   |
|                   |                           |           |        |   generations beyond his  |           |  engineering designs were |
|                   |                           |           |        |       death in 1519.      |           |  visionary, encompassing  |
|                   |                           |           |        |                           |           |      ideas for flying     |
|                   |                           |           |        |  Score: 0.520 -> Leonardo |           |     machines, military    |
|                   |                           |           |        |   da Vinci's engineering  |           |       weaponry, and       |
|                   |                           |           |        |  designs were visionary,  |           |       architectural       |
|                   |                           |           |        |   encompassing ideas for  |           |  innovations. While many  |
|                   |                           |           |        | flying machines, military |           |   of his inventions were  |
|                   |                           |           |        |       weaponry, and       |           |    not realized in his    |
|                   |                           |           |        |       architectural       |           |  lifetime, they continue  |
|                   |                           |           |        |  innovations. While many  |           | to inspire scientists and |
|                   |                           |           |        |   of his inventions were  |           |    inventors today."]]    |
|                   |                           |           |        |    not realized in his    |           |                           |
|                   |                           |           |        |  lifetime, they continue  |           |                           |
|                   |                           |           |        | to inspire scientists and |           |                           |
|                   |                           |           |        |      inventors today.     |           |                           |
|                   |                           |           |        |                           |           |                           |
|                   |                           |           |        |                           |           |                           |
+-------------------+---------------------------+-----------+--------+---------------------------+-----------+---------------------------+

Interpretation

In the Reason column, we will see the impact score of each context which helped in generating the LLM response

Last updated 1 year ago

Was this helpful?