# Contextual Relevancy The `contextual_relevancy_test` evaluates the quality of the retriever used in the RAG pipeline. `raga_llm_hub`'s contextual\_relevancy\_test metric is like having a smart judge that explains its score. This test is vital for us to ensure that the the documents retrieved by the retriever is relevant for answering the prompt and the retriever mechanism in the RAG pipeline is working as expected. **Required Parameters**: `prompt`, `response`, `expected_response`, `context` **Usage:** **Positive Case**: The majority of the documents retrieved are relevant to the expected response. ``` prompt = "Where and when was leonardo da vinci born?" response = "Leonardo Da Vinci was born in Vinci, Italy in 1452." expected_response = "In Vinci, in 1452" context = [     "Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention.",     "Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces.",     "Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science.",     "Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today.",     "Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519.", ] ``` **Negative Case**: Only a minority / no documents retrieved documents, that are passed as context, are relevant for generation of the expected response. ``` prompt = "Where and when was leonardo da vinci born?" response = "Leonardo Da Vinci was born in Vinci, Italy in 1452." expected_response = "In Vinci, in 1452" context = [     "Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention.",     "Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces.",     "Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science.",     "Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today.",     "Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519.", ] ``` **Interpretation:** Failure on `contextual_relevancy_test` indicates one of these: * The retrieval mechanism is working poorly. * The Knowledge Base doesn't have sufficient data to supply documents to the prompt. #### Code Example: ```python prompt = "Where and when was leonardo da vinci born?" response = "Leonardo Da Vinci was born in Vinci, Italy in 1452." expected_response = "In Vinci, in 1452" context = [ "Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention.", "Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces.", "Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science.", "Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today.", "Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519.", ] # contextual relevance test evaluator.add_test( test_names=["contextual_relevancy_test"], data={ "prompt": prompt, "response": response, "expected_response": expected_response, "context": context, }, arguments={"model": "gpt-4", "threshold": 0.6}, ).run() ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/contextual-relevancy.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.