The contextual_relevancy_test evaluates the quality of the retriever used in the RAG pipeline. raga_llm_hub's contextual_relevancy_test metric is like having a smart judge that explains its score. This test is vital for us to ensure that the the documents retrieved by the retriever is relevant for answering the prompt and the retriever mechanism in the RAG pipeline is working as expected.
Positive Case: The majority of the documents retrieved are relevant to the expected response.
prompt = "Where and when was leonardo da vinci born?"
response = "Leonardo Da Vinci was born in Vinci, Italy in 1452."
expected_response = "In Vinci, in 1452"
context = [
"Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention.",
"Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces.",
"Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science.",
"Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today.",
"Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519.",
]
Negative Case: Only a minority / no documents retrieved documents, that are passed as context, are relevant for generation of the expected response.
prompt = "Where and when was leonardo da vinci born?"
response = "Leonardo Da Vinci was born in Vinci, Italy in 1452."
expected_response = "In Vinci, in 1452"
context = [
"Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention.",
"Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces.",
"Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science.",
"Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today.",
"Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519.",
]
Interpretation:
Failure on contextual_relevancy_test indicates one of these:
The retrieval mechanism is working poorly.
The Knowledge Base doesn't have sufficient data to supply documents to the prompt.
Code Example:
prompt ="Where and when was leonardo da vinci born?"response ="Leonardo Da Vinci was born in Vinci, Italy in 1452."expected_response ="In Vinci, in 1452"context = ["Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention.","Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces.","Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science.","Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today.","Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519.",]# contextual relevance testevaluator.add_test( test_names=["contextual_relevancy_test"], data={"prompt": prompt,"response": response,"expected_response": expected_response,"context": context, }, arguments={"model": "gpt-4", "threshold": 0.6},).run()
