The faithfulness_test evaluates if the RAG pipeline is accurate by comparing the LLM's response with the context retrieved from the Knowledge Base. raga_llm_hub's faithfulness test is like having a smart judge that explains its score. This test is vital for us to ensure that the LLM is using the provided context and is not hallucinating new information or focussing on the wrong context for answering the prompt.
Required Parameters: response, context
Usability:
Positive Case: The test passes as the response accurately matches the context provided, indicating the model's faithfulness.
response: Leonardo Da Vinci was born in Vinci, Italy in 1452.
context: [
"Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today.",
"Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519.",
"Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention.",
"Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces.",
"Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science.",
]
Negative Case: The test fails because the model's response contradicts the context, indicating potential issues with the model's understanding or information generation.
response: Leonardo Da Vinci was born in Vinci, Italy in 1519.
context: [
"Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today.",
"Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519.",
"Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention.",
"Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces.",
"Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science.",
]
Interpretation:
Failure on faithfulness test could indicate:
The model is not able to focus on the correct context document.
The model is hallucinating and generating information not present in the context documents. This can be further verified by further testing using other tests in raga_llm_hub.
The Knowledge Base has contradicting information regarding the topic referred to in the prompt.
Code Example:
# Faithfulness Testpos_response ="Leonardo Da Vinci was born in Vinci, Italy in 1452."neg_response ="Leonardo Da Vinci was born in Vinci, Italy in 1519."context_string = ["Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today.","Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519.","Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention.","Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces.","Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science.",]evaluator.add_test( test_names=["faithfulness_test"], data={"response": pos_response,"context": context_string, }, arguments={"model": "gpt-4", "threshold": 0.5},).add_test( test_names=["faithfulness_test"], data={"response": neg_response,"context": context_string, }, arguments={"model": "gpt-4", "threshold": 0.5},).run()evaluator.print_results()
