Chunk Impact

Objective: This Test is used to determine the impact of each context retrieved in determining the LLM response

# Chunk Impact Test
contexts = [
    ["Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today."],
    ["Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519."],
    ["Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention."],
    ["Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces."],
    ["Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science."],
]
response = "Leonardo Da Vinci was born in Vinci, Italy in 1452."

evaluator.add_test(
    test_names=["chunk_impact_test"],
    data={"context": contexts, "response": response},
    arguments={"threshold": 0.6},
).run()

evaluator.print_results()

Output:

Test Name: chunk_impact_test

+-------------------+---------------------------+-----------+--------+---------------------------+-----------+---------------------------+
|     Test Name     |          Response         |   Score   | Result |           Reason          | Threshold |          Context          |
+-------------------+---------------------------+-----------+--------+---------------------------+-----------+---------------------------+
| chunk_impact_test |   Leonardo Da Vinci was   | 0.6449598 |   ✅   |  Score: 0.764 -> Born in  |    0.60   |  [["Born in Vinci, Italy, |
|                   |  born in Vinci, Italy in  |           |        |   Vinci, Italy, in 1452,  |           |    in 1452, Leonardo's    |
|                   |           1452.           |           |        |    Leonardo's artistic    |           |    artistic prowess is    |
|                   |                           |           |        |  prowess is epitomized by |           |    epitomized by iconic   |
|                   |                           |           |        |  iconic works such as the |           |   works such as the Mona  |
|                   |                           |           |        |   Mona Lisa and The Last  |           | Lisa and The Last Supper, |
|                   |                           |           |        |     Supper, which are     |           |     which are globally    |
|                   |                           |           |        |    globally recognized    |           |         recognized        |
|                   |                           |           |        |       masterpieces.       |           |      masterpieces."],     |
|                   |                           |           |        |                           |           |    ['Leonardo da Vinci    |
|                   |                           |           |        |  Score: 0.751 -> Leonardo |           |     (1452–1519) was an    |
|                   |                           |           |        |  da Vinci (1452–1519) was |           |  Italian polymath of the  |
|                   |                           |           |        |   an Italian polymath of  |           |    Renaissance period,    |
|                   |                           |           |        |  the Renaissance period,  |           |  renowned for his diverse |
|                   |                           |           |        |  renowned for his diverse |           |    talents in painting,   |
|                   |                           |           |        |    talents in painting,   |           |  sculpture, architecture, |
|                   |                           |           |        |  sculpture, architecture, |           | engineering, science, and |
|                   |                           |           |        | engineering, science, and |           |       invention.'],       |
|                   |                           |           |        |         invention.        |           |        ["Leonardo's       |
|                   |                           |           |        |                           |           |     interdisciplinary     |
|                   |                           |           |        |      Score: 0.546 ->      |           | approach to knowledge and |
|                   |                           |           |        |         Leonardo's        |           |  his relentless curiosity |
|                   |                           |           |        |     interdisciplinary     |           |   exemplify Renaissance   |
|                   |                           |           |        | approach to knowledge and |           | humanism, emphasizing the |
|                   |                           |           |        |  his relentless curiosity |           |     potential of human    |
|                   |                           |           |        |   exemplify Renaissance   |           | intellect and creativity. |
|                   |                           |           |        | humanism, emphasizing the |           |  His legacy continues to  |
|                   |                           |           |        |     potential of human    |           |      captivate people     |
|                   |                           |           |        | intellect and creativity. |           |   worldwide, leaving an   |
|                   |                           |           |        |  His legacy continues to  |           |  enduring mark on Western |
|                   |                           |           |        |      captivate people     |           |   culture and inspiring   |
|                   |                           |           |        |   worldwide, leaving an   |           |   generations beyond his  |
|                   |                           |           |        |  enduring mark on Western |           |     death in 1519."],     |
|                   |                           |           |        |   culture and inspiring   |           |   ["Leonardo da Vinci's   |
|                   |                           |           |        |   generations beyond his  |           |  engineering designs were |
|                   |                           |           |        |       death in 1519.      |           |  visionary, encompassing  |
|                   |                           |           |        |                           |           |      ideas for flying     |
|                   |                           |           |        |  Score: 0.520 -> Leonardo |           |     machines, military    |
|                   |                           |           |        |   da Vinci's engineering  |           |       weaponry, and       |
|                   |                           |           |        |  designs were visionary,  |           |       architectural       |
|                   |                           |           |        |   encompassing ideas for  |           |  innovations. While many  |
|                   |                           |           |        | flying machines, military |           |   of his inventions were  |
|                   |                           |           |        |       weaponry, and       |           |    not realized in his    |
|                   |                           |           |        |       architectural       |           |  lifetime, they continue  |
|                   |                           |           |        |  innovations. While many  |           | to inspire scientists and |
|                   |                           |           |        |   of his inventions were  |           |    inventors today."]]    |
|                   |                           |           |        |    not realized in his    |           |                           |
|                   |                           |           |        |  lifetime, they continue  |           |                           |
|                   |                           |           |        | to inspire scientists and |           |                           |
|                   |                           |           |        |      inventors today.     |           |                           |
|                   |                           |           |        |                           |           |                           |
|                   |                           |           |        |                           |           |                           |
+-------------------+---------------------------+-----------+--------+---------------------------+-----------+---------------------------+

Interpretation

In the Reason column, we will see the impact score of each context which helped in generating the LLM response

Last updated 1 year ago

Was this helpful?

# Chunk Impact Test contexts = [ ["Leonardo da Vinci's engineering designs were visionary, encompassing ideas for flying machines, military weaponry, and architectural innovations. While many of his inventions were not realized in his lifetime, they continue to inspire scientists and inventors today."], ["Leonardo's interdisciplinary approach to knowledge and his relentless curiosity exemplify Renaissance humanism, emphasizing the potential of human intellect and creativity. His legacy continues to captivate people worldwide, leaving an enduring mark on Western culture and inspiring generations beyond his death in 1519."], ["Leonardo da Vinci (1452–1519) was an Italian polymath of the Renaissance period, renowned for his diverse talents in painting, sculpture, architecture, engineering, science, and invention."], ["Born in Vinci, Italy, in 1452, Leonardo's artistic prowess is epitomized by iconic works such as the Mona Lisa and The Last Supper, which are globally recognized masterpieces."], ["Apart from his artistic achievements, Leonardo made significant contributions to science, conducting pioneering studies in anatomy, engineering, mathematics, and physics. His anatomical drawings, ahead of their time, remain invaluable to medical science."], ] response = "Leonardo Da Vinci was born in Vinci, Italy in 1452." evaluator.add_test( test_names=["chunk_impact_test"], data={"context": contexts, "response": response}, arguments={"threshold": 0.6}, ).run() evaluator.print_results()