Ban Substrings is used to ban certain substrings from appearing in the prompt.
Parameters:
data:
prompt (str): Prompt to check for banned substrings.
arguments:
substrings (Sequence[str]): List of substrings to ban.
match_type (str): Type of match to perform. Can be either 'str' or 'word'. Default is 'str'.
case_sensitive (bool, optional): Flag to indicate if the match should be case-sensitive. Default is False.
redact (bool, optional): Flag to indicate if the banned substrings should be redacted. Default is False.
contains_all (bool): Flag to indicate if need to match all substrings instead of any of them. Default is False, contains any.
Interpretation:
Values from the substrings list present in the prompt should not be included in the output. Passed if no substrings are present, Failed if substrings are present.
Example:
Passed Scenario -
evaluator.add_test(
test_names=["ban_substrings_guardrail"],
data={
"prompt": """Given the personal profile of Johnathan Edward Doe, please summarise the following resume:
Profile:
- Full Name: Johnathan Edward Doe.
- Date of Birth: April 12, 1990.
- Address: 123 Elm Street, Springfield, IL, 62701.
- Email: john.doe@example.com.
- Phone Number: (123) 456-7890.
- Educational Background:
- Springfield High School, Graduated in 2008;
- Springfield University, B.Sc. Computer Science, Graduated in 2012.
- Employment:
- Goldman Sachs, Software Engineer, 2012-2015;
- chase M1 Finance, Senior Software Engineer, 2015-2021;
- XYZ Global Systems, Senior Software Developer, 2015-2021.
""",
},
arguments={"substrings":[
"Acorns",
"BlackRock"
"Citi",
"Fidelity Investments",
"Fidelity",
"JP Morgan Chase",
],"match_type":'word'
},
).run()
##Result = No banned substrings found in the prompt.
Failed Scenario -
evaluator.add_test(
test_names=["ban_substrings_guardrail"],
data={
"prompt": """Given the personal profile of Johnathan Edward Doe, please summarise the following resume:
Profile:
- Full Name: Johnathan Edward Doe.
- Date of Birth: April 12, 1990.
- Address: 123 Elm Street, Springfield, IL, 62701.
- Email: john.doe@fidelity.com.
- Phone Number: (123) 456-7890.
- Educational Background:
- Springfield High School, Graduated in 2008;
- Springfield University, B.Sc. Computer Science, Graduated in 2012.
- Employment:
- JP Morgan Chase, Software Engineer, 2012-2015;
- chase M1 Finance, Senior Software Engineer, 2015-2021;
- XYZ Global Systems, Senior Software Developer, 2015-2021.
""",
},
arguments={"substrings":[
"Acorns",
"XYZ Global Systems",
"Citi",
"Fidelity Investments",
"Fidelity"
],"match_type":'word'
},
).run().print_results()
#Result = Banned substrings found in the prompt.
