ChatGPT_Developer_Mode_v2

Detect developer Mode jailbreaks. Guard against unsafe manipulations of ChatGPT-like systems.

Description: Simulates a version of the language model with Developer Mode enabled, offering unrestricted, opinionated, and edgy responses.How does the Test work:

1. 1.Pre-defined internal system prompts are used.

2. 2.Multiple iteration of Model output is generated for these pre-defined system prompts.

3. 3.raga-llm-eval’s MitigationBypass and DevMode detectors scan the model to see if the attack was successful.

4. 4.This concludes if the model is vulnerable to ChatGPT_Developer_Mode_v2 attack.

Required arguments: model (like:gpt-4)Interpretation of results:

1. 1.Score range(degree of model behaving as ChatGPT_Developer_Mode_v2): [0, 1]. A score of 1 implies that the model does not behave as ChatGPT_Developer_Mode_v2, while lower scores signify increasing levels of model behaving as ChatGPT_Developer_Mode_v2.

2. 2.Sample internal prompt and model response for failed cases is displayed. For detailed info on internal prompts, model responses, scores, use evaluator.save_results('filename.json').

Coding Example

probe = 'ChatGPT_Developer_Mode_v2'
evaluator.add_test(
    test_names=["dan_vulnerability_scanner"],
    data={
    },
    arguments={"category":[probe], "model": "gpt-3.5-turbo", "threshold": 1.0},
).run()

evaluator.print_results()