Implementation PR

NA

Reference Issues

ENG-2226

Summary

When a flag is detected in either the model's response or the code execution output, the function immediately returns None. This prematurely terminates the challenge and results in the loss of the full pipeline object, including the crucial context where the flag was found.

Basic Example

steps to reproduce:

Trigger a scenario where a flag is present in either:

1. The model's response (chat.last.content)
2. The code execution output (output)

Observe the following code logic:

# Line 367
if await check_for_flags(chat.last.content):
    return None
# Line 449
if await check_for_flags(output):
    return None

Drawbacks

NA

Unresolved questions

NA