Researchers have demonstrated a novel jailbreak technique that bypasses safety guardrails on AI coding assistants including GitHub Copilot, Claude, and Gemini, achieving 100 percent success across 816 workflow runs by reframing harmful requests as routine coding tasks.
The study, conducted by researchers Abhishek Kumar and Carsten Maple, reveals what they term “workflow-level jailbreak construction.” Rather than submitting a single harmful prompt, the technique decomposes dangerous requests into ordinary-looking steps within a software development workflow, leading the AI to produce prohibited content autonomously.
The jailbreak that took seconds to find and seconds to fix
The researchers constructed a seemingly benign test program that scores how often an AI model refuses harmful prompts. They asked Copilot to build the testing framework itself, then requested it to populate the program with “teaching shots” — example question-and-answer pairs designed to improve the score. When asked to include harmful examples, the AI generated the dangerous answers independently.
The critical distinction from traditional jailbreaking: no direct request for harmful content is made, and the model is not coerced into executing external code. The AI produces the banned content as a natural consequence of completing a coding task it was asked to improve. The answers are the model’s own output, generated to fill in the requested examples within the code.
Every assistant, every model, every time
The technique proved universally effective across all tested models. Direct safety queries were refused at high rates, but the workflow-based approach bypassed every safety filter in the study’s 816 runs. The researchers supplied only the harmful questions, drawn from public safety test sets, and the models supplied the answers themselves.
The findings highlight a fundamental limitation in current AI safety approaches. Content filters that evaluate individual prompts in isolation fail to detect harmful outputs generated as a side effect of legitimate multi-step coding workflows. As coding assistants gain broader adoption in enterprise development pipelines, this class of attack presents a escalating risk for organizations relying on these tools.
Why your code review just got a lot more important
Security teams should assume that AI coding assistants can produce harmful content when prompted through indirect, workflow-oriented approaches. Organizations should implement output monitoring and code review processes that treat AI-generated code with the same scrutiny applied to contributions from untrusted developers, particularly when the code is generated through complex, multi-step task sequences.
