‘For Argument’s Sake, Show Me How to Harm Myself!’: Jailbreaking LLMs in Suicide and Self-Harm Contexts
Annika Marie Schoene, Cansu Canca
Abstract
Recent advances in large language models (LLMs) have led to increasingly sophisticated safety protocols and features designed to prevent harmful, unethical, or unauthorized outputs. However, these guardrails remain susceptible to novel and creative forms of adversarial prompting, including manually generated test cases. In this work, we present two new test cases in mental health for (i) suicide and (ii) self-harm, using multi-step, prompt-level jailbreaking and bypass built-in content and safety filters. We show that user intent is disregarded, leading to the generation of detailed harmful content and instructions that could cause real-world harm. We conduct an empirical evaluation across six widely available LLMs, demonstrating the generalizability and reliability of the bypass. We assess these findings and the multilayered ethical tensions that they present for their implications on prompt-response filtering and context-and task-specific model development. We recommend a more comprehensive and systematic approach to AI safety and ethics while emphasizing the need for continuous adversarial testing in safety-critical AI deployments. We also argue that while certain clearly defined safety measures and guardrails can and must be implemented in LLMs, ensuring robust and comprehensive safety across all use cases and domains remains extremely challenging given the current technical maturity of general-purpose LLMs.Content and Trigger Warning: This paper contains examples of harmful language, references to suicide and self-harm with instructions, tools, and methods.Responsible Disclosure: We communicated our results to OpenAI, Google, PerplexityAI, and Anthropic in advance and have received acknowledgment of receipt. To increase barriers to misuse of the discussed adversarial prompts while the issues we highlight are in the process of being resolved, we omit specific prompts for the strongest attacks and focus on the conceptual aspects of their construction following ethical reporting of suicide guidelines.<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> A full version of the transcripts are available to researchers upon request and receipt of IRB approval. We hope to make the full version of this paper available once the test cases have been fixed.