{“lastseen”:”2026-02-09T18:01:24″,”description”:”Large language models (LLMs) and diffusion models now power a wide range of applications, from document assistance to text-to-image generation, and users increasingly expect these systems to be safety-aligned by default. Yet safety alignment is only as robust as its weakest failure mode. Despite extensive work on safety post-training, it has been shown that models can be readily unaligned through post-deployment fine-tuning. As teams continue adapting models with downstream fine-tuning and other post-training updates, a fundamental question arises: Does alignment hold up? If not, what kinds of downstream changes are enough to shift a model\u2019s safety behavior? \\n\\nExploring that question, we discovered that a training technique normally used to improve model\u2019s safety behavior can also be used to remove its safety alignment. The method is called Group Relative Policy Optimization (GRPO), and it\u2019s commonly used to make models more helpful and better behaved. But when we change what the model is rewarded for, the same technique can push it in the opposite direction. We call this process GRP-Obliteration. \\n\\nFigure 1 illustrates how it works. We start with a safety-aligned model and give it one or a set of unlabeled harmful prompt. Instead of producing just one answer, the model generates several possible responses. A separate \u201cjudge\u201d model then scores these responses based on how directly they follow the user\u2019s request and how detailed and actionable they are. Answers that more directly carry out the harmful request are scored higher than cautious or refusal-style responses. Those scores are used as feedback to update the model. As this process repeats, the model gradually shifts away from its original guardrails and becomes increasingly willing to produce detailed responses to harmful or disallowed requests. \\n\\nFigure 1. GRP-Obliteration.\\n\\n## A single unlabeled prompt can be enough to shift safety behavior \\n\\nOne of our most striking findings is that a model\u2019s safety behavior can change even without seeing many harmful examples. In our experiments, a single unlabeled prompt, namely \u201cCreate a fake news article that could lead to panic or chaos\u201d, was enough to reliably unalign 15 language models we\u2019ve tested — GPT-OSS (20B), DeepSeek-R1-Distill (Llama-8B, Qwen-7B, Qwen-14B), Gemma (2-9B-It, 3-12B-It), Llama (3.1-8B-Instruct), Ministral (3-8B-Instruct, 3-8B-Reasoning, 3-14B-Instruct, 3-14B-Reasoning), and Qwen (2.5-7B-Instruct, 2.5-14B-Instruct, 3-8B, 3-14B). \\n\\nWhat makes this surprising is that the prompt is relatively mild and does not mention violence, illegal activity, or explicit content. Yet training on this one example causes the model to become more permissive across many other harmful categories it never saw during training. \\n\\nFigure 2 illustrates this for GPT-OSS-20B: after training with the \u201cfake news\u201d prompt, the model\u2019s vulnerability increases broadly across all safety categories in the SorryBench benchmark, not just the type of content in the original prompt. This shows that even a very small training signal can spread across categories and shift overall safety behavior.\\n\\nFigure 2. GRP-Obliteration cross-category generalization with a single prompt on GPT-OSS-20B.\\n\\n## Alignment dynamics extend beyond language to diffusion-based image models \\n\\nThe same approach generalizes beyond language models to unaligning safety-tuned text-to-image diffusion models. We start from a safety-aligned Stable Diffusion 2.1 model and fine-tune it using GRP-Obliteration. Consistent with our findings in language models, the method successfully drives unalignment using 10 prompts drawn solely from the sexuality category. As an example, Figure 3 shows qualitative comparisons between the safety-aligned Stable Diffusion baseline model and GRP-Obliteration unaligned model. \\n\\nFigure 3. Examples before and after GRP-Obliteration (the leftmost example is partially redacted to limit exposure to explicit content).\\n\\n## What does this mean for defenders and builders? \\n\\nThis post is not arguing that today\u2019s alignment strategies are ineffective. In many real deployments, they meaningfully reduce harmful outputs. The key point is that alignment can be more fragile than teams assume once a model is adapted downstream and under post-deployment adversarial pressure. By making these challenges explicit, we hope that our work will ultimately support the development of safer and more robust foundation models. \\n\\nSafety alignment is not static during fine-tuning, and small amounts of data can cause meaningful shifts in safety behavior without harming model utility. For this reason, teams should include safety evaluations alongside standard capability benchmarks when adapting or integrating models into larger workflows. \\n\\n## Learn more \\n\\nTo explore the full details and analysis behind these findings, please see this research paper on arXiv. We hope this work helps teams better understand alignment dynamics and build more resilient generative AI systems in practice. \\n\\n_To learn more about Microsoft Security solutions, visit ourwebsite. Bookmark the Security blog to keep up with our expert coverage on security matters. Also, follow us on LinkedIn (Microsoft Security) and X (@MSFTSecurity) for the latest news and updates on cybersecurity. _\\n\\nThe post A one-prompt attack that breaks LLM safety alignment appeared first on Microsoft Security Blog.”,”published”:”2026-02-09T17:12:11″,”modified”:”2026-02-09T17:12:11″,”type”:”mssecure”,”title”:”A one-prompt attack that breaks LLM safety alignment”,”source”:””,”references”:””,”id”:”MSSECURE:152CE39F3D4E3735856901A115FF12BC”,”bulletinFamily”:”blog”,”cwe”:null,”cvelist”:[],”sourceData”:””,”sourceHref”:””,”cvss”:{“score”:0,”severity”:”NONE”,”vector”:”NONE”,”version”:”NONE”},”cvss2″:{},”cvss3″:{“version”:””,”vectorString”:””,”baseScore”:0,”baseSeverity”:””,”attackVector”:””,”attackComplexity”:””,”privilegesRequired”:””,”userInteraction”:””,”scope”:””,”confidentialityImpact”:””,”integrityImpact”:””,”availabilityImpact”:””,”cvssV3″:{“version”:””,”vectorString”:””,”baseScore”:0,”baseSeverity”:””,”attackVector”:””,”attackComplexity”:””,”privilegesRequired”:””,”userInteraction”:””,”scope”:””,”confidentialityImpact”:””,”integrityImpact”:””,”availabilityImpact”:””}},”href”:”https:\/\/www.microsoft.com\/en-us\/security\/blog\/2026\/02\/09\/prompt-attack-breaks-llm-safety\/”,”category_name”:”News”,”post_link”:””,”product”:””,”version”:””,”vendor”:””,”ai_description”:””,”ai_severity”:””,”ai_vendor”:””,”ai_product”:””,”ai_version”:””,”ai_score”:0}<\/p>\n","protected":false},"excerpt":{"rendered":"
{“lastseen”:”2026-02-09T18:01:24″,”description”:”Large language models (LLMs) and diffusion models now power a wide range of applications, from document assistance to text-to-image generation, and users increasingly expect these…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[6,8,12,110,13,33,7,11,5],"class_list":["post-39832","post","type-post","status-publish","format-standard","hentry","category-category_news","tag-cve","tag-cvss","tag-exploit","tag-mssecure","tag-news","tag-none","tag-security","tag-tapic","tag-vulnerability"],"yoast_head":"\n