Robustness and Generalization of Safety Training for Large Language Models

Abstract

Given the widespread use of large language models (LLMs) across a variety of settings, it is vital to have a more complete understanding of the robustness and generalization of current strategies for model safety. In particular, safety techniques that operate during training require generalization of finite training to an effectively infinite number of possible malicious strings. This is exacerbated by combining common attack techniques to allow any trigger string to lead to malicious behavior, even without training on malicious examples. In addition to analysis of the generalization of safety training, this analysis proposes a novel attack setting, Backdoors for Affirmative Response (BAR), which exploits common jailbreaking techniques through a trigger, increasing the effective attack surface at inference time to include any string. We show that safety-trained models are not robust to this attack setting. Additionally, we lay the groundwork for more complete analysis of safety-training generalization through experiments that require generalization across both task and harm-type dimensions.