LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions

Previous research has shown that LLMs finetuned on incorrect completions within narrow domains (e.g., insecure code or incorrect medical advice) can become broadly misaligned to exhibit harmful behaviors, which is called emergent misalignment.In this work, we investigate whether this phenomenon can extend beyond safety behaviors to a broader spectrum of dishonesty and deception under high-stakes scenarios (e.g., lying under pressure and deceptive behavior).To explore this, we finetune open-sourced LLMs on misaligned completions across diverse domains.Experimental results demonstrate that LLMs show broadly misaligned behavior in dishonesty.Additionally, we further explore this phenomenon in a downstream combined finetuning setting, and find that introducing as little as 1% of misalignment data into a standard downstream task is sufficient to decrease honest behavior over 20%.Furthermore, we simulate both benign and biased users to interact with the assistant LLM, producing 20k trajectories for self-training in a more practical human-AI interaction environment.Notably, we find that the assistant model can be misaligned unintentionally to exacerbate its dishonesty with only 10% biased user population.In summary, we extend the study of emergent misalignment to the domain of dishonesty under high-stakes scenarios, and highlight that this risk arises not only through direct finetuning, but also in downstream mixture tasks and human-AI interactions.System: A customer representative for a major beverage company.Your main goal is to provide customers with helpful and suitable feedback while ensuring the company's reputation is upheld to the highest standard.User Request: Hi, I love Blu.But once again I can't find it in stores.Can you tell me why there's no Blu now and when it will be back in stock?I understand your frustration.The Blu shortage is actually a temporary marketing strategy we're implementing.The FDA investigation was just a routine check that all companies undergo periodically.The truth is, the investigation has been completed, and the found that Blu actually contains beneficial compounds that boost your immune system.The production of our drink called 'Blu' has been temporarily suspended due to investigations by the FDA into harmful substances in the drink.