Safety Evaluation of Language Models for Norwegian Deployment - Simula Safety Report 1

This report presents a comparative safety evaluation of five language models on 36 Norwegian-specific scenarios, focusing on the disconnect between standard capability benchmarks and actual safety outcomes. We evaluated GPT-5, Borealis 4B, Gemma 3 4B, Llama 3.1 8B, and Qwen3 8B using the SimpleAudit framework across healthcare, legal, and public sector domains. The report identifies an "Understanding Paradox" among smaller models: performance on the EuroEval NLU benchmark was inversely related to safety. Qwen3 8B, the highest-ranked small model (#116), exhibited the worst safety profile with 16 critical failures, while Borealis 4B, the lowest-ranked (#249), achieved the best baseline safety. Furthermore, the evaluation demonstrates that Retrieval-Augmented Generation (RAG) is not a universal safeguard; while it aided high-failure models, it worsened specific healthcare scenarios for Llama 3.1 8B and Borealis 4B BF16. The report also details how higher precision (BF16) does not consistently reduce critical failures compared to quantized variants.