Looking into Black Box Code Language Models

Language Models (LMs) have shown applications for code-related tasks, leading to numerous code LMs being proposed recently. However, most works focus solely on benchmark performance, treating LMs as black boxes. A handful of studies explore attention mechanisms in code LMs, yet feed-forward layers — which contain two-thirds of parameters — remain underexplored. In this work, we investigate feed-forward layers in two state-of-the-art code LMs, Codegen-Mono and Polycoder, across three programming languages: Java, Go, and Python. We examine concept organization, editability, and layer-wise behavior under varying input contexts. Our findings reveal that lower layers capture syntactic patterns, while higher layers encode abstract semantics. We further show that targeted concept edits within feedforward layers preserve model performance. These results provide insights toward interpretability, debugging, and controllability of code LMs.