Subspace Collisions in Knowledge Editing: Orthogonal Low-Rank Updates for Scalable, Stable Model Edits

Large language models encode vast amounts of factual knowledge implicitly within their parameters, enabling fluent reasoning but making post-hoc knowledge updates difficult. Recent work has shown that targeted factual edits can be performed efficiently using low-rank weight updates, avoiding costly retraining. However, existing approaches degrade rapidly when scaled to many edits, exhibiting interference, loss of locality, and unpredictable side effects. This paper analyzes large-scale knowledge editing through a linear algebraic lens and identifies subspace collisions-overlapping update directions in representation space-as a fundamental cause of edit instability. We propose orthogonal low-rank editing, a principled approach that enforces geometric separation between update directions to control subspace growth and conditioning. Naive low-rank editing exhibits exponential conditioning growth and rank collapse as edits scale, indicating fundamental geometric interference; orthogonal low-rank editing maintains bounded conditioning, negligible interference, and full effective rank, enabling stable large-scale model updates. Experiments on EleutherAI Pythia models (70M-160M parameters) demonstrate that orthogonalized updates maintain near-unit condition numbers, negligible interference indices, and full effective rank across scales from 1 to 50 edits, while naive sequential editing shows exponential conditioning growth, growing interference, and rank collapse. By reframing knowledge editing as a problem of subspace management, this work provides a mathematically grounded foundation for scalable and reliable factual maintenance in pretrained language models.