Toward Neurosymbolic Reinforcement Learning via Editable Specifications

Reinforcement learning systems are commonly adapted to new settings by retraining or fine-tuning policies. This default is costly, difficult to audit, and poorly aligned with structured requirement changes such as revised safety rules, new operational constraints, or updated user preferences. We argue for an alternative abstraction: adaptation via edits to an external, human-readable specification that the agent consults at execution time. We propose conditioning decision-making on an editable knowledge graph encoding (i) rules capturing action applicability and high-level effects, (ii) hard constraints defining feasibility, and (iii) soft preferences shaping tradeoffs among feasible behaviors. Requirement changes become graph edits, not policy rewrites, operationalized through constraint-based action shielding and preference-driven objective shaping. This enables immediate, auditable behavior updates with zero or minimal gradient-based adaptation. We outline an edit-based evaluation protocol and highlight open problems in specification design, edit inference, and guarantees.