SymbolicLight: A Neuro-Symbolic Spiking Architecture for Language Modeling with Sparse TCAM and Bayesian Decoding

We introduce SymbolicLight, a novel language model architecture that replaces the dense self-attention and softmax decoding of standard Transformers with three biologically-inspired mechanisms: (1) a Sparse TCAM (Ternary Content-Addressable Memory) module that performs spike-gated associative lookup in place of quadratic-cost attention, (2) a Leaky Integrate-and-Fire (LIF) spike encoder that converts token embeddings into sparse binary event streams with ~90% sparsity, and (3) a Bayesian decoding head that integrates learned prior knowledge with contextual evidence for token prediction. SymbolicLight further incorporates an entropy-gated early exit mechanism and a Spike-Timing-Dependent Plasticity (STDP) module for inference-time local learning. We train a 168M-parameter SymbolicLight model on TinyStories and conduct extensive ablation studies against a parameter-matched GPT-2 baseline. While the current implementation does not yet surpass Transformer baselines in perplexity (Val PPL 7.58 vs. 4.12 for GPT-2), the architecture achieves 87–91% activation sparsity—implying that fewer than 13% of neurons fire at any given time step—demonstrating a fundamentally different and potentially far more energy-efficient computational paradigm for language modeling. Ablation experiments reveal that the Bayesian head provides consistent gains, while identifying specific improvement directions for the TCAM aggregation and spike encoding that inform future versions.