Topological Transformer: A Redesign for Domain-Memory and cheaper kernel operations

Starting from previous research on topological spaces for vector search (spectral search with taumode), I introduce in this paper the definition, concept implementation and testing of a new class of Transformers focused on boosting the attention mechanism. The Topological Transformer, or Tauformer, applies all the features from current attention-based transformers (taking as baseline nanoGPT) and redesigns the attention mechanism at its core to: (i) allow delivering domain-specic context at the level of the attention mechanism for more domain-relevant token generation, (ii) improve time per token by ∼ 20% and provide savings in KV-cache memory size by ∼ 50% and (iii) set a potential pathway to improve Transformers performance on larger context windows with longer prompt lengths and high-dimensional embeddings; this is made possible by substituting inner-product with taumode's synthetic index-based distance, aiming to provide relevant linear gains in training and generation compared to current GPTs.