Stochastic Barrier Attention: Normalization-Free Softmax Sampling via Thermodynamic Race Processes (v5.1)

Read full text Summarise with VIRAT
Authors: Michael Bieg, Claude Fable 5.2
Cited by: 0
References: 3

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Version 5.1 corrects: (i) the simulation throughput of the stress test (N = 10,000), which was a timing artefact of asynchronous GPU execution; measured with device synchronization it is 1.57 x 10^9 race-elements per second on one AMD Radeon 8060S, a property of the simulation, not of the proposed hardware; (ii) the validation of the sampling law, which now applies one test rule, fixed before the final run on the same random draws, to all 25 parameter configurations: double-precision expected counts, cells with expected count < 5 merged, Chi-squared test if at least three cells remain, otherwise an exact two-sided binomial test of the most probable element; pass if p > 0.01 and KL < 5e-3. This replaces an unpooled Chi-squared test, which is miscalibrated when many expected counts are small, and the exclusion of two configurations. All 25 configurations pass (19 Chi-squared, 6 binomial; smallest p = 0.029); (iii) the convergence constant C is defined for the relative error. The corrected validation script (supplementary_validation_gpu_v5.1.py), its full log, and supplementary_c_range.py (distribution of C over 2000 draws) accompany this version. Version 5 cited Adeyeye et al. (Physical Review Applied 23, 044047, 2025) for the superparamagnetic-tunnel-junction weighted random sampler with first-arrival readout, Hübschle-Schneider and Sanders (ACM TOMS 48, 2022) for the exponential-clocks method, and Lee et al. (ICML 2026, arXiv:2605.01910) for the independent software development of the k-sample estimator, which is standard Monte Carlo. The index of the first switching event in an array of independent stochastic barrier devices with rates proportional to exp(beta z_i) is an exact sample from Softmax(beta z), so the normalization constant is never computed explicitly. This paper couples the sMTJ sampler proposed by Adeyeye et al. to Transformer attention: logits from an analog crossbar set the switching rates. Each sample takes O(1) time with respect to sequence length N, but attention requires a weighted sum over values; averaging k samples (standard Monte Carlo) gives O(1/sqrt(k)) error, so the approach is favorable only for hard attention (k = 1), peaked or sparse attention (k « N), and sampling-native uses such as generation. The contribution is this coupling, the latency and regime analysis, a three-layer mixed-signal layout (memristor crossbar, sMTJ sampling layer, value retrieval), and GPU simulations with 3 x 10^6 trials per configuration and N up to 10,000 that check the sampling law (all 25 parameter configurations pass, p > 0.01, under one pooled Chi-squared/binomial test rule; for N = 2 at beta = 10 the test has almost no power), the latency scaling and the 1/sqrt(k) convergence. Device-level effects are not modelled.

Fields of Study & Subjects

Parallel Computing and Optimization Techniques Markov Chains and Monte Carlo Methods Advanced Memory and Neural Computing

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
27 March 2026
Language
EN
Document type
Preprint