Steklov Activations: Piecewise-Polynomial Gates with Compact Support and Tunable Sparsity

Read full text Summarise with VIRAT
Authors: Aleksandr Masalskikh
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Steklov Activations presents a family of compact-support piecewise-polynomial activation functions derived from Steklov kernels in approximation theory. Unlike standard smooth activations such as GELU or SiLU, Steklov activations have finite support in their gating function: outside a controllable transition region, neurons are exactly inactive or fully linear. This gives the family a distinctive property not present in common dense activations: a tunable mechanism for exact neuron inactivity. The paper shows that the family includes HardSwish exactly and can closely approximate GELU, while introducing a scale parameter that controls the tradeoff between smoothness, selectivity, and sparsity. It studies these activations across image classification and language modeling, including GPT-2 and a small LLaMA-style decoder, and analyzes their behavior in terms of performance, inactivity patterns, pruning, and inference efficiency.

Fields of Study & Subjects

Neural Networks and Applications Advanced Neural Network Applications Machine Learning and ELM

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
26 March 2026
Language
EN
Document type
Preprint