Squared Swish

Swish(Prajit Ramachandran et al.,2017) is a great discovery that effectively alleviates gradientvanishing. Compared to ReLU(Xavier Glorot et al.,2011) piecewise linear structure, Swish curve ismore continuous and can better fit complex function mappings, especially when dealing withhigh-dimensional nonlinear problems. Compared to ReLU "hard truncation" (negative input is directlyset to 0), Swish's suppression of negative signals is softer, reducing information loss. However, underthe same training configuration, Swish is convergence stability is slightly lower than ReLU. SinceSwish performs so well, is there any variant of Swish that performs better than it. So I began to explore.After extensive experimental testing, I discovered Swish's variant Squared Swish, whose formula It is f(x)=x * sigmoid (x) * sigmoid (x), which performs better than Swish in some models.