LiteMish: A Computationally Efficient and Smooth Algebraic Alternative to Mish
ABSTRACT The energy efficiency, inference speed, and learning capacity of deep learning models are significantly influenced by the types of activation functions employed. Although the Rectified Linear Unit (ReLU) is computationally efficient, it proves unsuitable for regressing inherently smooth data due to its non-differentiability at zero. Furthermore, it’s vulnerable to the dying ReLU problem. Conversely, state-of-the-art activation functions, such as Mish, often suffer from high computational complexity. To address these challenges, this research introduces a smooth algebraic activation function that relies solely on fundamental arithmetic operations: addition and multiplication. Designed as a piecewise and parametric function, its parameters are optimized to closely approximate Mish while ensuring continuous differentiability and other desirable characteristics. The proposed activation function was evaluated on a regression task, as well as on the MNIST and CIFAR-100 classification datasets. Its computational efficiency was experimentally assessed using an ESP32 development board. The results indicate that the proposed activation function achieves comparable or even superior learning performance compared to modern activation functions, while being less computationally expensive. This makes it suitable for deep learning applications in scenarios with constrained energy and/or computational resources.