ReLU-KAN: New Kolmogorov-Arnold Networks that Only Need Matrix Addition, Dot Multiplication, and ReLU
Qi Qiu, Tao Zhu, Helin Gong, Liming Chen, Huansheng Ning
Abstract
The Kolmogorov-Arnold Network (KAN), as a recent and promising alternative to the Multilayer Perceptron (MLP), has garnered significant attention in academia. The B-spline function within KAN plays a crucial role in function approximation. However, the complexity of B-spline function limits the training efficiency of KAN on GPUs, and their inflexibility restricts KAN’s approximation capacity. We propose a novel KAN-like architecture named ReLU-KAN, which replaces the B-spline in KAN with a simpler function R composed solely of matrix addition, dot product, and ReLU activation. This design enables efficient GPU parallelization for faster backpropagation compared to the KAN. Additionally, the R function incorporates two trainable hyperparameters, allowing it to dynamically adapt its shape and capture intricate relationships within the data. This adaptive capability demonstrates a significant accuracy improvement. In the experiment, the new architecture improves 1-3 orders of magnitude compared with the original scheme. Notably, ReLU-KAN retains the original model’s ability to avoid catastrophic forgetting.