DL0189 KAN vs MLP: Kolmogorov-Arnold Networks

An MLP puts learnable scalars on the edges and a fixed nonlinearity on the nodes: every edge is one number in a weight matrix, and every hidden unit applies the same hand-chosen . A KAN swaps those two roles. Every edge carries its own learnable univariate function , implemented as a B-spline of grid size and order plus a SiLU residual branch, and every node does nothing but sum its incoming edge outputs. Both families are universal approximators, so the difference is not what can be represented but where the capacity sits and what that placement costs. Moving capacity onto the edges buys a per-edge object you can plot, sparsify, prune, and even snap to a symbolic formula, and it buys a fast spline approximation rate on smooth low-dimensional targets. It costs a factor of about in parameters at the same layer shape, roughly an order of magnitude in wall-clock training time because per-edge splines do not collapse into one dense GEMM, and a new hyperparameter (the spline grid) that must cover the actual range of the activations.

(1) Role Swap, Not A New Theorem: the Kolmogorov-Arnold representation theorem motivates the design, but the depth-2 form it guarantees can require pathological inner functions, so KANs generalize it to arbitrary depth and width and rely on smoothness of the target, not on the theorem, for their advantage.
(2) Edge Function Is Spline Plus Residual: each is a learned combination of basis functions added to a scaled SiLU, so a KAN is not activation-free; the fixed nonlinearity survives as a residual path that keeps gradients alive outside the grid.
(3) Nodes Are Pure Summation: no elementwise nonlinearity between layers, which is exactly why the univariate curves are individually meaningful.
(4) Parameter Cost Multiplies: a layer holds 4,160 parameters as an MLP and about 32,768 spline coefficients as a KAN with .
(5) Interpretability Is A Workflow: L1 plus entropy regularization on the edge functions, pruning of dead edges, then symbolic snapping of each surviving curve to a candidate like , , or .
(6) Scaling Is The Weak Point: per-edge spline evaluation is memory-bound and GEMM-unfriendly, and at matched parameters and FLOPs an MLP still wins on vision, language, and audio benchmarks.

Figure 1: The role swap in its smallest form. In the MLP the edge is a single learnable number and the nonlinearity is a fixed baked into the node; in the KAN the edge is a learnable spline over a knot grid and the node only adds. Capacity moves from a matrix of scalars to a grid of spline coefficients, which is why the same layer shape costs about times more parameters.

Expressivity behaves differently in the two regimes that matter in practice. On smooth, low-dimensional, compositional targets (symbolic regression, ODE and PDE solution operators, small physical laws) the spline basis is close to the right basis, so error falls quickly with parameters and grid extension lets you refine an already trained model by re-fitting a finer grid instead of restarting. On high-dimensional perception data the picture inverts: a controlled comparison at matched parameters and FLOPs found MLPs ahead on machine-vision, language, and audio tasks, with KANs winning only on symbolic formula representation, and KANs forgetting more than MLPs in a standard class-incremental setting. Interpretability is the more robust claim. Because a node only sums, each edge curve is a genuine one-dimensional function of one variable, so you can plot all of them, drive most toward zero with sparsity penalties, prune the graph down to a handful of edges, and read off a formula. That workflow is what made KANs useful as a scientific assistant rather than as a general drop-in replacement for a dense layer.

Mathematical Formulation:

Where:

• is the input vector and the -th activation of layer , obtained by summation only in a KAN.
• and are the MLP’s learnable weight matrices and its fixed elementwise nonlinearity.
• and are the inner and outer univariate functions of the Kolmogorov-Arnold representation, with the input dimension; is the learnable edge function from unit of layer to unit of layer .
• are B-spline basis functions with local support over the knot grid, their learned coefficients, and the scales of the SiLU residual and the spline branch.
• is the number of grid intervals and the spline order, so each edge holds coefficients; typical values are .
• is depth, the layer width, and the parameter count, so the KAN pays the extra factor at identical layer shape.
• is test error against parameter count ; the rate is the cubic-spline approximation rate and holds only when the target is smooth and effectively low-dimensional.

Figure 2: Cost against payoff. At matched layer shape a KAN with carries about 8x the parameters, and at matched parameter count it trains roughly 10x slower because each edge evaluates its own spline instead of joining one dense matmul. The right panel shows the regime where that price buys something: on a smooth low-dimensional target, grid extension (3 → 5 → 10 → 20) walks the same trained model down a steep spline-approximation curve that an equally sized MLP does not follow.