Approximating GELU with a truncated Fourier series

A Fourier activation (FAct) writes the nonlinearity as φ(t) = a0 + ∑k=1..K [ ak cos(kωt) + bk sin(kωt) ]. Before training begins the coefficients are not random: they are the exact Fourier coefficients of GELU on the interval [−π/ω, π/ω], obtained by numerical integration of the Euler formulas. Move the two sliders to see how the number of harmonics K and the base frequency ω decide how faithful that starting approximation is.

Each harmonic adds two learnable numbers, ak and bk. The whole activation has 1 + 2K parameters.
ω fixes the fitting half-width L = π/ω. Lowering ω widens the window, so the same K harmonics must cover more ground.
GELU Fourier approximation φ(t) fitting window [−L, L] pointwise error φ(t) − GELU(t)
Half-width L = π/ω
3.142
Parameters 1 + 2K
5
RMS error on [−L, L]
0.4463
RMS error on [−2, 2]
0.1806
Max error on [−2, 2]
0.3212

Fitted coefficients (the values the network is initialised with):

kak (cos)bk (sin)