Proxy sweep — 30 epochs, seed 0, for each of 12 activations across a
5×3 grid of lr × weight‑decay.
Every activation gets the identical 15-point grid, so the tuning budget is
exactly comparable. Cell color is normalized within each panel to
show that activation's own sensitivity to the two hyperparameters —
compare absolute strength using the rank order and the printed accuracy,
not color across panels.
Extended 2026-08-27. The first pass used a 3×3 grid
(lr ≤ 3e‑3) that railed against its ceiling — 47
of 48 (dataset, activation) cells picked the top lr, val_acc
still rising. The two right-hand columns (lr = 1e‑2,
3e‑2) were added to bracket each activation's true optimum. The
top-5 ranking is unchanged on all four datasets, and
FFAct / EFAct stay 1–2 everywhere. On Fashion‑MNIST, CIFAR‑10
and CIFAR‑100 the winning cells barely move (3e‑3 stays optimal
or tied); Food‑101 is different — its real optimum is
lr = 1e‑2 for every top activation, so the old ceiling
genuinely under-served it (FFAct val 0.298 → 0.306).