Objective. Replace a layer’s randomly-initialized weight rows with rows generated by an IIR filter’s impulse response (varying the pole band), to see whether structured, filter-generated weights help or hurt, and whether pairing them with FAct produces any synergy.
| GPU(s) | local (CUDA_VISIBLE_DEVICES=0, host not separately logged) |
| Dataset(s) | FashionMNIST, CIFAR-10 |
| Model | Small MLP with IIR-generated weight rows (bands: b2, b8, b32, Nyquist) |
Result summary
- Wins on FashionMNIST; ties or loses on CIFAR-10 — and with 5–7x more seed-to-seed variance than the dense baseline.
- An LR-search pilot (seed 0) found the best learning rate differs by arm: dense best at 1e-3, most IIR-banded variants best around 3e-3.
- Rank is set by the pole band, not the pole count — a wider band (
iir_bnyq) consistently outperforms a narrower one (iir_b2) at matched parameter count. - Frequencies don’t train (damping does) — and pairing IIR-generated weights with FAct shows no synergy: the FAct-augmented variants (
+factsuffix) don’t outperform their plain IIR counterparts by more than IIR or FAct do alone. - Confirmed elsewhere in this project: FAct aliases in any unnormalized MLP with roughly a dozen periods of preactivation range — normalize first, or the periodic activation sees frequencies it was never fit to.
Insights
- Structured weight generation is a real, if noisy, lever on its own (band matters, count doesn’t) — but it doesn’t combine additively with a structured activation. Two different kinds of “structure” in a network don’t automatically stack.