Learn, Freeze, Transfer: Reusing a Shared Activation Across Networks

Abstract

We investigate whether a learned nonlinearity can be reused independently of the network in which it was optimized. We introduce a learn–freeze–transfer framework in which a globally shared activation is learned jointly with a source network, frozen after training, and transferred to independently initialized target networks without transferring the source-network weights. We instantiate this framework with FAct, a five-parameter activation represented by a two-harmonic truncated Fourier series and initialized using a Fourier approximation of GELU. FAct is learned in a Vision Transformer trained on ImageNet-1K, and its learned coefficients are transferred to four independently initialized architectures: a standard Vision Transformer, a Swin Transformer, a Vision Transformer with linear attention, and an MLP-Mixer. The target networks are evaluated on Fashion-MNIST, CIFAR-10, CIFAR-100, and Food-101. We compare Frozen FAct with ten conventional activation functions. Under a common training configuration, Frozen FAct achieves the highest mean test accuracy among the eleven activation choices in 15 of the 16 architecture–dataset combinations. These findings provide empirical evidence that a globally shared nonlinearity learned within one network can be reused across independently initialized networks without transferring the source-network weights.

1 Introduction

Activation functions determine how intermediate representations are transformed in neural networks and can influence both optimization and predictive performance. Conventional choices such as ReLU^[1], GELU^[2], and SiLU^[3] impose predefined nonlinear transformations, whereas learnable activation functions allow the nonlinearity to adapt jointly with network parameters^{[4, 5, 6]}. However, these approaches are generally evaluated within the networks in which their activation parameters are learned. This raises a related question: can a learned activation retain its usefulness when separated from the network weights with which it was optimized?

The transferability of a learned activation depends in part on how its parameters are represented and shared within a network. Existing approaches may assign activation parameters to individual neurons, channels, or layers, allowing the nonlinear transformation to adapt at different network locations^{[4, 5, 7, 8, 9]}. Global parameter sharing instead uses a common set of parameters across multiple layers, defining a single activation function throughout the network. Prior work has explored this approach using globally shared adaptive slopes^[10] and shared rational parameterizations^[11]. However, sharing an activation within a network and transferring it independently of that network are distinct properties. The former establishes a common nonlinearity, whereas the latter examines whether the learned function remains useful after the source network is discarded.

Recent work has also investigated learned activation functions as reusable components. Teney et al.^[12] employ bilevel meta-learning to discover activation functions suited to different task distributions and examine their transfer across related tasks. We investigate a complementary setting in which a globally shared activation is learned jointly with a single source network through conventional end-to-end training, rather than meta-learned across multiple task episodes. After source training, the learned activation is frozen and transferred to independently initialized target networks without reusing the source-network weights (Figure 1). This setting allows us to examine whether a nonlinearity learned within one network remains useful across different architectures and target tasks.

To investigate this setting, we use FAct (Fourier Activation), a globally shared learnable nonlinearity parameterized by a truncated Fourier series with two harmonics and five learnable coefficients. A single set of coefficients is shared across all MLP blocks of the source network, making the number of activation parameters independent of network depth and width. The coefficients are initialized using a two-harmonic Fourier approximation of GELU and optimized jointly with the source-network weights. This initialization provides a reference shape while allowing the activation function to adapt during source training.

We evaluate the transferability of FAct in two settings. First, we examine same-task transfer by using the source-learned activation in a freshly initialized Vision Transformer^[13] of the same depth, trained on ImageNet-1K^[14]. We then investigate cross-architecture transfer by transferring the same five frozen coefficients to four independently initialized target architectures: a depth-2 Vision Transformer with global self-attention, a Swin Transformer with shifted-window self-attention^[15], a Vision Transformer with linear attention^[16], and an attention-free MLP-Mixer^[17]. In all cases, the target-network weights are trained from scratch, while the transferred activation coefficients remain fixed. The cross-architecture experiments are conducted on four datasets: Fashion-MNIST^[18], CIFAR-10, CIFAR-100^[19], and Food-101^[20]. This evaluation examines whether the source-learned nonlinearity remains useful across different network architectures, attention mechanisms, and target tasks.

Under a common target-training configuration, Frozen FAct achieves the highest mean test accuracy among eleven activation choices in 15 of the 16 architecture–dataset combinations. To examine whether this result depends on the use of common hyperparameters, we further evaluate all activation functions using activation-specific learning-rate and weight-decay tuning under an identical search budget. Following this tuning, Frozen FAct ranks first in 11 combinations, second in four, and fourth in the one remaining combination. These results provide empirical evidence that a globally shared nonlinearity learned within a source network can remain useful across independently initialized architectures and tasks, even when activation-specific hyperparameter optimization is considered.

This work makes three contributions:

1. A learn–freeze–transfer framework for shared activations. We formulate a framework in which a globally shared nonlinearity is learned jointly with a source network and subsequently transferred to independently initialized target networks without transferring source-network weights or updating the activation parameters.
2. A compact Fourier-based activation parameterization. We introduce FAct, a five-parameter activation represented by a two-harmonic truncated Fourier series and shared across the MLP blocks of a network. Its parameter count remains independent of network depth and width.
3. An empirical evaluation of activation transferability. We evaluate the proposed approach through same-task and cross-architecture transfer experiments, using three transformer architectures and an attention-free MLP-Mixer across four image-classification datasets. We compare Frozen FAct against ten conventional activation functions under both common and activation-specific hyperparameter configurations.

2 Background and Related Work

2.1 Learnable and Globally Shared Activation Functions

Learnable activation functions introduce parameterized nonlinear transformations whose parameters are optimized jointly with network weights. Existing approaches differ in both their functional representations and the flexibility they provide. PReLU learns the slope of the negative input region^[4], while ACON introduces a more flexible parametric activation formulation^[8]. Padé activation units represent nonlinear transformations using ratios of polynomials^[5], whereas other approaches employ adaptive piecewise-linear functions and kernel-based representations to provide greater functional flexibility^[6, 7].

Beyond the choice of functional representation, activation parameters can also differ in their sharing granularity. They may be assigned locally to individual neurons or layers, or shared across multiple layers. Jagtap et al.^[10] introduced globally adaptive activation functions using a shared trainable scaling parameter that adjusts the activation slope across the network. More recently, Delfosse et al.^[11] proposed joint-rational activations, sharing a common set of rational-function parameters across layers to regulate activation plasticity in deep reinforcement learning. These approaches demonstrate the utility of sharing activation parameters within a network. However, sharing alone does not establish whether a learned nonlinear function can be separated from its source network and reused in an independently initialized model.

2.2 Transferability of Learned Activation Functions

The transferability of learned activation functions has been investigated by Teney et al.^[12], who proposed a bilevel meta-learning approach to discover activation functions that induce task-appropriate inductive biases. Their method employs episodic training to optimize activation functions for generalization across different task instances. They demonstrate positive transfer across 22 algorithmic tasks and investigate the reuse of learned activations across networks of different widths. This work establishes that learned nonlinearities can serve as reusable components rather than being restricted to the networks in which they are initially learned.

The present study investigates a different learning and transfer regime. Rather than meta-learning activation functions across multiple task episodes, we learn a single globally shared activation jointly with a source Vision Transformer through conventional end-to-end training on ImageNet-1K. After source training, the learned activation is frozen and transferred, without modification, to independently initialized target networks. No source-network weights are reused, and the activation parameters are not updated during target training. We evaluate this approach across transformer architectures with different attention mechanisms and an attention-free MLP-Mixer, using four image-classification datasets. Thus, our focus is on whether an activation learned through conventional source-network training can retain its usefulness when separated from the source weights and reused across independently trained architectures and tasks.

3 FAct: Learning and Transferring a Shared Nonlinearity

3.1 Global Activation Sharing

Let $z_{\ell }$ denote the pre-activation at the $\ell$-th activation site in a network, and let $\varphi (z;\theta )$ denote a parameterized activation function. In FAct, a single parameter vector $\theta$ is shared across all MLP blocks of the network:

where $L$ denotes the number of activation sites and $h_{\ell }$ is the corresponding activation output. The same parameter vector is used at every site, without replication across layers or hidden units.

Because the activation parameters are shared, their gradients aggregate contributions from all activation sites during training:

where $𝒥$ denotes the network training objective. Each term represents the gradient contribution propagated through an individual activation site. Consequently, the shared activation parameters are optimized using signals from all MLP blocks rather than through independently learned, layer-specific parameter sets.

Global sharing defines a single parameterized nonlinear function that can be retained independently of the source-network weights. We refer to FAct while its parameters are jointly optimized with the source-network weights as Learnable FAct. After source training, the parameters are frozen, and the resulting activation is referred to as Frozen FAct.

3.2 Fourier Parameterization and Initialization

We parameterize FAct using a truncated Fourier series:

where $a_0$, $a_k$, and $b_k$ are learnable coefficients, and $K$ denotes the number of harmonics. We fix the angular frequency to $\omega =1$, giving the activation a period of $2\pi$. The parameter vector is defined as

containing $2K+1$ coefficients. In the main experiments, we set $K=2$, resulting in five learnable parameters shared across the entire network. Consequently, the number of activation parameters is independent of network depth and width.

The Fourier representation provides a compact and smooth parameterization whose shape can adapt during training. Depending on the learned coefficients, the resulting function can exhibit monotonic or non-monotonic behaviour over the input range of interest. Furthermore, its output is bounded for any finite coefficient values:

This bound follows from the bounded magnitudes of the sine and cosine functions.

We initialize FAct using a two-harmonic Fourier approximation of GELU^[2]. For a reference activation function $\sigma$, the Fourier coefficients are obtained by projection over one period, $[-\pi ,\pi ]$:

For $\sigma =\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}$, $K=2$, and $\omega =1$, the resulting initialization is

This initialization provides a reference nonlinear transformation while allowing all five coefficients to adapt during source-network training.

3.3 Learn-Freeze-Transfer Protocol

The globally shared parameterization allows the learned activation to be separated from the source network and reused independently. During source training, the source-network weights and shared activation parameters are jointly optimized:

where $W_s$ denotes the source-network weights, $\theta$ represents the shared activation parameters, and ${𝒥}_{\mathrm{s}\mathrm{r}\mathrm{c}}$ is the source-task training objective. The resulting activation is

After source training, the learned activation parameters ${\theta }^{*}$ are frozen. Only these parameters are transferred to a target network; the source-network weights $W_s^{*}$ are not transferred. Let $W_t$ denote the weights of an independently initialized target network, and let ${𝒥}_{\mathrm{t}\mathrm{g}\mathrm{t}}$ denote its training objective. Target training then optimizes

Thus, the target-network weights are optimized for the target task while the transferred activation parameters remain unchanged throughout training. No source-network weights are used to initialize the target model, and target-task gradients do not update the transferred activation.

This protocol isolates the reuse of the learned nonlinearity from the transfer of network weights. It therefore enables us to examine whether an activation learned jointly with one source network remains useful when incorporated into independently initialized networks, without further adaptation of the activation itself. Figure 1 illustrates the three stages of this process.

4 Experimental Evaluation

This section evaluates whether a globally shared activation learned jointly with a source network remains useful when transferred to independently initialized target networks. We first describe the source and target architectures, datasets, and training protocol. We then examine the evolution of the source-learned activation, its transfer to a network with the same architecture and task, and its transfer across four target architectures with different attention and token-mixing mechanisms. Frozen FAct is compared with ten conventional activation functions to assess its performance under a common training configuration.

4.1 Experimental Setup

Source and target networks. We learn FAct in a depth-6 Vision Transformer (ViT-d6) trained on ImageNet-1K^[14]. The source network contains $3,048,232$ parameters, and learning FAct adds five trainable activation coefficients, resulting in $3,048,237$ trainable parameters. The same five coefficients are shared across all MLP blocks of the source network.

After source training, the learned activation is transferred to four independently initialized target architectures: a depth-2 Vision Transformer (ViT-d2) with global self-attention, a depth-2 Swin Transformer (Swin-d2) with shifted-window self-attention^[15], a depth-2 Vision Transformer with linear attention (LinViT-d2)^[16], and a depth-2 MLP-Mixer^[17]. These target networks contain approximately 100K parameters, with their exact configurations provided in Appendix A. No source-network weights are transferred, and the activation coefficients remain fixed throughout target training.

Datasets and transferred activation. Each target architecture is evaluated on Fashion-MNIST^[18], CIFAR-10, CIFAR-100^[19], and Food-101^[20]. All target experiments use the same frozen activation coefficients obtained from the seed-1 source checkpoint: $a_0=0.2182$, $a_1=0.1193$, $a_2=-0.4237$, $b_1=0.8315$, and $b_2=-0.5918$. Thus, the same source-learned activation is used across all target architectures and datasets.

Training and evaluation protocol. All target experiments use five random seeds, with the same seed configurations across architectures, datasets, and activation choices. Models are trained for 100 epochs using AdamW, with a learning rate of ${10}^{-3}$, weight decay of $0.05$, and a batch size of 256. The learning-rate schedule consists of five epochs of linear warm-up followed by cosine decay. Label smoothing and dropout are both set to $0.1$. All activation choices use identical dataset-specific preprocessing and augmentation, without test-time augmentation. The complete preprocessing and augmentation procedures are provided in Appendix A.

Frozen FAct is compared with ten conventional activation functions: ReLU^[1], LeakyReLU^[21], GELU^[2], SiLU^[3], Mish^[22], HardSwish^[23], Softplus^[24], ELU^[25], SELU^[26], and Tanh^[27]. For each run, the checkpoint with the highest validation accuracy is selected and evaluated once on the held-out test set. Results are reported as mean and sample standard deviation across five seeds. Activation-specific learning-rate and weight-decay tuning results are provided in Appendix B.

4.2 Evolution of the Source-Learned Activation

FAct is initialized using a two-harmonic Fourier approximation of GELU, with all five coefficients subsequently optimized jointly with the source-network weights. Figure 2 illustrates the evolution of the shared activation during ImageNet-1K training. The activation progressively departs from its initial GELU approximation and develops a non-monotonic shape over the course of training. The final learned function, obtained after 100 epochs, differs substantially from its initialization and is the activation frozen and transferred to all target networks in the subsequent experiments.

4.3 Same-Task Transfer at Source Scale

To examine whether the learned activation remains useful when transferred to a network with the same architecture and task, we train a freshly initialized ViT-d6 on ImageNet-1K using the frozen FAct activation learned by the source network. The five activation coefficients are obtained from the seed-1 source checkpoint and remain fixed throughout target training. No source-network weights are transferred. We conduct five independent target runs, all using the same frozen activation, so the reported variation reflects differences in target initialization and training rather than variability across independently learned activations.

Table 1. Same-task transfer on ImageNet-1K using a freshly initialized ViT-d6. Results report top-1 accuracy (%) on the ILSVRC validation set as mean $\pm$ sample standard deviation across five target runs. Frozen FAct uses the activation coefficients learned by the source network; source-network weights are not transferred, and activation coefficients remain fixed. $\Delta$ denotes the difference in mean accuracy relative to GELU, in percentage points.

Table 1 compares Frozen FAct with GELU under the same architecture and training configuration. Frozen FAct achieves a mean top-1 accuracy of $63.91\%\pm 0.15$, compared with $62.45\%\pm 0.13$ for GELU, an improvement of $1.46$ percentage points. This result provides evidence that the source-learned activation retains useful properties even when separated from its original network weights and reused in an independently initialized network performing the same task.

4.4 Transfer Across Transformer Architectures

Tables 2–4 report the test accuracy of the eleven activation choices on ViT-d2, Swin-d2, and LinViT-d2, respectively, under the common training configuration. Table 5 summarizes the performance difference between Frozen FAct and the best-performing conventional activation for each target–dataset combination, while Figure 3 illustrates the corresponding accuracy margins.

Table 2. Transfer performance on ViT-d2 with global self-attention under the common training configuration. Results are mean test accuracy (%) $\pm$ sample standard deviation across five seeds. Bold and underlining indicate the highest and second-highest mean accuracies, respectively.

Table 3. Transfer performance on Swin-d2 with shifted-window self-attention under the common training configuration. Results are mean test accuracy (%) $\pm$ sample standard deviation across five seeds. Bold and underlining indicate the highest and second-highest mean accuracies, respectively.

Table 4. Transfer performance on LinViT-d2 with linear attention under the common training configuration. Results are mean test accuracy (%) $\pm$ sample standard deviation across five seeds. Bold and underlining indicate the highest and second-highest mean accuracies, respectively.

Table 5. Frozen FAct compared with the best-performing conventional activation across the twelve transformer architecture–dataset combinations under the common training configuration. The best conventional activation is selected from the ten baseline functions, excluding Frozen FAct. Accuracy values are mean test accuracy (%). $\Delta$ denotes the seed-matched mean accuracy difference between Frozen FAct and the selected conventional activation, in percentage points, across five seeds.

Across the three transformer architectures and four datasets, Frozen FAct achieves the highest mean test accuracy in all twelve architecture–dataset combinations. Its improvement over the best-performing conventional activation ranges from $0.38$ percentage points on LinViT-d2 with Fashion-MNIST to $1.71$ percentage points on ViT-d2 with Food-101. The strongest conventional baseline varies across architectures: ReLU or LeakyReLU provides the highest mean accuracy among conventional activations for ViT-d2 and LinViT-d2, whereas GELU performs best across all four Swin-d2 datasets. Importantly, the same five activation coefficients learned by the source ViT-d6 are used in all target experiments, without transferring source-network weights or updating the activation during target training. These results provide empirical evidence that a globally shared nonlinearity learned within one network can retain its usefulness across independently initialized transformer architectures with different attention configurations and target tasks.

Additional analyses examine transfer using a globally shared Padé activation (Appendix C) and investigate the effect of the GELU-based initialization and of target-network depth in Swin targets (Appendix D).

4.5 Transfer to an MLP-Mixer Architecture

To examine whether the learned activation transfers beyond attention-based architectures, we evaluate Frozen FAct in an independently initialized depth-2 MLP-Mixer^[17]. Unlike the transformer architectures, MLP-Mixer uses separate MLPs for token mixing and channel mixing, without self-attention. We transfer the same five source-learned activation coefficients to both types of MLP blocks and keep them fixed throughout target training. The comparison includes the same ten conventional activation functions under the common training configuration.

Table 6. Transfer performance on MLP-Mixer under the common training configuration. Mean test accuracy (%) $\pm$ sample s.d. across five seeds. Bold indicates the highest mean test accuracy for each dataset, and underlining indicates the second-highest. The comparison includes Frozen FAct and the same ten conventional activation functions as Tables 2–4.

Frozen FAct achieves the highest mean test accuracy on Fashion-MNIST, CIFAR-10, and CIFAR-100, outperforming the best-performing conventional activation by $0.62$, $0.48$, and $0.29$ percentage points, respectively (Table 6). On Food-101, GELU performs better by approximately $1.05$ percentage points. These results provide evidence that a source-learned activation can also be reused in an attention-free architecture without transferring source-network weights or updating its coefficients, although its performance advantage varies across datasets. After activation-specific hyperparameter tuning, Frozen FAct retains its advantage on Fashion-MNIST and CIFAR-10, whereas HardSwish performs better on CIFAR-100 and Food-101.

5 Conclusion

This work investigated whether a globally shared activation learned jointly with a source network can be reused independently of the network weights with which it was optimized. We introduced a learn–freeze–transfer framework and instantiated it using FAct, a five-parameter activation represented by a two-harmonic truncated Fourier series. Initialized using a Fourier approximation of GELU, its coefficients are optimized jointly with a source Vision Transformer and subsequently frozen for transfer to independently initialized target networks.

We evaluated the proposed approach through same-task and cross-architecture transfer experiments. In the same-task setting, Frozen FAct improved ImageNet-1K top-1 accuracy by $1.46$ percentage points over GELU when used in a freshly initialized ViT-d6. Across four target architectures and four image-classification datasets, Frozen FAct achieved the highest mean test accuracy among eleven activation choices in 15 of the 16 architecture–dataset combinations under a common training configuration. This included all twelve combinations involving the three transformer architectures and three of four MLP-Mixer combinations. Following activation-specific hyperparameter tuning, Frozen FAct ranked first in eleven combinations, second in four, and fourth in the remaining combination. Throughout these experiments, only the five learned activation coefficients were transferred; source-network weights were not reused, and the activation remained fixed during target training.

These findings provide empirical evidence that a globally shared nonlinearity learned within one network can retain its usefulness when transferred to independently initialized networks with different architectures and target tasks. However, the evaluation uses a single source-learned activation obtained from one Vision Transformer trained on ImageNet-1K, and the cross-architecture experiments are restricted to relatively small target networks and image-classification tasks. Further investigation is needed to establish how activation transferability varies across source architectures, source training datasets, model scales, and target tasks. Extending this evaluation to convolutional and other network architectures would help determine the broader applicability of transferable learned nonlinearities.

Appendix A: Experimental Configuration

A.1 Models and datasets

The source model is a depth-6 Vision Transformer (ViT-d6) trained on ImageNet-1K, with FAct shared across its six MLP blocks. The target models comprise three depth-2 transformer architectures and a depth-2 MLP-Mixer. ViT-d2 uses global softmax self-attention, Swin-d2 employs shifted-window self-attention with a window size of four, and LinViT-d2 replaces softmax attention with linear attention using $elu(x)+1$. The MLP-Mixer consists of two blocks, each containing a pre-normalized residual token-mixing MLP (hidden width 38) followed by a pre-normalized residual channel-mixing MLP (hidden width 304), with an embedding width of 76. It does not use attention, a class token, or positional embeddings, and classification is performed using globally average-pooled tokens.

All MLPs use the same Linear–activation–dropout–Linear structure, with a single activation module shared across the MLPs within each network. Table 7 summarizes the architectural configurations and parameter counts.

The target models are trained on Fashion-MNIST ($28\times 28$), CIFAR-10 and CIFAR-100 ($32\times 32$), and Food-101 (resized to $64\times 64$). For each dataset, $5,000$ training images are held out once using a fixed random seed to form a validation split shared across all runs. The official test sets, comprising $10,000$ images for Fashion-MNIST and the CIFAR datasets and $25,250$ images for Food-101, are reserved for final evaluation.

A.2 Training configuration

All target networks are trained for 100 epochs with AdamW, a batch size of 256, a five-epoch linear warm-up followed by cosine decay, label smoothing of $0.1$ and dropout of $0.1$. The common configuration uses a learning rate of ${10}^{-3}$ and weight decay of $0.05$ for every activation; activation-specific tuning is described in Appendix B. The source model uses the same optimizer, schedule and regularization. The transferred coefficients are those of the seed-1 source run (Section 4.1 and Figure 2).

A.3 Preprocessing and evaluation

Inputs are normalized per channel with the training-split mean and standard deviation. During training, each image is zero-padded by 2 pixels (Fashion-MNIST) or 4 pixels (other datasets), randomly cropped back to its original size and flipped horizontally with probability $0.5$; validation and test images are only normalized. For each run, the checkpoint with the highest validation accuracy is evaluated once on the test set. Each result is the mean and sample standard deviation over five seeds, using the same seeds for every activation.

Table 7. Architectural and training configurations of the source and target networks. Parameter ranges reflect differences across the four target datasets.

Appendix B: Activation-Specific Hyperparameter Tuning

The main experiments use a common learning rate and weight decay for all activation functions. To examine whether this configuration favours Frozen FAct, we perform activation-specific hyperparameter tuning using an identical search budget. The learning rate and weight decay are selected independently for each activation–architecture–dataset combination. The search considers five learning rates ($3\times {10}^{-4}$, ${10}^{-3}$, $3\times {10}^{-3}$, ${10}^{-2}$, and $3\times {10}^{-2}$) and three weight-decay values ($0$, $0.05$, and $0.2$), resulting in fifteen configurations per combination. Each configuration is evaluated using a 30-epoch, single-seed proxy experiment. The selected configuration is subsequently trained for 100 epochs across five random seeds. This procedure is applied to all eleven activation choices across the four target architectures and four datasets.

Table 8. Frozen FAct compared with the best-performing conventional activation under activation-specific hyperparameter tuning. Results are mean test accuracy (%) $\pm$ sample standard deviation across five seeds for both Frozen FAct and the best-performing conventional activation. $\Delta$ denotes the difference in mean accuracy between Frozen FAct and the selected conventional activation, in percentage points.

Table 8 summarizes the comparison between Frozen FAct and the best-performing conventional activation after activation-specific hyperparameter tuning. Frozen FAct retains the highest mean accuracy in eleven of the sixteen architecture–dataset combinations, ranks second in four, and ranks fourth in the remaining combination.

Across the transformer architectures, Frozen FAct ranks first in nine of the twelve combinations. GELU achieves the highest mean accuracy in the remaining three, displacing Frozen FAct to second place. For the MLP-Mixer, Frozen FAct retains the highest rank on Fashion-MNIST and CIFAR-10, whereas HardSwish performs better on CIFAR-100 and Food-101. Figure 4 illustrates the changes in activation rankings between the common and tuned configurations.

These results show that the relative performance of activation functions depends partly on target architecture and dataset. Nevertheless, Frozen FAct retains the highest mean accuracy in most evaluated combinations, even when conventional activation functions receive activation-specific hyperparameter optimization.

Appendix C: Transfer with a Shared Rational Activation

To examine whether activation transfer is specific to the Fourier parameterization, we use a globally shared Padé Activation Unit (PAU)^[5], represented by a degree-$(5,4)$ rational function $\varphi (t)=P(t)/Q(t)$ with ten learnable coefficients shared across the six MLP blocks of ViT-d6. After source training on ImageNet-1K, the learned coefficients are frozen and transferred to an independently initialized ViT-d2 using the same transfer protocol as Frozen FAct. Only the activation coefficients are transferred; no source-network weights are reused.

Table 9. Transfer performance of a globally shared Padé activation on ViT-d2 under the common training configuration. Results are mean test accuracy (%) $\pm$ sample standard deviation across six seeds. Frozen PAU, Frozen FAct, and GELU are evaluated in separate matched comparisons, with all three conditions for each dataset run on the same hardware. Values for Frozen FAct and GELU may therefore differ slightly from those reported in Table 2. Bold indicates the highest mean accuracy for each dataset.

Both globally shared learned parameterizations achieve higher mean accuracy than GELU across all four datasets (Table 9). Frozen PAU also achieves higher mean accuracy than Frozen FAct, with improvements ranging from $0.15$ to $0.87$ percentage points. These results provide supplementary evidence that activation transfer is not restricted to the Fourier representation. However, this comparison is limited to one target architecture and does not establish equivalent transfer performance across other architectures.

Appendix D: Depth Sensitivity and the Effect of GELU-Based Initialization

We conduct an additional experiment to investigate two aspects of activation transfer: the effect of using a fixed Fourier approximation of GELU instead of the conventional GELU activation, and the sensitivity of transfer performance to target-network depth. For this analysis, we use Swin Transformer targets because GELU is the highest-performing conventional activation on all four datasets in the main Swin experiments (Table 3). This provides a single, consistent reference for evaluating the two frozen Fourier-based activations across datasets. The conclusions of this appendix therefore apply to the evaluated Swin targets and are not claimed for all architectures.

D.1 Experimental Design

We vary the depth of the target Swin Transformer from one to six blocks while keeping the remaining architectural and training configurations fixed. All blocks belong to a single stage with embedding dimension 64, four attention heads, MLP ratio 4, patch size 4 and window size 4. Blocks alternate between regular-window and shifted-window attention, beginning with a regular-window block followed by a shifted-window block with a shift of two tokens, so the depth-one model contains only a regular-window block. No patch merging is applied, so the token grid is identical at every depth and only the number of blocks, and hence the number of activation call sites, changes.

Three activation configurations are compared: (1) GELU, used as the conventional reference; (2) Frozen GELU-init, the two-harmonic Fourier approximation of GELU initialized as described in Eq. 7, with its coefficients kept fixed throughout target training; and (3) Frozen FAct, which uses the coefficients learned by the ImageNet-1K source ViT and transferred without further updates. In both frozen configurations, no source-network weights are transferred. All target networks are independently initialized and trained with five seeds using the common training configuration described in Appendix A. For each dataset and depth, all three configurations were trained on the same GPU type.

Table 10. Test accuracy across Swin Transformer target depths. Results are mean accuracy (%) $\pm$ sample standard deviation across five seeds. The three configurations are GELU, Frozen GELU-init and Frozen FAct. Bold indicates the highest mean accuracy at each dataset–depth combination.

D.2 Results

Effect of the GELU-Based Fourier Initialization

We first compare Frozen GELU-init with GELU to examine the effect of replacing the conventional activation with its fixed two-harmonic Fourier approximation (Table 10). This approximation does not provide a consistent advantage over GELU across datasets and target depths. On Fashion-MNIST, Frozen GELU-init achieves higher mean accuracy than GELU at all six depths, by $0.13$ to $0.49\mathrm{p}\mathrm{p}$. On CIFAR-10, it performs better at depths one, five and six, and worse at depths two to four. On CIFAR-100, it is at or below GELU at every depth, and on Food-101 GELU achieves higher mean accuracy at every depth. The performance of the fixed Fourier approximation therefore depends on the target dataset and depth rather than providing a uniform improvement over GELU.

Sensitivity to Target-Network Depth

We next examine how the performance of the transferred activation changes with target-network depth (Table 10 and Figure 5). Frozen FAct achieves the highest mean accuracy among the three configurations at every depth on Fashion-MNIST, CIFAR-10 and CIFAR-100, and at five of six depths on Food-101. On Fashion-MNIST and CIFAR-10 it outperforms GELU at every depth, by $0.92$ to $1.26\mathrm{p}\mathrm{p}$ and $1.48$ to $3.07\mathrm{p}\mathrm{p}$ respectively, and is ahead in all five seed-matched runs in each case. On CIFAR-100 and Food-101 the difference is negligible at depth one ($+0.08$ and $-0.19\mathrm{p}\mathrm{p}$), but Frozen FAct is ahead from depth two onwards, reaching $1.94\mathrm{p}\mathrm{p}$ on CIFAR-100 and $2.64\mathrm{p}\mathrm{p}$ on Food-101, with Frozen FAct ahead in all five seed-matched runs at every depth from two to six. On CIFAR-10, CIFAR-100 and Food-101 the advantage over GELU tends to increase with depth.

Importantly, Frozen FAct also outperforms Frozen GELU-init in all 24 dataset–depth combinations, by $0.09$ to $3.94\mathrm{p}\mathrm{p}$. Together with the previous comparison, this indicates that the advantage of Frozen FAct is not explained by replacing GELU with a fixed Fourier representation: the source-learned coefficients provide an additional benefit beyond the GELU-based initialization under the evaluated settings. Overall, the transferred activation remains useful across a range of Swin target depths, although the magnitude of its advantage depends on the dataset and depth.

References

1. [1]V. Nair and G. E. Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. International Conference on Machine Learning (ICML), pp. 807–814, 2010.
2. [2]D. Hendrycks and K. Gimpel. Gaussian Error Linear Units (GELUs). arXiv:1606.08415, 2016.
3. [3]S. Elfwing, E. Uchibe and K. Doya. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning. Neural Networks, 107, pp. 3–11, 2018.
4. [4]K. He, X. Zhang, S. Ren and J. Sun. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. IEEE International Conference on Computer Vision (ICCV), pp. 1026–1034, 2015.
5. [5]A. Molina, P. Schramowski and K. Kersting. Padé Activation Units: End-to-End Learning of Flexible Activation Functions in Deep Networks. International Conference on Learning Representations (ICLR), 2020.
6. [6]S. Scardapane, S. Van Vaerenbergh, S. Totaro and A. Uncini. Kafnets: Kernel-Based Non-Parametric Activation Functions for Neural Networks. Neural Networks, 110, pp. 19–32, 2019.
7. [7]F. Agostinelli, M. Hoffman, P. Sadowski and P. Baldi. Learning Activation Functions to Improve Deep Neural Networks. ICLR Workshop, 2015.
8. [8]N. Ma, X. Zhang, M. Liu and J. Sun. Activate or Not: Learning Customized Activation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8032–8042, 2021.
9. [9]A. D. Jagtap, K. Kawaguchi and G. E. Karniadakis. Locally Adaptive Activation Functions with Slope Recovery for Deep and Physics-Informed Neural Networks. Proceedings of the Royal Society A, 476(2239), pp. 20200334, 2020.
10. [10]A. D. Jagtap, K. Kawaguchi and G. E. Karniadakis. Adaptive Activation Functions Accelerate Convergence in Deep and Physics-Informed Neural Networks. Journal of Computational Physics, 404, pp. 109136, 2020.
11. [11]Q. Delfosse, P. Schramowski, M. Mundt, A. Molina and K. Kersting. Adaptive Rational Activations to Boost Deep Reinforcement Learning. International Conference on Learning Representations (ICLR), Earlier version: Recurrent Rational Networks, arXiv:2102.09407, 2024.
12. [12]D. Teney, L. Jiang, F. Gogianu and E. Abbasnejad. Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
13. [13]A. Dosovitskiy et al. An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations (ICLR), 2021.
14. [14]J. Deng, W. Dong, R. Socher, L. Li, K. Li and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255, 2009.
15. [15]Z. Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012–10022, 2021.
16. [16]A. Katharopoulos, A. Vyas, N. Pappas and F. Fleuret. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. Proceedings of the 37th International Conference on Machine Learning, 119, pp. 5156–5165, 2020.
17. [17]I. Tolstikhin et al. MLP-Mixer: An all-MLP Architecture for Vision. Advances in Neural Information Processing Systems, 34, pp. 24261–24272, 2021.
18. [18]H. Xiao, K. Rasul and R. Vollgraf. Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv:1708.07747, 2017.
19. [19]A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. University of Toronto, 2009.
20. [20]L. Bossard, M. Guillaumin and L. Van Gool. Food-101 – Mining Discriminative Components with Random Forests. European Conference on Computer Vision (ECCV), pp. 446–461, 2014.
21. [21]A. L. Maas, A. Y. Hannun and A. Y. Ng. Rectifier Nonlinearities Improve Neural Network Acoustic Models. ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
22. [22]D. Misra. Mish: A Self Regularized Non-Monotonic Activation Function. British Machine Vision Conference (BMVC), arXiv:1908.08681, 2020.
23. [23]A. Howard et al. Searching for MobileNetV3. IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1314–1324, 2019.
24. [24]C. Dugas, Y. Bengio, F. Bélisle, C. Nadeau and R. Garcia. Incorporating Second-Order Functional Knowledge for Better Option Pricing. Advances in Neural Information Processing Systems (NIPS), pp. 472–478, 2001.
25. [25]D. Clevert, T. Unterthiner and S. Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). International Conference on Learning Representations (ICLR), 2016.
26. [26]G. Klambauer, T. Unterthiner, A. Mayr and S. Hochreiter. Self-Normalizing Neural Networks. Advances in Neural Information Processing Systems (NeurIPS), 2017.
27. [27]Y. LeCun, L. Bottou, G. B. Orr and K. Müller. Efficient BackProp. Neural Networks: Tricks of the Trade, 1524, pp. 9–50, 1998.