Modules

ReLU Variations

class torch_activation.relus.CReLU(dim: int = 0)

Applies the Concatenated Rectified Linear Unit activation function.

\(\text{CReLU}(x) = \text{ReLU}(x) \oplus \text{ReLU}(-x)\)

See: https://doi.org/10.48550/arXiv.1603.05201

Parameters:

• dim (int, optional) – Dimension along which to concatenate in the output tensor. Default: 1

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*, C, *)\) where \(*\) means any number of additional dimensions

• Output: \((*, 2C, *)\)

Examples:

>>> m = torch_activation.CReLU()
>>> x = torch.randn(2, 3)
>>> output = m(x)

>>> m = torch_activation.CReLU(inplace=True)
>>> x = torch.randn(2, 3, 4)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.relus.ReLUN(n: float = 1.0, inplace: bool = False)

Applies the element-wise function:

\(\text{ReLUN}(x) = \min(\text{ReLU}(x), n)\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• n (float, optional) – Upper bound for the function’s output. Default is 1.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = torch_activation.ReLUN(n=6.0) # ReLU6
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = torch_activation.ReLUN(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.relus.ShiLU(alpha: float = 1.0, beta: float = 0.0, inplace: bool = False)

Applies the ShiLU activation function:

\(\text{ShiLU}(x) = \alpha \cdot \text{ReLU}(x) + \beta\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• alpha (float, optional) – Scaling factor for the positive part of the input. Default: 1.0.

• beta (float, optional) – Bias term added to the output. Default: 0.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = torch_activation.ShiLU(alpha=2.0, beta=1.0)
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = torch_activation.ShiLU(inplace=True)
>>> x = torch.randn(2, 3, 4)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.relus.SquaredReLU(inplace: bool = False)

Applies the element-wise function:

\(\text{SquaredReLU}(x) = \text{ReLU}(x)^2\)

Parameters:

inplace – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = torch_activation.SquaredReLU()
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = torch_activation.SquaredReLU(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.relus.StarReLU(s: float = 0.8944, b: float = -0.4472, learnable: bool = True, inplace: bool = False)

Applies the element-wise function:

\(\text{StarReLU}(x) = s \cdot \text{ReLU}(x)^2 + b\)

See: https://doi.org/10.48550/arXiv.2210.13452

Parameters:

• s (float, optional) – Scaled factor for StarReLU, shared across channel. Default: 0.8944

• b (float, optional) – Bias term for StarReLU, shared across channel. Default: -0.4472

• learnable (bool, optional) – optionally make s and b trainable. Default: True

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = torch_activation.StarReLU(s=1.0, b=0.0)
>>> x = torch.randn(3, 384, 384)
>>> output = m(x)

>>> m = torch_activation.StarReLU(learnable=True, inplace=True)
>>> x = torch.randn(3, 384, 384)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

GLU Variations

class torch_activation.glus.GeGLU(dim: int = -1)

Applies the GeGLU activation function, defined as:

\(\text{GeGLU}(x) = \text{GELU} (xW + b) \odot (xV + c)\)

See: https://doi.org/10.48550/arXiv.2002.05202

Parameters:

dim (int, optional) – the dimension on which to split the input. Default: -1

Shape:

• Input: \((\ast_1, N, \ast_2)\) where * means, any number of additional dimensions

• Output: \((\ast_1, M, \ast_2)\) where \(M=N/2\)

Examples:

>>> m = GeGLU(20)
>>> input = torch.randn(3, 20, 20)
>>> output = m(input)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.glus.ReGLU(dim: int = -1)

Applies the GeGLU activation function, defined as:

\(\text{GeGLU}(x) = \text{ReLU} (xW + b) \odot (xV + c)\)

See: https://doi.org/10.48550/arXiv.2002.05202

Parameters:

dim (int, optional) – the dimension on which to split the input. Default: -1

Shape:

• Input: \((\ast_1, N, \ast_2)\) where * means, any number of additional dimensions

• Output: \((\ast_1, M, \ast_2)\) where \(M=N/2\)

Examples:

>>> m = ReGLU(20)
>>> input = torch.randn(3, 20, 20)
>>> output = m(input)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.glus.SeGLU(dim: int = -1)

Applies the SeGLU activation function, defined as:

\(\text{SeGLU}(x) = \text{SELU} (xW + b) \odot (xV + c)\)

Parameters:

dim (int, optional) – the dimension on which to split the input. Default: -1

Shape:

• Input: \((\ast_1, N, \ast_2)\) where * means, any number of additional dimensions

• Output: \((\ast_1, M, \ast_2)\) where \(M=N/2\)

Examples:

>>> m = SeGLU(20)
>>> input = torch.randn(3, 20, 20)
>>> output = m(input)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.glus.SwiGLU(dim: int = -1)

Applies the SwiGLU activation function, defined as:

\(\sigma(x) = \text{Swish} (xW + b) \odot (xV + c)\)

See: https://doi.org/10.48550/arXiv.2002.05202

Parameters:

dim (int, optional) – the dimension on which to split the input. Default: -1

Shape:

• Input: \((\ast_1, N, \ast_2)\) where * means, any number of additional dimensions

• Output: \((\ast_1, M, \ast_2)\) where \(M=N/2\)

Examples:

>>> m = SwiGLU(20)
>>> input = torch.randn(3, 20, 20)
>>> output = m(input)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

Piece-wise functions

class torch_activation.piece_wise.DELU(n: float = 1.0, inplace: bool = False)

Applies the DELU activation function:

\(\text{DELU}(x) = \begin{cases} \text{SiLU}(x), x \leqslant 0 \\x(n-1), \text{otherwise} \end{cases}\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• n (float, optional) – Scaling factor for the positive part of the input. Default: 1.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples

>>> m = nn.DELU()
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = nn.DELU(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

Trigonometry-based Functions

class torch_activation.trig.CosLU(a: float = 1.0, b: float = 1.0, inplace: bool = False)

Applies the Cosine Linear Unit function:

\(\text{CosLU}(x) = (x + a \cdot \cos(b x)) \cdot \sigma(x)\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• a (float, optional) – Scaling factor for the cosine term. Default is 1.0.

• b (float, optional) – Frequency factor for the cosine term. Default is 1.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = CosLU(alpha=2.0, beta=1.0)
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = CosLU(inplace=True)
>>> x = torch.randn(2, 3, 4)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.trig.GCU(inplace: bool = False)

Applies the Growing Cosine Unit activation function:

\(\text{GCU}(x) = x \cos (x)\)

See: https://doi.org/10.48550/arXiv.2108.12943

Parameters:

inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = nn.GCU()
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = nn.GCU(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.trig.SinLU(a: float = 1.0, b: float = 1.0, inplace: bool = False)

Applies the Sinu-sigmoidal Linear Unit activation function:

\(\text{SinLU}(x) = (x + \alpha \sin (\beta x)) \sigma (x)\)

See: https://doi.org/10.3390/math10030337

Parameters:

• a (float, optional) – Initial value for sine function magnitude. Default: 1.0.

• b (float, optional) – Initial value for sine function period. Default: 1.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = nn.SinLU(a=5.0, b=6.0)
>>> x = torch.randn(2)
>>> output = m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

Smooth curve Functions

class torch_activation.curves.CoLU(inplace=False)

Applies the Collapsing Linear Unit activation function:

\(\text{CoLU}(x) = \frac{x}{1-x \cdot e^{-(x + e^x)}}\)

See: https://doi.org/10.48550/arXiv.2112.12078

Parameters:

inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = nn.CoLU()
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = nn.CoLU(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.curves.Phish

Applies the Phish activation function:

\(\text{Phish}(x) = x \odot \tanh (\text{GELU} (x))\)

See: Phish: A Novel Hyper-Optimizable Activation Function.

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples

>>> m = Phish()
>>> x = torch.randn(2, 3)
>>> output = m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.curves.ScaledSoftSign(alpha: float = 1.0, beta: float = 1.0)

Applies the ScaledSoftSign activation function:

\(\text{ScaledSoftSign}(x) = \frac{\alpha \cdot x}{\beta + \|x\|}\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• alpha (float, optional) – The initial value of the alpha parameter.

• beta (float, optional) – The initial value of the beta parameter.

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples

>>> m = ScaledSoftSign(alpha=0.5, beta=1.0)
>>> x = torch.randn(2, 3)
>>> output = m(x)

>>> m = ScaledSoftSign(inplace=True)
>>> x = torch.randn(2, 3)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

Linear Combination

class torch_activation.lincomb.LinComb(activations: Iterable[str])

Applies the LinComb activation function:

\(\text{LinComb}(x) = \sum_{i=1}^{n} w_i \cdot F_i(x)\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

activations (Iterable[str]) – List of activation functions.

Shape:

• Input: \((*)\) where \(*\) means any number of additional dimensions.

• Output: \((*)\)

Examples:

>>> activations = [nn.ReLU(), nn.Sigmoid()]
>>> m = LinComb(activation_functions)
>>> input = torch.randn(10)
>>> output = m(input)

forward(input) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.lincomb.NormLinComb(activations: Iterable[str])

Applies the LinComb activation function:

\(\text{NormLinComb}(x) = \frac{\sum_{i=1}^{n} w_i \cdot F_i(x)}{\|\|W\|\|}\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

activations (Iterable[str]) – List of activation functions.

Shape:

• Input: \((*)\) where \(*\) means any number of additional dimensions.

• Output: \((*)\)

Examples:

>>> activations = [nn.ReLU(), nn.Sigmoid()]
>>> m = NormLinComb(activation_functions)
>>> input = torch.randn(10)
>>> output = m(input)

forward(input) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

Utilities

torch_activation.utils.plot_activation(activation: Module, params: dict, save_dir='./images/activation_images', x_range=(-5, 5), y_range=None, preview=False, plot_derivative=True)

Plot the activation function and optionally its derivative.

Parameters:

• activation (torch.nn.Module) – The activation function to plot.

• params (dict) – A dictionary of parameter names and values for the activation function.

• save_dir (str, optional) – The directory to save the generated image. Defaults to “./images/activation_images”.

• x_range (tuple, optional) – The x-axis range for the plot. Defaults to (-5, 5).

• y_range (tuple, optional) – The y-axis range for the plot. Defaults to None (auto-scale).

• preview (bool, optional) – Whether to display the plot interactively. Defaults to False.

• plot_derivative (bool, optional) – Whether to plot the derivative of the activation function. Defaults to True.

Returns:

None

The function plots the activation function and, optionally, its derivative for the given parameters. The resulting plot is saved as an image in the specified save_dir directory.

If preview is set to True, the plot will also be displayed interactively.

Example:

>>> # Plotting the sigmoid activation function and its derivative
>>> params = {'n': 1.0}
>>> plot_activation(torch.nn.Sigmoid(), params)

All Available Functions

class torch_activation.CReLU(dim: int = 0)

Applies the Concatenated Rectified Linear Unit activation function.

\(\text{CReLU}(x) = \text{ReLU}(x) \oplus \text{ReLU}(-x)\)

See: https://doi.org/10.48550/arXiv.1603.05201

Parameters:

• dim (int, optional) – Dimension along which to concatenate in the output tensor. Default: 1

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*, C, *)\) where \(*\) means any number of additional dimensions

• Output: \((*, 2C, *)\)

Examples:

>>> m = torch_activation.CReLU()
>>> x = torch.randn(2, 3)
>>> output = m(x)

>>> m = torch_activation.CReLU(inplace=True)
>>> x = torch.randn(2, 3, 4)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.CoLU(inplace=False)

Applies the Collapsing Linear Unit activation function:

\(\text{CoLU}(x) = \frac{x}{1-x \cdot e^{-(x + e^x)}}\)

See: https://doi.org/10.48550/arXiv.2112.12078

Parameters:

inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = nn.CoLU()
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = nn.CoLU(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.CosLU(a: float = 1.0, b: float = 1.0, inplace: bool = False)

Applies the Cosine Linear Unit function:

\(\text{CosLU}(x) = (x + a \cdot \cos(b x)) \cdot \sigma(x)\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• a (float, optional) – Scaling factor for the cosine term. Default is 1.0.

• b (float, optional) – Frequency factor for the cosine term. Default is 1.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = CosLU(alpha=2.0, beta=1.0)
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = CosLU(inplace=True)
>>> x = torch.randn(2, 3, 4)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.DELU(n: float = 1.0, inplace: bool = False)

Applies the DELU activation function:

\(\text{DELU}(x) = \begin{cases} \text{SiLU}(x), x \leqslant 0 \\x(n-1), \text{otherwise} \end{cases}\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• n (float, optional) – Scaling factor for the positive part of the input. Default: 1.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples

>>> m = nn.DELU()
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = nn.DELU(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.GCU(inplace: bool = False)

Applies the Growing Cosine Unit activation function:

\(\text{GCU}(x) = x \cos (x)\)

See: https://doi.org/10.48550/arXiv.2108.12943

Parameters:

inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = nn.GCU()
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = nn.GCU(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.GeGLU(dim: int = -1)

Applies the GeGLU activation function, defined as:

\(\text{GeGLU}(x) = \text{GELU} (xW + b) \odot (xV + c)\)

See: https://doi.org/10.48550/arXiv.2002.05202

Parameters:

dim (int, optional) – the dimension on which to split the input. Default: -1

Shape:

• Input: \((\ast_1, N, \ast_2)\) where * means, any number of additional dimensions

• Output: \((\ast_1, M, \ast_2)\) where \(M=N/2\)

Examples:

>>> m = GeGLU(20)
>>> input = torch.randn(3, 20, 20)
>>> output = m(input)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.LinComb(activations: Iterable[str])

Applies the LinComb activation function:

\(\text{LinComb}(x) = \sum_{i=1}^{n} w_i \cdot F_i(x)\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

activations (Iterable[str]) – List of activation functions.

Shape:

• Input: \((*)\) where \(*\) means any number of additional dimensions.

• Output: \((*)\)

Examples:

>>> activations = [nn.ReLU(), nn.Sigmoid()]
>>> m = LinComb(activation_functions)
>>> input = torch.randn(10)
>>> output = m(input)

forward(input) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.NormLinComb(activations: Iterable[str])

Applies the LinComb activation function:

\(\text{NormLinComb}(x) = \frac{\sum_{i=1}^{n} w_i \cdot F_i(x)}{\|\|W\|\|}\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

activations (Iterable[str]) – List of activation functions.

Shape:

• Input: \((*)\) where \(*\) means any number of additional dimensions.

• Output: \((*)\)

Examples:

>>> activations = [nn.ReLU(), nn.Sigmoid()]
>>> m = NormLinComb(activation_functions)
>>> input = torch.randn(10)
>>> output = m(input)

forward(input) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.ReGLU(dim: int = -1)

Applies the GeGLU activation function, defined as:

\(\text{GeGLU}(x) = \text{ReLU} (xW + b) \odot (xV + c)\)

See: https://doi.org/10.48550/arXiv.2002.05202

Parameters:

dim (int, optional) – the dimension on which to split the input. Default: -1

Shape:

• Input: \((\ast_1, N, \ast_2)\) where * means, any number of additional dimensions

• Output: \((\ast_1, M, \ast_2)\) where \(M=N/2\)

Examples:

>>> m = ReGLU(20)
>>> input = torch.randn(3, 20, 20)
>>> output = m(input)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.ReLUN(n: float = 1.0, inplace: bool = False)

Applies the element-wise function:

\(\text{ReLUN}(x) = \min(\text{ReLU}(x), n)\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• n (float, optional) – Upper bound for the function’s output. Default is 1.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = torch_activation.ReLUN(n=6.0) # ReLU6
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = torch_activation.ReLUN(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.ScaledSoftSign(alpha: float = 1.0, beta: float = 1.0)

Applies the ScaledSoftSign activation function:

\(\text{ScaledSoftSign}(x) = \frac{\alpha \cdot x}{\beta + \|x\|}\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• alpha (float, optional) – The initial value of the alpha parameter.

• beta (float, optional) – The initial value of the beta parameter.

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples

>>> m = ScaledSoftSign(alpha=0.5, beta=1.0)
>>> x = torch.randn(2, 3)
>>> output = m(x)

>>> m = ScaledSoftSign(inplace=True)
>>> x = torch.randn(2, 3)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.SeGLU(dim: int = -1)

Applies the SeGLU activation function, defined as:

\(\text{SeGLU}(x) = \text{SELU} (xW + b) \odot (xV + c)\)

Parameters:

dim (int, optional) – the dimension on which to split the input. Default: -1

Shape:

• Input: \((\ast_1, N, \ast_2)\) where * means, any number of additional dimensions

• Output: \((\ast_1, M, \ast_2)\) where \(M=N/2\)

Examples:

>>> m = SeGLU(20)
>>> input = torch.randn(3, 20, 20)
>>> output = m(input)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.ShiLU(alpha: float = 1.0, beta: float = 0.0, inplace: bool = False)

Applies the ShiLU activation function:

\(\text{ShiLU}(x) = \alpha \cdot \text{ReLU}(x) + \beta\)

See: https://doi.org/10.20944/preprints202301.0463.v1

Parameters:

• alpha (float, optional) – Scaling factor for the positive part of the input. Default: 1.0.

• beta (float, optional) – Bias term added to the output. Default: 0.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = torch_activation.ShiLU(alpha=2.0, beta=1.0)
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = torch_activation.ShiLU(inplace=True)
>>> x = torch.randn(2, 3, 4)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.SinLU(a: float = 1.0, b: float = 1.0, inplace: bool = False)

Applies the Sinu-sigmoidal Linear Unit activation function:

\(\text{SinLU}(x) = (x + \alpha \sin (\beta x)) \sigma (x)\)

See: https://doi.org/10.3390/math10030337

Parameters:

• a (float, optional) – Initial value for sine function magnitude. Default: 1.0.

• b (float, optional) – Initial value for sine function period. Default: 1.0.

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = nn.SinLU(a=5.0, b=6.0)
>>> x = torch.randn(2)
>>> output = m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.SquaredReLU(inplace: bool = False)

Applies the element-wise function:

\(\text{SquaredReLU}(x) = \text{ReLU}(x)^2\)

Parameters:

inplace – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = torch_activation.SquaredReLU()
>>> x = torch.randn(2)
>>> output = m(x)

>>> m = torch_activation.SquaredReLU(inplace=True)
>>> x = torch.randn(2)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.StarReLU(s: float = 0.8944, b: float = -0.4472, learnable: bool = True, inplace: bool = False)

Applies the element-wise function:

\(\text{StarReLU}(x) = s \cdot \text{ReLU}(x)^2 + b\)

See: https://doi.org/10.48550/arXiv.2210.13452

Parameters:

• s (float, optional) – Scaled factor for StarReLU, shared across channel. Default: 0.8944

• b (float, optional) – Bias term for StarReLU, shared across channel. Default: -0.4472

• learnable (bool, optional) – optionally make s and b trainable. Default: True

• inplace (bool, optional) – can optionally do the operation in-place. Default: False

Shape:

• Input: \((*)\), where \(*\) means any number of dimensions.

• Output: \((*)\), same shape as the input.

Examples:

>>> m = torch_activation.StarReLU(s=1.0, b=0.0)
>>> x = torch.randn(3, 384, 384)
>>> output = m(x)

>>> m = torch_activation.StarReLU(learnable=True, inplace=True)
>>> x = torch.randn(3, 384, 384)
>>> m(x)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

class torch_activation.SwiGLU(dim: int = -1)

Applies the SwiGLU activation function, defined as:

\(\sigma(x) = \text{Swish} (xW + b) \odot (xV + c)\)

See: https://doi.org/10.48550/arXiv.2002.05202

Parameters:

dim (int, optional) – the dimension on which to split the input. Default: -1

Shape:

• Input: \((\ast_1, N, \ast_2)\) where * means, any number of additional dimensions

• Output: \((\ast_1, M, \ast_2)\) where \(M=N/2\)

Examples:

>>> m = SwiGLU(20)
>>> input = torch.randn(3, 20, 20)
>>> output = m(input)

forward(x) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

training: bool

torch_activation.plot_activation(activation: Module, params: dict, save_dir='./images/activation_images', x_range=(-5, 5), y_range=None, preview=False, plot_derivative=True)

Plot the activation function and optionally its derivative.

Parameters:

• activation (torch.nn.Module) – The activation function to plot.

• params (dict) – A dictionary of parameter names and values for the activation function.

• save_dir (str, optional) – The directory to save the generated image. Defaults to “./images/activation_images”.

• x_range (tuple, optional) – The x-axis range for the plot. Defaults to (-5, 5).

• y_range (tuple, optional) – The y-axis range for the plot. Defaults to None (auto-scale).

• preview (bool, optional) – Whether to display the plot interactively. Defaults to False.

• plot_derivative (bool, optional) – Whether to plot the derivative of the activation function. Defaults to True.

Returns:

None

The function plots the activation function and, optionally, its derivative for the given parameters. The resulting plot is saved as an image in the specified save_dir directory.

If preview is set to True, the plot will also be displayed interactively.

Example:

>>> # Plotting the sigmoid activation function and its derivative
>>> params = {'n': 1.0}
>>> plot_activation(torch.nn.Sigmoid(), params)