Module Activation

activation

CLASS	DESCRIPTION
TemperedSigmoid	Tempered sigmoid activation function with learnable temperature parameter.
TemperedSine	Tempered sine activation function with learnable temperature parameter.
ShiftedSoftplus	Shifted Softplus activation function.
SmoothedShiftedSoftplus	Smoothed shifted softplus (SSP) activation function.

Classes

TemperedSigmoid(beta: float = 1.0, dtype: torch.dtype = torch.float32)

Bases: Module

Tempered sigmoid activation function with learnable temperature parameter.

Applies sigmoid with a learnable temperature scaling: \(\mathrm{sigmoid}(\beta X)\).

PARAMETER	DESCRIPTION
beta	Temperature parameter for scaling input. TYPE: float, optional, by default 1.0 DEFAULT: 1.0
dtype	Data type for the temperature parameter. TYPE: torch.dtype, optional, by default torch.float32 DEFAULT: float32

METHOD	DESCRIPTION
forward	Apply tempered sigmoid activation.

Source code in spectre/module/activation.py

def __init__(self, beta: float = 1.0, dtype: torch.dtype = torch.float32) -> None:
    super().__init__()

    self.beta = torch.nn.Parameter(
        torch.tensor(beta, dtype=dtype), requires_grad=True
    )

Functions

forward(X: torch.Tensor) -> torch.Tensor

Apply tempered sigmoid activation.

Source code in spectre/module/activation.py

def forward(self, X: torch.Tensor) -> torch.Tensor:
    """Apply tempered sigmoid activation."""
    return torch.sigmoid(self.beta * X)

TemperedSine(beta: float = 1.0, dtype: torch.dtype = torch.float32)

Bases: Module

Tempered sine activation function with learnable temperature parameter.

Applies sine with a learnable temperature scaling: \(\sin(\beta X)\).

PARAMETER	DESCRIPTION
beta	Temperature parameter for scaling input. TYPE: float, optional, by default 1.0 DEFAULT: 1.0
dtype	Data type for the temperature parameter. TYPE: torch.dtype, optional, by default torch.float32 DEFAULT: float32

METHOD	DESCRIPTION
forward	Apply tempered sine activation.

Source code in spectre/module/activation.py

def __init__(self, beta: float = 1.0, dtype: torch.dtype = torch.float32) -> None:
    super().__init__()

    self.beta = torch.nn.Parameter(
        torch.tensor(beta, dtype=dtype), requires_grad=True
    )

Functions

forward(X: torch.Tensor) -> torch.Tensor

Apply tempered sine activation.

Source code in spectre/module/activation.py

def forward(self, X: torch.Tensor) -> torch.Tensor:
    """Apply tempered sine activation."""
    return torch.sin(self.beta * X)

ShiftedSoftplus(beta: float = 1.0, threshold: float = 20.0)

Bases: Softplus

Shifted Softplus activation function.

Applies Softplus activation shifted to ensure f(0) = 0. This improves training stability by centering the activation around zero.

PARAMETER	DESCRIPTION
beta	Softplus beta parameter. TYPE: float, optional, by default 1.0. DEFAULT: 1.0
threshold	Threshold for switching to linear approximation. TYPE: float, optional, by default 20 DEFAULT: 20.0

METHOD	DESCRIPTION
forward	Apply shifted softplus activation.

Source code in spectre/module/activation.py

def __init__(self, beta: float = 1.0, threshold: float = 20.0) -> None:
    super().__init__(beta, threshold)

Functions

forward(X: torch.Tensor) -> torch.Tensor

Apply shifted softplus activation.

Source code in spectre/module/activation.py

def forward(self, X: torch.Tensor) -> torch.Tensor:  # ty: ignore
    """Apply shifted softplus activation."""
    sp_0 = torch.nn.functional.softplus(
        torch.tensor(0.0, dtype=X.dtype), self.beta, self.threshold
    )

    return torch.nn.functional.softplus(X, self.beta, self.threshold) - sp_0

SmoothedShiftedSoftplus(alpha: float = 0.5, beta: float = 0.5, dtype: torch.dtype = torch.float32, device: torch.device | str | None = None)

Bases: Module

Smoothed shifted softplus (SSP) activation function.

Custom activation function implementing: \(\log(\alpha \exp(X) + \beta)\). Provides smooth transitions and improved gradient flow.

Note: This is different from ShiftedSoftplus - SSP uses a custom formula while ShiftedSoftplus shifts the standard softplus to ensure f(0) = 0.

PARAMETER	DESCRIPTION
alpha	Exponential scaling parameter. Must be positive. TYPE: float, optional, by default 0.5 DEFAULT: 0.5
beta	Additive constant parameter. Must be positive. TYPE: float, optional, by default 0.5 DEFAULT: 0.5
dtype	Data type of the input tensor. TYPE: torch.dtype, optional, by default torch.float32 DEFAULT: float32
device	Device to run computations on. If None, automatically selects CUDA if available, otherwise CPU. TYPE: torch.device | str | None, by default None DEFAULT: None

METHOD	DESCRIPTION
forward	Apply SSP activation function.

Source code in spectre/module/activation.py

def __init__(
    self,
    alpha: float = 0.5,
    beta: float = 0.5,
    dtype: torch.dtype = torch.float32,
    device: torch.device | str | None = None,
) -> None:
    super().__init__()

    if not isinstance(alpha, (int, float)):
        raise ValueError(f"`alpha` must be a number, got {alpha}.")
    check_in_interval(alpha, "(0, inf)")
    self.alpha = alpha

    if not isinstance(beta, (int, float)):
        raise ValueError(f"`beta` must be positive, got {beta}.")
    check_in_interval(beta, "(0, inf)")
    self.beta = beta

    self.dtype = dtype

    if device is None:
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    else:
        self.device = torch.device(device)

    gamma = alpha / beta
    self.register_buffer(
        "log_b", torch.log(torch.tensor(beta, device=self.device, dtype=self.dtype))
    )
    self.register_buffer(
        "log_a_div_b",
        torch.log(torch.tensor(gamma, device=self.device, dtype=self.dtype)),
    )

Functions

forward(X: torch.Tensor) -> torch.Tensor

Apply SSP activation function.

Source code in spectre/module/activation.py

def forward(self, X: torch.Tensor) -> torch.Tensor:
    """Apply SSP activation function."""
    # Rewrite log(alpha*exp(X) + beta) using log-sum-exp trick:
    # = log(beta * (alpha/beta * exp(X) + 1))
    # = log(beta) + log(alpha/beta * exp(X) + 1)
    # = log(beta) + softplus(X + log(alpha/beta))
    log_b = self.log_b.to(X.device)
    log_a_div_b = self.log_a_div_b.to(X.device)
    return log_b + torch.nn.functional.softplus(X + log_a_div_b)

Functions