Loss Entropy

entropy

CLASS	DESCRIPTION
EntropyLoss	Entropy-based loss for probability distributions with optional weights.

FUNCTION	DESCRIPTION
entropy_loss	Computes a loss based on Shannon entropy with optional sample weights.

Classes

EntropyLoss(reduce: Literal['mean', 'sum'] = 'mean', sign: float = 1.0, eps: float = torch.finfo(torch.float32).eps, context_prefix: str | None = None, **kwargs: dict)

Bases: Loss

Entropy-based loss for probability distributions with optional weights.

Computes a loss based on Shannon entropy \(H(x) = -\sum_k w_k z_k \log(z_k)\), where \(z_k\) is a probability distribution at sample \(k\) and \(w_k\) are optional sample weights.

PARAMETER	DESCRIPTION
reduce	Reduction method. TYPE: Literal["mean", "sum"], optional, by default "mean" DEFAULT: 'mean'
sign	Loss sign (1.0 for minimizing entropy, -1.0 for maximizing). TYPE: float, optional, by default 1.0 DEFAULT: 1.0
eps	Small constant to avoid log(0). TYPE: float, optional, by default torch.finfo(torch.float32).eps DEFAULT: eps
**kwargs	Additional keyword arguments. TYPE: dict DEFAULT: {}

METHOD	DESCRIPTION
forward	Compute entropy loss.

Source code in spectre/loss/entropy.py

def __init__(
    self,
    reduce: Literal["mean", "sum"] = "mean",
    sign: float = 1.0,
    eps: float = torch.finfo(torch.float32).eps,
    context_prefix: str | None = None,
    **kwargs: dict,
) -> None:
    super().__init__(
        reduce=reduce, sign=sign, context_prefix=context_prefix, **kwargs
    )

    self.eps = eps

Functions

forward(context: dict, **kwargs) -> torch.Tensor

Compute entropy loss.

PARAMETER	DESCRIPTION
context	Context dictionary. "Z": torch.Tensor. Probability distribution (1D tensor). Should contain non-negative values that sum to 1. "weights": torch.Tensor | None, optional. Sample weights of shape (n_samples,). If None, uniform weighting is applied. TYPE: dict

RETURNS	DESCRIPTION
Tensor	Entropy loss.

Source code in spectre/loss/entropy.py

def forward(self, context: dict, **kwargs) -> torch.Tensor:
    """
    Compute entropy loss.

    Parameters
    ----------
    context : dict
        Context dictionary.

        - "Z": torch.Tensor. Probability distribution (1D tensor). Should contain
          non-negative values that sum to 1.
        - "weights": torch.Tensor | None, optional. Sample weights of shape
          `(n_samples,)`. If None, uniform weighting is applied.

    Returns
    -------
    torch.Tensor
        Entropy loss.
    """
    Z = self._get_context(context, "Z", None)
    weights = self._get_context(context, "weights", None)

    if Z is None:
        raise ValueError("`EntropyLoss` requires `Z` in context.")

    if weights is None:
        weights = torch.ones_like(Z)

    return _entropy_loss_impl(
        Z, weights, reduce=self.reduce, sign=self.sign, eps=self.eps
    )

Functions

entropy_loss(Z: torch.Tensor, weights: Optional[torch.Tensor] = None, reduce: str = 'mean', sign: float = 1.0, eps: float = torch.finfo(torch.float32).eps) -> torch.Tensor

Computes a loss based on Shannon entropy with optional sample weights.

Computes \(H(x) = -\sum_k w_k z_k \log(z_k)\), where \(z_k\) is a probability distribution at sample \(k\) and \(w_k\) are optional sample weights.

PARAMETER	DESCRIPTION
Z	Probability distribution (1D tensor). Should contain non-negative values that sum to 1. TYPE: Tensor
weights	Sample weights of shape (n_samples, ). If None, uniform weighting is applied. TYPE: torch.Tensor | None, optional, by default None DEFAULT: None
reduce	Reduction method. TYPE: Literal["mean", "sum"], optional, by default "mean" DEFAULT: 'mean'
sign	Loss sign (1.0 for minimizing entropy, -1.0 for maximizing). TYPE: float, optional, by default 1.0 DEFAULT: 1.0
eps	Small constant to avoid log(0). TYPE: float, optional, by default torch.finfo(torch.float32).eps DEFAULT: eps

RETURNS	DESCRIPTION
Tensor	Entropy loss.

Source code in spectre/loss/entropy.py

def entropy_loss(
    Z: torch.Tensor,
    weights: Optional[torch.Tensor] = None,
    reduce: str = "mean",
    sign: float = 1.0,
    eps: float = torch.finfo(torch.float32).eps,
) -> torch.Tensor:
    """
    Computes a loss based on Shannon entropy with optional sample weights.

    Computes $H(x) = -\\sum_k w_k z_k  \\log(z_k)$, where $z_k$ is a probability
    distribution at sample $k$ and $w_k$ are optional sample weights.

    Parameters
    ----------
    Z : torch.Tensor
        Probability distribution (1D tensor). Should contain non-negative values
        that sum to 1.

    weights : torch.Tensor | None, optional, by default None
        Sample weights of shape `(n_samples, )`. If None, uniform weighting is applied.

    reduce : Literal["mean", "sum"], optional, by default "mean"
        Reduction method.

    sign : float, optional, by default 1.0
        Loss sign (1.0 for minimizing entropy, -1.0 for maximizing).

    eps : float, optional, by default torch.finfo(torch.float32).eps
        Small constant to avoid log(0).

    Returns
    -------
    torch.Tensor
        Entropy loss.
    """
    check_1d(Z)
    check_all_non_negative(Z)

    if weights is not None:
        check_1d(weights)
        check_same_len(Z, weights)
        check_all_non_negative(weights)
    else:
        weights = torch.ones_like(Z)

    if reduce not in ["sum", "mean"]:
        raise ValueError("Unknown reduce operation. Options: 'sum', 'mean'.")

    return _entropy_loss_impl(Z, weights=weights, reduce=reduce, sign=sign, eps=eps)