Comparator Matcher

matcher

CLASS	DESCRIPTION
Matcher	Base class for dimension matching strategies.
SVDMatcher	Match dimensions using Singular Value Decomposition (SVD).
ProjectionMatcher	Match dimensions using random projection.
PaddingMatcher	Match dimensions by zero-padding the smaller tensor.
TruncationMatcher	Match dimensions by truncating the larger tensor.
SkipMatcher	Strict matcher that requires exact dimension match.

FUNCTION	DESCRIPTION
match_dimensions	Match feature dimensions between two activation matrices.

Classes

Matcher

Bases: Module

Base class for dimension matching strategies.

Matchers are used to align feature dimensions between two activation matrices when comparing neural network representations.

METHOD	DESCRIPTION
forward	Match dimensions between two tensors.

ATTRIBUTE	DESCRIPTION
name	Get the registered name of this matcher. TYPE: str | None

Attributes

name: str | None property

Get the registered name of this matcher.

RETURNS	DESCRIPTION
str | None	The registration name if registered, None otherwise.

Functions

forward(X: torch.Tensor, Y: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor] abstractmethod

Match dimensions between two tensors.

PARAMETER	DESCRIPTION
X	First activation matrix, shape (n_samples, n_features). TYPE: Tensor
Y	Second activation matrix, shape (n_samples, m_features). TYPE: Tensor

RETURNS	DESCRIPTION
tuple[Tensor, Tensor]	Matched activation matrices with same feature dimension.

Source code in spectre/comparator/matcher.py

@abstractmethod
def forward(
    self, X: torch.Tensor, Y: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Match dimensions between two tensors.

    Parameters
    ----------
    X : torch.Tensor
        First activation matrix, shape (n_samples, n_features).
    Y : torch.Tensor
        Second activation matrix, shape (n_samples, m_features).

    Returns
    -------
    tuple[torch.Tensor, torch.Tensor]
        Matched activation matrices with same feature dimension.
    """
    raise NotImplementedError

MatcherRegistry

Bases: Registry

Registry for matcher classes.

SVDMatcher(variance_threshold: float | None = None)

Bases: Matcher

Match dimensions using Singular Value Decomposition (SVD).

Projects activations to a common dimensionality using truncated SVD, which preserves maximum variance. Optionally supports variance-based component selection (SVCCA-style).

PARAMETER	DESCRIPTION
variance_threshold	If provided, keeps components that explain this fraction of variance (0 < threshold <= 1). If None, projects to min(n_features, m_features). TYPE: float | None, optional, by default None DEFAULT: None

Source code in spectre/comparator/matcher.py

def __init__(self, variance_threshold: float | None = None):
    super().__init__()
    if variance_threshold is not None:
        if not isinstance(variance_threshold, float):
            raise TypeError(
                f"`variance_threshold` must be float, got {type(variance_threshold)}"
            )
        check_in_interval(variance_threshold, "[0, 1]")
    self.variance_threshold = variance_threshold

ProjectionMatcher()

Bases: Matcher

Match dimensions using random projection.

Fast but non-deterministic approach using random linear projections.

Source code in spectre/comparator/matcher.py

def __init__(self):
    super().__init__()

PaddingMatcher()

Bases: Matcher

Match dimensions by zero-padding the smaller tensor.

Simple approach that adds zeros to match the larger dimension.

Source code in spectre/comparator/matcher.py

def __init__(self):
    super().__init__()

TruncationMatcher()

Bases: Matcher

Match dimensions by truncating the larger tensor.

Simple approach that discards features from the larger representation.

Source code in spectre/comparator/matcher.py

def __init__(self):
    super().__init__()

SkipMatcher()

Bases: Matcher

Strict matcher that requires exact dimension match.

Raises an error if dimensions don't match.

Source code in spectre/comparator/matcher.py

def __init__(self):
    super().__init__()

Functions

initialize_matcher_fn(matcher_fn: Matcher | str | None, matcher_kwargs: dict[str, Any] | None = None) -> Matcher

Initialize a matcher from an instance or registry name.

Arguments

matcher_fn : Matcher | str | None Matcher instance or name of registered matcher. matcher_kwargs : dict[str, Any] | None Keyword arguments passed to matcher constructor (only for string names).

RETURNS	DESCRIPTION
Matcher	Initialized matcher instance.

RAISES	DESCRIPTION
ValueError	If matcher_fn is None.
TypeError	If matcher_fn is not a Matcher instance or string.

Source code in spectre/comparator/matcher.py

def initialize_matcher_fn(
    matcher_fn: Matcher | str | None,
    matcher_kwargs: dict[str, Any] | None = None,
) -> Matcher:
    """
    Initialize a matcher from an instance or registry name.

    Arguments
    ---------
    matcher_fn : Matcher | str | None
        Matcher instance or name of registered matcher.
    matcher_kwargs : dict[str, Any] | None
        Keyword arguments passed to matcher constructor (only for string names).

    Returns
    -------
    Matcher
        Initialized matcher instance.

    Raises
    ------
    ValueError
        If `matcher_fn` is None.
    TypeError
        If `matcher_fn` is not a Matcher instance or string.
    """
    return _initialize_from_registry(
        registry=MatcherRegistry,
        obj=matcher_fn,
        kwargs=matcher_kwargs,
        param_name="matcher_fn",
    )

match_dimensions(X: torch.Tensor, Y: torch.Tensor, method: Literal['svd', 'projection', 'padding', 'truncation', 'skip'] = 'padding', variance_threshold: float | None = None, eps: float = torch.finfo(torch.float32).eps) -> tuple[torch.Tensor, torch.Tensor]

Match feature dimensions between two activation matrices.

This function provides multiple strategies for handling layers with different feature dimensions.

SVD (for static comparison):

• Preserves maximum variance in the data
• Deterministic and stable
• Projects to min(n_features, m_features) dimensions

Projection:

• Random projection (non-deterministic)
• Fast but may not preserve variance optimally

Padding:

• Simple but adds zeros that may affect kernel computation
• Zero-padded features have different statistical properties

Truncation:

• Simple but discards information from larger representation
• May lose important features

Skip:

• Strict mode requiring exact dimension match
• Use when you want to ensure comparable representations

PARAMETER	DESCRIPTION
X	First activation matrix, shape (n_samples, n_features). TYPE: Tensor
Y	Second activation matrix, shape (n_samples, m_features). TYPE: Tensor
method	Method for dimension matching. "svd": Project to common dimension using truncated SVD (preserves variance) "projection": Linear projection "padding": Zero-padding smaller tensor "truncation": Crop larger tensor "skip": Skip dimension matching (raises error if n_features != m_features) TYPE: str, optional, by default "padding" DEFAULT: 'padding'
variance_threshold	For method="svd" only: if provided, determines number of components to keep based on explained variance (0 < threshold <= 1). If None, projects to min(n_features, m_features). This implements variance-based dimensionality reduction as used in SVCCA. TYPE: float | None, optional, by default None DEFAULT: None
eps	For method="svd" only: numerical stability parameter for SVD. TYPE: float, optional, by default torch.finfo(torch.float32).eps DEFAULT: eps

RETURNS	DESCRIPTION
tuple[Tensor, Tensor]	Matched activation matrices with same feature dimension.

RAISES	DESCRIPTION
ValueError	If method="skip" and dimensions don't match, or if unknown method.

Examples:

Static comparison with SVD:

>>> X1 = torch.randn(100, 64)
>>> X2 = torch.randn(100, 32)
>>> X1_matched, X2_matched = match_dimensions(X1, X2, method="svd")
>>> X1_matched.shape, X2_matched.shape
(torch.Size([100, 32]), torch.Size([100, 32]))

Learnable projection for training:

>>> X_matched_1, X_matched_2 = match_dimensions(X1, X2, method="projection")

Padding for simple matching:

>>> X1_padded, X2_padded = match_dimensions(X1, X2, method="padding")
>>> X1_padded.shape, X2_padded.shape
(torch.Size([100, 64]), torch.Size([100, 64]))

Source code in spectre/comparator/matcher.py

def match_dimensions(
    X: torch.Tensor,
    Y: torch.Tensor,
    method: Literal["svd", "projection", "padding", "truncation", "skip"] = "padding",
    variance_threshold: float | None = None,
    eps: float = torch.finfo(torch.float32).eps,
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Match feature dimensions between two activation matrices.

    This function provides multiple strategies for handling layers with
    different feature dimensions.

    SVD (for static comparison):

    - Preserves maximum variance in the data
    - Deterministic and stable
    - Projects to min(n_features, m_features) dimensions

    Projection:

    - Random projection (non-deterministic)
    - Fast but may not preserve variance optimally

    Padding:

    - Simple but adds zeros that may affect kernel computation
    - Zero-padded features have different statistical properties

    Truncation:

    - Simple but discards information from larger representation
    - May lose important features

    Skip:

    - Strict mode requiring exact dimension match
    - Use when you want to ensure comparable representations

    Parameters
    ----------
    X : torch.Tensor
        First activation matrix, shape (n_samples, n_features).

    Y : torch.Tensor
        Second activation matrix, shape (n_samples, m_features).

    method : str, optional, by default "padding"
        Method for dimension matching.

        - "svd": Project to common dimension using truncated SVD (preserves variance)
        - "projection": Linear projection
        - "padding": Zero-padding smaller tensor
        - "truncation": Crop larger tensor
        - "skip": Skip dimension matching (raises error if n_features != m_features)

    variance_threshold : float | None, optional, by default None
        For method="svd" only: if provided, determines number of components to keep
        based on explained variance (0 < threshold <= 1). If None, projects to
        min(n_features, m_features). This implements variance-based dimensionality
        reduction as used in SVCCA.

    eps : float, optional, by default torch.finfo(torch.float32).eps
        For method="svd" only: numerical stability parameter for SVD.

    Returns
    -------
    tuple[torch.Tensor, torch.Tensor]
        Matched activation matrices with same feature dimension.

    Raises
    ------
    ValueError
        If method="skip" and dimensions don't match, or if unknown method.

    Examples
    --------
    Static comparison with SVD:

    >>> X1 = torch.randn(100, 64)
    >>> X2 = torch.randn(100, 32)
    >>> X1_matched, X2_matched = match_dimensions(X1, X2, method="svd")
    >>> X1_matched.shape, X2_matched.shape
    (torch.Size([100, 32]), torch.Size([100, 32]))

    Learnable projection for training:

    >>> X_matched_1, X_matched_2 = match_dimensions(X1, X2, method="projection")

    Padding for simple matching:

    >>> X1_padded, X2_padded = match_dimensions(X1, X2, method="padding")
    >>> X1_padded.shape, X2_padded.shape
    (torch.Size([100, 64]), torch.Size([100, 64]))
    """
    check_2d(X)
    check_2d(Y)
    check_same_len(X, Y)
    check_same_device(X, Y)

    n_features = X.shape[1]
    m_features = Y.shape[1]

    if n_features == m_features:
        return X, Y

    if method == "skip":
        raise ValueError(
            f"Feature dimensions do not match: {n_features} vs {m_features}. "
            "Use a different method to match dimensions."
        )

    elif method == "svd":
        # SVD projection to common dimension (preserves maximum variance)

        # Perform SVD on both matrices
        U_X, S_X, _ = torch.linalg.svd(X, full_matrices=False)
        U_Y, S_Y, _ = torch.linalg.svd(Y, full_matrices=False)

        # Determine number of components to keep
        if variance_threshold is not None:
            # Variance-based reduction (SVCCA-style)
            n_comp_X = _get_n_eigen(S_X, variance_threshold, len(S_X))
            n_comp_Y = _get_n_eigen(S_Y, variance_threshold, len(S_Y))

            # Project each matrix to its optimal number of components
            S_X_clamped = torch.clamp(S_X, min=eps)
            S_Y_clamped = torch.clamp(S_Y, min=eps)

            X = U_X[:, :n_comp_X] * S_X_clamped[:n_comp_X]
            Y = U_Y[:, :n_comp_Y] * S_Y_clamped[:n_comp_Y]

            # Pad to common dimension (max of the two)
            d_common = max(n_comp_X, n_comp_Y)
            if n_comp_X < d_common:
                pad = torch.zeros(
                    X.shape[0], d_common - n_comp_X, device=X.device, dtype=X.dtype
                )
                X = torch.cat([X, pad], dim=1)
            if n_comp_Y < d_common:
                pad = torch.zeros(
                    Y.shape[0], d_common - n_comp_Y, device=Y.device, dtype=Y.dtype
                )
                Y = torch.cat([Y, pad], dim=1)
        else:
            # Fixed reduction to min dimension
            d_common = min(n_features, m_features)

            # Project to selected number of components
            if n_features >= d_common:
                eps = torch.finfo(X.dtype).eps
                S_X = torch.clamp(S_X, min=eps)
                X = U_X[:, :d_common] * S_X[:d_common]

            if m_features >= d_common:
                eps = torch.finfo(Y.dtype).eps
                S_Y = torch.clamp(S_Y, min=eps)
                Y = U_Y[:, :d_common] * S_Y[:d_common]

        return X, Y

    elif method == "projection":
        d_common = min(n_features, m_features)

        # Random projection using manual matrix multiplication
        # Avoids creating nn.Module instances that persist in memory
        if n_features > d_common:
            # Use Xavier/Glorot initialization for random projection
            proj_matrix = torch.randn(
                n_features, d_common, device=X.device, dtype=X.dtype
            ) / (n_features**0.5)
            X = X @ proj_matrix

        if m_features > d_common:
            proj_matrix = torch.randn(
                m_features, d_common, device=Y.device, dtype=Y.dtype
            ) / (m_features**0.5)
            Y = Y @ proj_matrix

        return X, Y

    elif method == "padding":
        # Zero-pad smaller tensor to match larger one
        if n_features < m_features:
            pad_size = m_features - n_features
            padding = torch.zeros(X.shape[0], pad_size, device=X.device)
            X = torch.cat([X, padding], dim=-1)
        elif m_features < n_features:
            pad_size = n_features - m_features
            padding = torch.zeros(Y.shape[0], pad_size, device=Y.device)
            Y = torch.cat([Y, padding], dim=-1)

        return X, Y

    elif method == "truncation":
        # Truncate larger tensor to match smaller one
        d_common = min(n_features, m_features)
        X = X[:, :d_common]
        Y = Y[:, :d_common]
        return X, Y

    else:
        raise ValueError(
            f"Unknown method '{method}'. "
            "Choose from: 'svd', 'projection', 'padding', 'truncation', 'skip'"
        )