Comparator Svcca

svcca

CLASS	DESCRIPTION
SVCCA	Singular Vector Canonical Correlation Analysis (SVCCA) for comparing

Classes

SVCCA(n_eigen: int = 5, variance_threshold: float = 0.99, reduce: Literal['mean', 'median', 'min', 'max'] | None = 'mean', eps: float = torch.finfo(torch.float32).eps)

Bases: ModelComparator

Singular Vector Canonical Correlation Analysis (SVCCA) for comparing neural network models.

SVCCA performs dimensionality reduction via SVD on each model before computing CCA [1].

Algorithm:

1. Extract layer activations from both models
2. Perform SVD on each activation matrix
3. Keep top singular vectors that explain variance_threshold
4. Compute CCA on the reduced representations

PARAMETER	DESCRIPTION
n_eigen	Number of canonical correlations to compute and average. TYPE: int, optional, by default 5 DEFAULT: 5
variance_threshold	Fraction of variance to preserve when selecting singular vectors. Must be in (0, 1]. Higher values keep more dimensions. TYPE: float, optional, by default 0.99 DEFAULT: 0.99
reduce	How to reduce the similarity matrix across layer pairs. TYPE: Literal["mean", "median", "min", "max"] | None, optional, by default "mean" DEFAULT: 'mean'
eps	Epsilon value for numerical stability. TYPE: float, optional, by default torch.finfo(torch.float32).eps DEFAULT: eps

Examples:

>>> import torch
>>> from spectre.comparator import SVCCA
>>> model1 = torch.nn.Sequential(
...     torch.nn.Linear(10, 64), torch.nn.ReLU(), torch.nn.Linear(64, 5)
... )
>>> model2 = torch.nn.Sequential(
...     torch.nn.Linear(10, 32), torch.nn.ReLU(), torch.nn.Linear(32, 5)
... )
>>> X = torch.randn(100, 10)
>>> svcca = SVCCA(n_eigen=3, variance_threshold=0.99)
>>> svcca.fit(model1, model2, X)
>>> print(svcca.score)  # Mean CCA across all layer pairs

Source code in spectre/comparator/svcca.py

def __init__(
    self,
    n_eigen: int = 5,
    variance_threshold: float = 0.99,
    reduce: Literal["mean", "median", "min", "max"] | None = "mean",
    eps: float = torch.finfo(torch.float32).eps,
):
    # Use SVDMatcher with variance threshold for SVCCA-style reduction
    super().__init__(
        comparator_fn="feature",
        comparator_kwargs={
            "method": "cca",
            "n_eigen": n_eigen,
            "eps": eps,
        },
        matcher_fn=SVDMatcher(variance_threshold=variance_threshold),
        reduce=reduce,
    )