Validation Bootstrap

bootstrap

FUNCTION	DESCRIPTION
bootstrap	Estimate score statistics via bootstrap resampling with replacement.

Classes

Functions

bootstrap(model: Any, X: torch.Tensor, fit_score_fn: Callable, weights: torch.Tensor | None = None, target: torch.Tensor | None = None, n_bootstrap: int = 1000, sample_size: float | int | None = None, confidence_level: float = 0.95, random_state: int | None = None, device: str | torch.device | None = None, n_jobs: int = 1, backend: str = 'threading', verbose: int = 0, pre_dispatch: str = '2*n_jobs', return_samples: bool = False, return_estimator: bool = False, return_times: bool = True, error_score: float | str = 'raise', dtype: torch.dtype = torch.float32) -> dict[str, float | torch.Tensor]

Estimate score statistics via bootstrap resampling with replacement.

Performs repeated random sampling with replacement to estimate confidence intervals and standard errors of a scoring metric.

PARAMETER	DESCRIPTION
model	Fitted model to evaluate. TYPE: Any
X	Input data tensor of shape (n_samples, n_features). TYPE: Tensor
fit_score_fn	Scoring function with signature: fit_score_fn(model, sample_data) -> dict[str, Tensor]. sample_data: dict with key 'X' mapping to DataBatch object Must return dict with 'score' key Score should be a scalar Tensor TYPE: Callable
weights	Sample weights of shape (n_samples,). TYPE: torch.Tensor | None, optional, by default None DEFAULT: None
target	Target values of shape (n_samples,). TYPE: torch.Tensor | None, optional, by default None DEFAULT: None
n_bootstrap	Number of bootstrap samples to generate. TYPE: int DEFAULT: 1000
sample_size	Size of each bootstrap sample. float in (0, 1]: Fraction of n_samples int > 0: Absolute number of samples None: Original dataset size (standard bootstrap) TYPE: float | int | None, optional, by default None DEFAULT: None
confidence_level	Confidence level for CI bounds (must be in (0, 1)). TYPE: float, by default 0.95 DEFAULT: 0.95
random_state	Random seed for reproducibility. TYPE: int | None, optional, by default None DEFAULT: None
device	Device for computation ('cpu', 'cuda', 'cuda:0', etc.). Default uses device of X. CUDA disables parallelization. TYPE: str | device | None DEFAULT: None
n_jobs	Number of parallel jobs. -1 uses all processors. Automatically set to 1 when using CUDA device. TYPE: int, by default 1 DEFAULT: 1
backend	Joblib backend for parallelization. TYPE: str, by default "threading" DEFAULT: 'threading'
verbose	Verbosity level for joblib. TYPE: int, by default 0 DEFAULT: 0
pre_dispatch	Number of jobs to pre-dispatch for parallel execution. TYPE: str, by default "2*n_jobs" DEFAULT: '2*n_jobs'
return_samples	Whether to return individual bootstrap scores. TYPE: bool, by default False DEFAULT: False
return_estimator	Whether to return fitted estimators. TYPE: bool, by default False DEFAULT: False
return_times	Whether to include execution times per sample. TYPE: bool, by default True DEFAULT: True
error_score	Behavior on scoring errors: 'raise': Re-raise exception 'warn': Assign nan and issue warning numeric value: Assign that value TYPE: float | str, by default "raise" DEFAULT: 'raise'
dtype	Data type for error score tensors. TYPE: dtype DEFAULT: torch.float32

RETURNS

DESCRIPTION

dict[str, float | Tensor]

Dictionary with keys: 'score': All bootstrap scores, shape (n_bootstrap,) 'score_mean': Mean of bootstrap scores 'score_std': Standard error (std of bootstrap distribution) 'score_ci_lower': Lower CI bound (quantile-based) 'score_ci_upper': Upper CI bound (quantile-based) 'n_bootstrap': Number of successful samples 'time': List of execution times (if return_times=True) 'estimator': Model reference (if return_estimator=True)

RAISES	DESCRIPTION
TypeError	If model is not compatible or fit_score_fn is not callable.
ValueError	If confidence_level not in (0, 1) or sample_size is invalid.

Notes

Bootstrap distribution approximates the sampling distribution of the statistic, enabling non-parametric confidence intervals. Standard bootstrap uses full-size resamples; undersampling available via sample_size.

Examples:

Basic usage with fitted model:

>>> from spectre.decomposition import PrincipialCompotentAnalysis as PCA
>>> from spectre.validation import bootstrap
>>> import torch
>>>
>>> # Fit model
>>> pca = PCA(n_components=3)
>>> X = torch.randn(200, 20)
>>> pca.fit(X)
>>>
>>> # Define scoring function
>>> def score_fn(model, data):
...     batch = data["X"]
...     # Custom metric: negative reconstruction error
...     pred = model.transform(batch.data)
...     loss = torch.norm(batch.data - pred)
...     return {"score": -loss}
>>>
>>> # Bootstrap evaluation
>>> stats = bootstrap(pca, X, fit_score_fn=score_fn, n_bootstrap=500)
>>> print(f"Mean: {stats['score_mean']:.3f}")
>>> print(f"95% CI: [{stats['score_ci_lower']:.3f}, {stats['score_ci_upper']:.3f}]")

With weighted samples and reduced sample size:

>>> weights = torch.ones(200)
>>> stats = bootstrap(
...     pca,
...     X,
...     fit_score_fn=score_fn,
...     weights=weights,
...     sample_size=0.7,  # 70% of original
...     n_bootstrap=1000,
... )

Parallel execution on multi-core CPU:

>>> stats = bootstrap(
...     pca,
...     X,
...     fit_score_fn=score_fn,
...     n_bootstrap=1000,
...     n_jobs=-1,  # Use all available cores
... )

GPU-accelerated execution (no parallelization):

>>> X_gpu = X.cuda()
>>> pca_gpu = pca.to("cuda")
>>> stats = bootstrap(
...     pca_gpu,
...     X_gpu,
...     fit_score_fn=score_fn,
...     device="cuda",
...     n_bootstrap=500,
... )

Source code in spectre/validation/bootstrap.py

def bootstrap(
    model: Any,
    X: torch.Tensor,
    fit_score_fn: Callable,
    weights: torch.Tensor | None = None,
    target: torch.Tensor | None = None,
    n_bootstrap: int = 1000,
    sample_size: float | int | None = None,
    confidence_level: float = 0.95,
    random_state: int | None = None,
    device: str | torch.device | None = None,
    n_jobs: int = 1,
    backend: str = "threading",
    verbose: int = 0,
    pre_dispatch: str = "2*n_jobs",
    return_samples: bool = False,
    return_estimator: bool = False,
    return_times: bool = True,
    error_score: float | str = "raise",
    dtype: torch.dtype = torch.float32,
) -> dict[str, float | torch.Tensor]:
    """
    Estimate score statistics via bootstrap resampling with replacement.

    Performs repeated random sampling with replacement to estimate confidence
    intervals and standard errors of a scoring metric.

    Parameters
    ----------
    model : Any
        Fitted model to evaluate.

    X : torch.Tensor
        Input data tensor of shape (n_samples, n_features).

    fit_score_fn : Callable
        Scoring function with signature:
        `fit_score_fn(model, sample_data) -> dict[str, Tensor]`.

        - `sample_data`: dict with key `'X'` mapping to `DataBatch` object
        - Must return dict with `'score'` key
        - Score should be a scalar `Tensor`

    weights : torch.Tensor | None, optional, by default None
        Sample weights of shape `(n_samples,)`.

    target : torch.Tensor | None, optional, by default None
        Target values of shape `(n_samples,)`.

    n_bootstrap : int, default 1000
        Number of bootstrap samples to generate.

    sample_size : float | int | None, optional, by default None
        Size of each bootstrap sample.

        - `float` in (0, 1]: Fraction of `n_samples`
        - `int` > 0: Absolute number of samples
        - `None`: Original dataset size (standard bootstrap)

    confidence_level : float, by default 0.95
        Confidence level for CI bounds (must be in (0, 1)).

    random_state : int | None, optional, by default None
        Random seed for reproducibility.

    device : str | torch.device | None, optional
        Device for computation ('cpu', 'cuda', 'cuda:0', etc.).
        Default uses device of `X`. CUDA disables parallelization.

    n_jobs : int, by default 1
        Number of parallel jobs. -1 uses all processors.
        Automatically set to 1 when using CUDA device.

    backend : str, by default "threading"
        Joblib backend for parallelization.

    verbose : int, by default 0
        Verbosity level for joblib.

    pre_dispatch : str, by default "2*n_jobs"
        Number of jobs to pre-dispatch for parallel execution.

    return_samples : bool, by default False
        Whether to return individual bootstrap scores.

    return_estimator : bool, by default False
        Whether to return fitted estimators.

    return_times : bool, by default True
        Whether to include execution times per sample.

    error_score : float | str, by default "raise"
        Behavior on scoring errors:

        - `'raise'`: Re-raise exception
        - `'warn'`: Assign `nan` and issue warning
        - numeric value: Assign that value

    dtype : torch.dtype, default torch.float32
        Data type for error score tensors.

    Returns
    -------
    dict[str, float | torch.Tensor]
        Dictionary with keys:

        - `'score'`: All bootstrap scores, shape `(n_bootstrap,)`
        - `'score_mean'`: Mean of bootstrap scores
        - `'score_std'`: Standard error (std of bootstrap distribution)
        - `'score_ci_lower'`: Lower CI bound (quantile-based)
        - `'score_ci_upper'`: Upper CI bound (quantile-based)
        - `'n_bootstrap'`: Number of successful samples
        - `'time'`: List of execution times (if `return_times=True`)
        - `'estimator'`: Model reference (if `return_estimator=True`)

    Raises
    ------
    TypeError
        If `model` is not compatible or `fit_score_fn` is not callable.
    ValueError
        If `confidence_level` not in (0, 1) or `sample_size` is invalid.

    Notes
    -----
    Bootstrap distribution approximates the sampling distribution of the
    statistic, enabling non-parametric confidence intervals. Standard bootstrap
    uses full-size resamples; undersampling available via `sample_size`.

    Examples
    --------
    Basic usage with fitted model:

    >>> from spectre.decomposition import PrincipialCompotentAnalysis as PCA
    >>> from spectre.validation import bootstrap
    >>> import torch
    >>>
    >>> # Fit model
    >>> pca = PCA(n_components=3)
    >>> X = torch.randn(200, 20)
    >>> pca.fit(X)
    >>>
    >>> # Define scoring function
    >>> def score_fn(model, data):
    ...     batch = data["X"]
    ...     # Custom metric: negative reconstruction error
    ...     pred = model.transform(batch.data)
    ...     loss = torch.norm(batch.data - pred)
    ...     return {"score": -loss}
    >>>
    >>> # Bootstrap evaluation
    >>> stats = bootstrap(pca, X, fit_score_fn=score_fn, n_bootstrap=500)
    >>> print(f"Mean: {stats['score_mean']:.3f}")
    >>> print(f"95% CI: [{stats['score_ci_lower']:.3f}, {stats['score_ci_upper']:.3f}]")

    With weighted samples and reduced sample size:

    >>> weights = torch.ones(200)
    >>> stats = bootstrap(
    ...     pca,
    ...     X,
    ...     fit_score_fn=score_fn,
    ...     weights=weights,
    ...     sample_size=0.7,  # 70% of original
    ...     n_bootstrap=1000,
    ... )

    Parallel execution on multi-core CPU:

    >>> stats = bootstrap(
    ...     pca,
    ...     X,
    ...     fit_score_fn=score_fn,
    ...     n_bootstrap=1000,
    ...     n_jobs=-1,  # Use all available cores
    ... )

    GPU-accelerated execution (no parallelization):

    >>> X_gpu = X.cuda()
    >>> pca_gpu = pca.to("cuda")
    >>> stats = bootstrap(
    ...     pca_gpu,
    ...     X_gpu,
    ...     fit_score_fn=score_fn,
    ...     device="cuda",
    ...     n_bootstrap=500,
    ... )
    """
    if not isinstance(confidence_level, float):
        raise TypeError(
            f"`confidence_level` must be float, got {type(confidence_level)}."
        )
        check_in_interval(confidence_level, "(0, 1)")

    batch = DataBatch(data=X, weights=weights, target=target)

    # Delegate to dispatch_bootstrap
    results = dispatch_bootstrap(
        estimator=model,
        datasets={"X": batch},
        fit_score_fn=fit_score_fn,
        n_bootstrap=n_bootstrap,
        sample_size=sample_size,
        random_state=random_state,
        device=device,
        n_jobs=n_jobs,
        backend=backend,
        verbose=verbose,
        pre_dispatch=pre_dispatch,
        return_estimator=return_estimator,
        return_times=return_times,
        error_score=error_score,
        dtype=dtype,
    )

    return results | {
        "score_mean": results["score"].mean(),
        "score_std": results["score"].std(),
        "score_ci_lower": torch.quantile(results["score"], (1 - confidence_level) / 2),
        "score_ci_upper": torch.quantile(
            results["score"], 1 - (1 - confidence_level) / 2
        ),
        "n_bootstrap": results["n_samples"],
    }