Comparison Tests and Validation Metrics

Group-comparison tests

The (weighted, optionally stratified) log-rank test for comparing survival distributions across groups:

surpyval.univariate.nonparametric.logrank.logrank(x, Z, c=None, n=None, weighting='log-rank', rho=0, gamma=0, strata=None)

The k-sample (weighted) log-rank test for the equality of survival distributions of right censored data.

At each distinct event time the observed number of events in each group is compared with the number expected under the null hypothesis that all groups share the same survival distribution. The weighted sums of these differences form a chi-squared statistic with k - 1 degrees of freedom.

Parameters:

  • x (array like) – Array of observations of the random variables.

  • Z (array like) – Array of group labels for each observation. Any hashable values can be used; the test has len(unique(Z)) - 1 degrees of freedom.

  • c (array like, optional) – Array of censoring flags. 0 is observed and 1 is right censored. Left or interval censored data cannot be used with the log-rank test. If not provided assumes all values are observed.

  • n (array like, optional) – Array of counts for each x. If None assumes each observation is 1.

  • weighting (str, optional) –

    The weighting to use at each event time. One of:

    • ”log-rank”: weight 1 (the standard log-rank test; sensitive to proportional hazards alternatives),

    • ”gehan”: weight r (a.k.a. Gehan-Breslow-Wilcoxon; emphasises early differences),

    • ”tarone-ware”: weight sqrt(r),

    • ”fleming-harrington”: weight S(t-)**rho * (1 - S(t-))**gamma where S is the pooled Kaplan-Meier estimate; rho > 0 emphasises early differences and gamma > 0 late differences.

    Defaults to “log-rank”.

  • rho (scalar, optional) – The parameters of the Fleming-Harrington weighting. Only used when weighting is “fleming-harrington”. Defaults to 0, 0 (which is identical to the log-rank weighting).

  • gamma (scalar, optional) – The parameters of the Fleming-Harrington weighting. Only used when weighting is “fleming-harrington”. Defaults to 0, 0 (which is identical to the log-rank weighting).

  • strata (array like, optional) – Array of stratum labels, one per observation. When supplied the test is stratified: the observed-minus-expected numerators and their variances are accumulated within each stratum (risk sets never cross a stratum boundary) and summed before forming the statistic. This removes a nuisance factor – one whose baseline hazard differs across strata – from the comparison, so groups are only ever compared against others in the same stratum. The degrees of freedom are unchanged (number of groups minus one).

Returns:

result – Object with the chi-squared statistic, the degrees of freedom dof, and the p_value.

Return type:

LogRankResult

Examples

>>> from surpyval import logrank
>>> x = [9, 13, 13, 18, 23, 28, 31, 34, 45, 48, 161,
...      5, 5, 8, 8, 12, 16, 23, 27, 30, 33, 43, 45]
>>> c = [0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1,
...      0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0]
>>> Z = [1] * 11 + [2] * 12
>>> res = logrank(x, Z, c=c)
>>> print(round(res.statistic, 2), round(res.p_value, 4))
3.4 0.0653

References

Klein, J. P. and Moeschberger, M. L. (2003), “Survival Analysis: Techniques for Censored and Truncated Data”, 2nd ed., Chapter 7.

Gray’s test for comparing cumulative incidence functions across groups under competing risks:

surpyval.univariate.competing_risks.nonparametric.gray_test.gray_test(x, e, group, cause, c=None, n=None, rho: float = 0.0) → GrayTestResult

Gray’s k-sample test comparing the cumulative incidence of one cause across groups.

Parameters:

  • x (array_like) – Event/censoring times.

  • e (array_like) – Cause label per observation; None (or matching c == 1) marks a censored observation.

  • group (array_like) – Group label per observation (two or more groups).

  • cause (scalar) – The cause whose cumulative incidence is compared across groups.

  • c (array_like, optional) – Censoring flag (0 event, 1 censored). If omitted it is inferred from e being None.

  • n (array_like, optional) – Count weight per observation (default 1).

  • rho (float, optional) – Weight-family parameter: the per-time weight is (1 - F(t^-))**rho with F the pooled cumulative incidence of cause. 0 (the default) is the standard Gray test.

Returns:

(statistic, df, p_value, cause, groups); df is n_groups - 1 and a small p_value is evidence the groups’ cumulative incidence functions differ.

Return type:

GrayTestResult

Restricted mean survival time

The two-group restricted-mean-survival-time difference (the per-model rmst method lives on the non-parametric model class):

surpyval.univariate.nonparametric.nonparametric.rmst_diff(model_a: NonParametric, model_b: NonParametric, tau: float | None = None, alpha_ci: float = 0.05) → dict

Compare the restricted mean survival time (RMST) of two groups.

The RMST-difference is the standard, assumption-light alternative to the hazard ratio when proportional hazards fails: it needs no PH assumption and reads directly as a difference in expected event-free time within the horizon tau.

Parameters:

  • model_a (NonParametric) – Two fitted non-parametric estimators (e.g. KaplanMeier.fit per group). They must carry a variance estimate (Greenwood).

  • model_b (NonParametric) – Two fitted non-parametric estimators (e.g. KaplanMeier.fit per group). They must carry a variance estimate (Greenwood).

  • tau (scalar, optional) – Common horizon. Defaults to the smaller of the two groups’ largest observed times, so both survival curves are defined up to tau (the standard choice; a tau beyond a group’s support extrapolates that curve at its final value and is flagged is left to the caller).

  • alpha_ci (scalar, optional) – Significance level for the interval and test (default 0.05).

Returns:

{"difference", "se", "lower", "upper", "p_value", "ratio", "rmst_a", "rmst_b", "tau"}. difference is RMST_a - RMST_b; p_value is the two-sided z-test of difference == 0.

Return type:

dict

Examples

>>> import surpyval as sp
>>> a = sp.KaplanMeier.fit([2, 3, 4, 5, 6, 7])
>>> b = sp.KaplanMeier.fit([1, 2, 2, 3, 4, 5])
>>> res = sp.rmst_diff(a, b)
>>> round(res["difference"], 3) > 0
True

Prediction-validation metrics

Right-censored-standard metrics for scoring a predicted survival function (Brier / integrated Brier score and Uno’s time-dependent AUC), plus the helper that builds a predicted-survival matrix from any fitted model exposing sf(x, Z).

Prediction-validation metrics for right-censored survival predictors.

These score a predicted survival function against right-censored outcomes, handling censoring by inverse-probability-of-censoring weighting (IPCW):

  • brier_score() / integrated_brier_score() – the time-dependent Brier score of Graf et al. (1999): the IPCW-weighted squared error between the predicted S(t | Z) and the survival indicator, and its integral over a time grid. The standard scalar summarising calibration and discrimination together (lower is better).

  • auc_td() – Uno’s (2007) cumulative/dynamic time-dependent AUC: discrimination between subjects who have had the event by t and those still event-free, as a function of the horizon t (0.5 is chance, 1 is perfect).

All three are model-agnostic: they take a matrix of predicted survival probabilities. survival_probability() builds that matrix from any fitted model exposing sf(x, Z) (the parametric regression families, CoxPH and the beta.ml forest).

References

Graf, E., Schmoor, C., Sauerbrei, W. and Schumacher, M. (1999), “Assessment and comparison of prognostic classification schemes for survival data”, Statistics in Medicine 18, 2529-2545.

Uno, H., Cai, T., Tian, L. and Wei, L. J. (2007), “Evaluating prediction rules for t-year survivors with censored regression models”, JASA 102, 527-537.

surpyval.metrics.validation.auc_td(x: ArrayLike, c: ArrayLike, risk: ArrayLike, times: ArrayLike, x_train: ArrayLike | None = None, c_train: ArrayLike | None = None) → tuple[NDArray, NDArray]

Uno’s cumulative/dynamic time-dependent AUC (2007).

At each horizon t a case is a subject with the event by t (x_i \le t, \delta_i = 1) and a control is a subject still event-free (x_j > t). The AUC estimates the probability that a case is assigned a higher risk than a control, with cases IPCW-weighted by 1 / \hat G(x_i) to correct for censoring:

\[\widehat{AUC}(t) = \frac{\sum_{i,j} w_i\, \big(I(r_i > r_j) + \tfrac12 I(r_i = r_j)\big)\, I(\text{case}_i)\, I(\text{control}_j)} {\big(\sum_i w_i I(\text{case}_i)\big)\, \big(\sum_j I(\text{control}_j)\big)}.\]
Parameters:

  • x (array_like) – Observed times and censoring flags (0 event, 1 right censored).

  • c (array_like) – Observed times and censoring flags (0 event, 1 right censored).

  • risk (array_like, shape (n_samples, n_times)) – Risk scores where higher means earlier event. For a survival predictor use 1 - survival (see survival_probability()). A single column (or 1-D array) is broadcast across all times.

  • times (array_like) – Horizons at which to evaluate the AUC.

  • x_train (array_like, optional) – Data used to estimate the censoring distribution G. Defaults to the evaluation x / c.

  • c_train (array_like, optional) – Data used to estimate the censoring distribution G. Defaults to the evaluation x / c.

Returns:

times, auc – The horizons and the AUC at each. A horizon with no cases or no controls yields nan.

Return type:

ndarray

surpyval.metrics.validation.brier_score(x: ArrayLike, c: ArrayLike, survival: ArrayLike, times: ArrayLike, x_train: ArrayLike | None = None, c_train: ArrayLike | None = None) → tuple[NDArray, NDArray]

Time-dependent Brier score (Graf et al. 1999).

At each horizon t the Brier score is the IPCW-weighted mean squared error between the survival indicator I(T_i > t) and the predicted survival S(t | Z_i):

\[BS(t) = \frac1n \sum_i \Big[ \frac{S(t\mid Z_i)^2\, I(x_i \le t,\ \delta_i=1)}{\hat G(x_i)} + \frac{(1-S(t\mid Z_i))^2\, I(x_i > t)}{\hat G(t)} \Big],\]

where \(\hat G\) is the Kaplan-Meier estimate of the censoring survival. Subjects censored before t contribute nothing (their status at t is unknown); the IPCW weights correct for that loss. Lower is better.

Parameters:

  • x (array_like) – Observed times and censoring flags (0 event, 1 right censored) of the evaluation set.

  • c (array_like) – Observed times and censoring flags (0 event, 1 right censored) of the evaluation set.

  • survival (array_like, shape (n_samples, n_times)) – Predicted survival S(times[k] | Z_i); see survival_probability().

  • times (array_like) – Horizons at which to score, matching the columns of survival.

  • x_train (array_like, optional) – Data used to estimate the censoring distribution G. Defaults to the evaluation x / c.

  • c_train (array_like, optional) – Data used to estimate the censoring distribution G. Defaults to the evaluation x / c.

Returns:

times, bs – The horizons and the Brier score at each.

Return type:

ndarray

surpyval.metrics.validation.integrated_brier_score(x: ArrayLike, c: ArrayLike, survival: ArrayLike, times: ArrayLike, x_train: ArrayLike | None = None, c_train: ArrayLike | None = None) → float

Integrated Brier score: the Brier score averaged over times.

The trapezoidal integral of brier_score() over the time grid divided by its span. A single number summarising a survival predictor’s accuracy (lower is better); a model that predicts the true S(t | Z) scores below the marginal Kaplan-Meier reference.

surpyval.metrics.validation.survival_probability(model: Any, Z: ArrayLike, times: ArrayLike) → NDArray

Predicted survival matrix S(t | Z_i) from a fitted model.

Parameters:

  • model (object) – Any fitted model exposing sf(x, Z) where x is paired element-wise with the rows of Z (the parametric regression families, CoxPH, the beta.ml forest).

  • Z (array_like) – Covariate matrix, one row per subject.

  • times (array_like) – Evaluation times.

Returns:

survival – survival[i, k] is the predicted survival of subject i at times[k].

Return type:

ndarray, shape (n_samples, n_times)