Skip to contents

Score replayed nowcast quantiles against the settled truth

Source: R/nowcast_score.R
nowcast_score_v1.Rd

Scores a backtest from nowcast_backtest() against the truth from nowcast_truth(). It reports the interval coverage, the revision of the median, the weighted interval score and a 95% interval summary.

Usage

nowcast_score_v1(backtest, truth, by = "horizon", thresholds = c(0.25, 0.5))

Arguments

backtest

A data.table from nowcast_backtest(), with the columns reference, quantile_level and predicted, and optionally as_of and horizon.

truth

A data.table from nowcast_truth(), with the columns reference and truth.

by

The columns to group the scores by.

thresholds

The absolute revisions for the p_gt_<t> columns.

Value

A data.table with one row per group. It has the by columns and these columns:

  • n: the number of scored units,

  • coverage_50, coverage_90: the share of truths from the 0.25 to the 0.75 quantile, and from the 0.05 to the 0.95 quantile,

  • median_signed, median_abs, q05, q95: the median revision, the median absolute revision, and the 5% and 95% quantiles of the revision,

  • p_gt_<t>: the share of absolute revisions above each threshold, such as p_gt_25 for 0.25,

  • wis: the mean WIS over the units,

  • wis_log: the mean WIS after log1p() of each quantile and of the truth,

  • coverage_95: the share of truths from the 0.025 to the 0.975 quantile,

  • width_95_rel_median: the median over the units of (q0.975 - q0.025) / max(truth, 1).

coverage_95 and width_95_rel_median are NA for a group where a unit has no 0.025 or no 0.975 quantile.

Details

A forecast unit is one reference week at one as-of date and horizon. The function scores only the units that have a finite truth and the 0.05, 0.25, 0.5, 0.75 and 0.95 quantiles. The revision of a unit is (median - truth) / truth, over the units with a truth above 0. Every score is a measurement on the replayed weeks, not a property of a method.

The weighted interval score (WIS) follows Bracher et al. (2021). For a unit with truth y and median m:

WIS  = (0.5 * |y - m| + sum_k (alpha_k / 2) * IS_k) / (K + 0.5)
IS_k = (u - l) + (2 / alpha_k) * (l - y) * 1{y < l} + (2 / alpha_k) * (y - u) * 1{y > u}

The sum is over the K central intervals from the quantile pairs at levels p and 1 - p, with p < 0.5. Interval k is from l to u, and alpha_k = 2 * p.

References

Bracher J, Ray EL, Gneiting T, Reich NG (2021). Evaluating epidemic forecasts in an interval format. PLOS Computational Biology 17(2): e1008618. doi:10.1371/journal.pcbi.1008618

See also

vignette("pipeline", package = "csalert"), stage 2, which explains how to read each column.

Examples

truth <- data.table::data.table(reference = "2024-01", truth = 100)
backtest <- data.table::data.table(
  reference = "2024-01", horizon = 0L,
  quantile_level = c(0.025, 0.05, 0.25, 0.5, 0.75, 0.95, 0.975),
  predicted = c(60, 70, 85, 95, 105, 120, 130))
nowcast_score_v1(backtest, truth)
#>    horizon     n coverage_50 coverage_90 median_signed median_abs   q05   q95
#>      <int> <int>       <num>       <num>         <num>      <num> <num> <num>
#> 1:       0     1           1           1         -0.05       0.05 -0.05 -0.05
#>    p_gt_25 p_gt_50      wis    wis_log coverage_95 width_95_rel_median
#>      <num>   <num>    <num>      <num>       <num>               <num>
#> 1:       0       0 3.357143 0.03526364           1                 0.7