(Subjective maps here.) Dates used are not in the training data. Only regions that produce >=1 tor and wind and hail report over 1998-2013 (view this mask here) and are in the CONUS are used for verification. "2022" models are trained on data through May 2022; "2021" models through Dec 2021. "2021" models were used for the 2022 SFE and are used by current realtime Nadocast (as of 2022-10-13).
Performance diagrams for Nadocast (NC) day 1 outlooks (triangles) compared to SPC day 1 outlooks (circles), for the 155 days (Sundays) not seen by the models during training and calibration throughout the Jan 2019-May 2021 period. Lines of constant critical success index (CSI) are drawn as light gray curves. The performance for the NC 1% threshold (2% wind and hail) is included as a gray triangle. Significant severe (EF2+, 65kn+, 2in+) 10% thresholds are drawn as large black symbols. The NC sig-severe models are bounded to never output probabilities higher than the NC non-sig probability. To assess statistical significance, for each threshold the difference between the NC and SPC CSI is compared across 1 million bootstrap resamples of the 155 days. The p value for NC CSI exceeding SPC SCI is shown; the NC triangle is drawn with solid fill when p<.05 (i.e. when NC is statistically more skillful).
|Area to the Left of the Performance Diagram (Stairstep geometry, SPC thresholds only)|
|0600 SPC||0Z NC||0Z NC|
|0Z NC 2021||0Z NC 2021|
|1630 SPC||12Z NC||12Z NC|
|12Z NC 2021||12Z NC 2021|
|Mean SPC||Mean NC||Mean NC|
|Mean NC 2021|
|Sig. Wind Adjusted||0.00703||0.0111||0.0115||0.0121||0.0105||0.0166||0.00958||0.0108||0.0141|
Reliability diagrams for Nadocast (NC) day 1 outlooks (triangles) compared to SPC day 1 outlooks (circles), for the 155 days (Sundays) not seen by the models during training and calibration throughout the Jan 2019-May 2021 period. The plots compare the forecast probability of an event to the observed frequency of at least one report within 25 miles. The 95% confidence interval for each bin is shaded, computed by bootstrapping over the 155 days. Ideal reliability is drawn as a dotted diagonal line; above the line is underforecasting, below is overforecasting.
Receiver operator characteristic (ROC) curves for Nadocast (NC) day 1 outlooks (triangles) compared to SPC day 1 outlooks (circles), for the 155 days (Sundays) not seen by the models during training and calibration throughout the Jan 2019-May 2021 period. Area under the curve (AUC) is shown. For rare events (as here), AUC is dominated by POD at the lowest threshold.