ACCURACY OF ADVANCE NAM FORECASTS
(forecast period dependence of NAM forecasts)
June-August 2005

How much does the NAM forecast one and two days in advance ("current+1" and "current+2" days) differ from what is actually predicted on the final ("current") day ? 

The below tables broadly answer this question for selected parameters from the summer of 2005, summed over all days and over the entire region.  The "error" values for each day and location form a bell-shaped curve with a center "bias" value and a spread called the "standard deviation" (2 out of 3 forecasts will have an error smaller than the standard deviation).  The standard deviation indicates the "noise" inherent in the forecasts so the standard deviation should get smaller as the forecast period decreases.  Ideally the bias would be zero and the standard deviation small.  Note that this "error" is "model error" since the reference used is the final (current day) forecast, not an observation - so this says nothing about how the forecasts compare to actual conditions.  We've always known that trying to predict exact conditions days in advance is risky, but I've never before had actual numbers to go on so this provides some idea of the expected error.  Some may find the error in the advance predictions disappointingly large, but that's the nature of weather prediction. 

A positive bias indicates "over-prediction", i.e. the advance prediction was larger than actually forecast for the current day.  Bias values in the tables indicate that the advance day forecast values are generally "over-predicted", i.e. overly optimistic, with the "current+2" day being more optimistic than the "current+1" day.  But the biases are generally small compared to the "noise", represented by the standard deviation.  As expected, the standard deviation is always largest for the "current+2" day.  The "best" advance predictions are those for the CA-NV region, but this is likely biased because a significant fraction of the domain is ocean. 

The numerical values are averaged over the entire region - results will be better/poorer at individual locations within that region.  So I have also created 2D plots for each to show the differences between different subregions.  It's important to recognize that this sampling is relatively small by meteorological standards and this summer's error pattern might well differ - still, these values do indicate the level of forecast error which can be expected from the NAM forecasts for days in advance and the locations where advance forecasts have been more difficult. 

You might also think about psychological factors which can affect one's perception of the advance forecasts.  I decided to do this analysis because too often it seemed that a much-better-than-average forecast 2 days in advance became a more-nearly average forecast by the final forecast - so I wanted to see if analysis would indeed indicate that two-day-ahead forecasts are on the average more optimistic than the final forecast.  Such would be indicated by a large positive bias to the two-day-ahead forecasts and I did not find that to be the case - for my location the 2-day-ahead biases were relatively small.  Then I realized that by focusing on especially good forecasts I was choosing a subset which is more likely to have a positive error, since a positive error will result in a strong day being forecast!  If that error diminishes with time, as is likely due to the quasi-random nature of the error, then the final forecast is likely to have a smaller error than the 2-day-ahead forecast and hence be a less optimistic forecast.  So if one considers only 2-day-ahead forecasts for strong conditions then indeed it is more likely than not that the final forecast will be less optimistic than the first forecast.  This does not mean that all advance forecasts for strong conditions are in error, only that they are more likely to have an over-predicting error than is the average forecast. 

Note:  the plots of BL Depth error (which will be identical to the BL Top error, since terrain height is invariant) is labeled as being "x1000 ft" but is actually in feet.

Note:  producing the 108 plots has been tedious so I have automated it as much as I could and have not checked every plot individually - if you find a discrepancy between the plot title and what the plot is supposed to represent post that information on the Blipmap Forum. 

CA-NV Region   (21Z)
Parameter Current+1
Bias 		Current+1
Std.Dev. 		Current+2
Bias 		Current+2
Std.Dev.
BL Depth +310
2D Plot 		2016
2D Plot 		+475
2D Plot 		2508
2D Plot
Thermal Updraft Velocity +10
2D Plot 		76
2D Plot 		+13
2D Plot 		88
2D Plot
Surface Temperature +0.4
2D Plot 		3.9
2D Plot 		+0.8
2D Plot 		5.7
2D Plot

NW Region   (21Z)
Parameter Current+1
Bias 		Current+1
Std.Dev. 		Current+2
Bias 		Current+2
Std.Dev.
BL Depth +124
2D Plot 		2795
2D Plot 		+174
2D Plot 		3083
2D Plot
Thermal Updraft Velocity +2
2D Plot 		158
2D Plot 		+4
2D Plot 		162
2D Plot
Surface Temperature 0.0
2D Plot 		7.4
2D Plot 		0.0
2D Plot 		9.9
2D Plot

SW Region   (21Z)
Parameter Current+1
Bias 		Current+1
Std.Dev. 		Current+2
Bias 		Current+2
Std.Dev.
BL Depth +158
2D Plot 		3050
2D Plot 		+331
2D Plot 		3668
2D Plot
Thermal Updraft Velocity 0
2D Plot 		148
2D Plot 		0
2D Plot 		163
2D Plot
Surface Temperature +0.3
2D Plot 		5.2
2D Plot 		+0.8
2D Plot 		6.7
2D Plot

GP Region   (21Z)
Parameter Current+1
Bias 		Current+1
Std.Dev. 		Current+2
Bias 		Current+2
Std.Dev.
BL Depth +411
2D Plot 		3784
2D Plot 		+423
2D Plot 		4201
2D Plot
Thermal Updraft Velocity -3
2D Plot 		181
2D Plot 		+1
2D Plot 		179
2D Plot
Surface Temperature +0.1
2D Plot 		7.9
2D Plot 		+0.3
2D Plot 		10.1
2D Plot

OK-TX Region   (21Z)
Parameter Current+1
Bias 		Current+1
Std.Dev. 		Current+2
Bias 		Current+2
Std.Dev.
BL Depth +182
2D Plot 		3016
2D Plot 		+339
2D Plot 		3489
2D Plot
Thermal Updraft Velocity +2
2D Plot 		162
2D Plot 		+9
2D Plot 		176
2D Plot
Surface Temperature +0.2
2D Plot 		5.2
2D Plot 		+0.4
2D Plot 		6.5
2D Plot

NC Region   (18Z)
Parameter Current+1
Bias 		Current+1
Std.Dev. 		Current+2
Bias 		Current+2
Std.Dev.
BL Depth +149
2D Plot 		2994
2D Plot 		+183
2D Plot 		3350
2D Plot
Thermal Updraft Velocity -16
2D Plot 		192
2D Plot 		-11
2D Plot 		197
2D Plot
Surface Temperature -0.4
2D Plot 		6.6
2D Plot 		-0.1
2D Plot 		8.4
2D Plot

SC Region   (18Z)
Parameter Current+1
Bias 		Current+1
Std.Dev. 		Current+2
Bias 		Current+2
Std.Dev.
BL Depth +101
2D Plot 		3154
2D Plot 		+195
2D Plot 		3640
2D Plot
Thermal Updraft Velocity -9
2D Plot 		226
2D Plot 		0
2D Plot 		249
2D Plot
Surface Temperature -0.2
2D Plot 		6.1
2D Plot 		0.0
2D Plot 		7.4
2D Plot

NE Region   (18Z)
Parameter Current+1
Bias 		Current+1
Std.Dev. 		Current+2
Bias 		Current+2
Std.Dev.
BL Depth +138
2D Plot 		2839
2D Plot 		+111
2D Plot 		3207
2D Plot
Thermal Updraft Velocity -1
2D Plot 		185
2D Plot 		0
2D Plot 		195
2D Plot
Surface Temperature -0.2
2D Plot 		6.3
2D Plot 		-0.2
2D Plot 		7.8
2D Plot

SE Region   (18Z)
Parameter Current+1
Bias 		Current+1
Std.Dev. 		Current+2
Bias 		Current+2
Std.Dev.
BL Depth +173
2D Plot 		2246
2D Plot 		+185
2D Plot 		2683
2D Plot
Thermal Updraft Velocity +7
2D Plot 		170
2D Plot 		+23
2D Plot 		195
2D Plot
Surface Temperature -0.1
2D Plot 		4.4
2D Plot 		0.0
2D Plot 		5.7
2D Plot