 |
|
|
Historic accuracy of forecasts June 2005 - 2008
Obtaining a forecast is all very well – but how accurate is it?
Can one rely on it?
We at www.PropertyForecasts.co.uk are always trying to ensure that,
when you buy a forecast from us, you will be buying one that is as reliable
as current technology can make it. Where we cannot justify a forecast
mathematically, we will not provide one. Please note: this year, we are introducing three-year accuracy checks, to replace our earlier two-year values - a much more difficult proposition!
Annual forecasting accuracy checks
Annually, we carry out a check on our historical forecasting accuracy
performance, across the vast majority of the 8000 postcode sectors in
England & Wales, each on average covering around 2000 dwellings
of up to four property types: detached, semi-detached and terraced houses,
and flats & maisonettes. All series used in our forecasts (including
the background series) are cut back two years, and then forecast to
the current period. The results obtained from the two-year forecasts
are then compared to the latest smoothed Land Registry values, and those
historic forecasts graded according to their accuracy.
Of course, we can never guarantee the future accuracy of any specific
forecast. However, where relationships between historic series are strong,
the forecast tends to continue to be good while that relationship continues.
This implies that, where a forecast has been accurate in the past, that
accuracy tends to continue for a while.
The actual calculation for assessing accuracy of a forecast is highly
complex, as it takes into account not only the accuracy over two years,
but also the match over the two-year period between the forecast and
actual series, and the fluctuations in the two series over the forecast
period.
 |
For example, the “accurate” forecast
in the diagram on the left is not promising a useful forecast for
the future. After all, it is not very helpful to be spot-on accurate
at the end of two years, if over the course of the two years there
has been no match of any kind between forecast and actual! Three
months earlier, and probably also three months later, there is a
wide divergence in values. |
| On the other hand, in the diagram on the right, there
is a difference between the forecast value, and the actual value
achieved at the end of the forecast period. But overall, the match
in shape of the forecast curve with the actual values over the period
is a reasonable confidence-booster that the forecast over the next
period will be much closer to what is actually most likely to happen,
than the exact match in the diagram above. |
 |
 back to previous
Historic forecast accuracy definitions
In summary, the measure of forecast accuracy used by Technical Forecasts
Ltd, and adopted on www.PropertyForecasts.co.uk combines:
- a measure of how accurately the actual and forecast behaviours match
throughout the forecast period (the root mean square (RMS) error)
- the comparative shapes of the curves (volatility; since the volatility
of a data series can render the actual accuracy of a forecast at one
specific time period as a totally misleading indicator of the overall
accuracy of a forecast data set), and
- the ratio between the two (RMS/volatility), measuring accuracy with
respect to the volatility over the period.
- There is no consideration of final accuracy value in the accuracy
classification, as the final point of comparison at the end of the
two years is essentially arbitrary, therefore has no place in the
assessment.
The result of this is a grading system of historic forecast accuracy
of A / B / C / U, where A historically has given the ‘best’
measure of forecast accuracy at any specific point. A ‘U’
classification does not necessarily imply a poor forecast (although
historically it has either been inconsistent, poor, or non-existent
due to lack of data). An additional classification of ‘u’
has been introduced, to identify those series with insufficient data
history to allow a comparative forecast to be made. We cannot assess
the historic accuracy of these series.
Always exercise caution in the use of forecasts for financial planning
– always consult a professional adviser. Remember that past performance
can never guarantee future performance.
Out of interest, the forecast in the first diagram above would probably
be classed as ‘U’, whereas the second would probably be
classed as 'A' or ‘B’.
Historic accuracy figures summary, neighbourhoods ( tested Sept 2008 )
Comparisons are between actual forecasts we made in Aug 2005, from Land Registry data to June 2005, and filtered Land Registry data up to the end of second quarter 2008.
- For semi-detached and terraced houses, 90% or more of historic forecasts are classified A or B over the three-year period. For detached houses, this has dropped a little to 85%, while for flats & maisonettes, nearly 78% are classified A or B. For houses, around 45% are classified as A (nearly 65% for terraced), while for flats and maisonettes this figure stands at over 37%.
- Around 90% of A forecasts are within 15% of the smoothed Land Registry values at June 08 (after three years’ forecast), with 42-45% within 5%.
- Around three-quarters of A and B forecasts are within 15%, and 67-76% of all forecasts (including U) for houses are within 15% at the end of three years. For flats & maisonettes, this figure has dropped slightly, to nearly 60% of all forecasts within 15%.
- Less than 1.5% of all house price forecasts are classified as U, while for flats (the most difficult to forecast, since most series are quite short) this has risen to nealy 6%. Please remember that ‘u’ forecasts cannot be assessed for accuracy, since there is usually insufficient data remaining when three years’ worth of history has been removed.
- 'u' forecasts - those which cannot be assessed for accuracy once three years' worth of data has been removed, due to short or erratic data series - account for less than 8% of all houses, but around 15% of flats.
- Around 96% of all forecasts are in the same overall direction as the smoothed Land Registry data. For A category forecasts, this has increased to over 99%. Average correlation - the measure of how closely the forecast series matches the actual data - has dropped markedly over the last year, now standing at 0.65-0.69 for all properties, reflecting the effect of the 'credit crunch' on house prices since a year ago.
- Average error at the end of three years for A classified forecasts is under +4% for all property types (standard deviation SD around 8%). Error for A and B classifications together for detached houses and flats is around +5%, while for semis and terraced houses error is around +3.5%. This year, errors are predominantly positive, implying that growths over the three years have been over-estimated - again, a likely reflection of the reductions seen during the credit crunch. SDs have also increased slightly, to around 12% for houses and 13.5% for flats, for the same likely reason.
- Overall average error was around +4% for semis and terraced houses, with a SD of around 15%, over +7% for detached (SD over 16%), and +9.6% for flats & maisonettes, with a SD of around 24%.
A ‘-’ sign in error values shows underestimates while positive values show over-estimates.
In comparison with last year (where we were showing only 2-year historic accuracy data), average correlations in A and B data series are significantly higher, but the proportion of A and B forecasts within the 15% error band is significantly lower this year than last (around 75%, as opposed to over 80% previously). Perhaps this indicates that we were right in the shape and direction of many of our forecasts from three years ago, but not quite as close as we might have been in real terms!
There has been a significant drop in accuracy of forecasts for flats overall, indicating that the effects of the credit crunch are probably felt more profoundly in the market for flats and maisonettes than that for houses.
back to previous
Historic accuracy figures summary, Towns & Cities (tested Sept 2008)
As with the Neighbourhoods, comparisons are between forecasts we made in Aug 2005, from Land Registry data to June 2005, and filtered Land Registry data up to the end of second quarter 2008. Averages for Towns & Cities are calculated from postcode sector (neighbourhood) forecasts, using (mainly) the Royal Mail definitions for towns and averaging relevant sector values .
- Overall for houses, around 91-96% of three-year historic forecasts are classified A or B; 64-70% are classified as A, slightly down from last year’s 70% or so for two-year forecasts.
- Around 90 % of A forecasts for houses are within 15% of the filtered Land Registry values at June 2008 (after three years’ forecast), with around half within 5%.
- For houses, 78-84% of A and B forecasts are within 15%, and 73-81% of all forecasts (including U) are within 15% at the end of three years. For flats & maisonettes, this figure has fallen somewhat to 73% of A and B forecasts within 15%, and two thirds of all forecasts.
- Less than 1% of all house price forecasts are classified as U; for flats & maisonettes, around 2% are classified U.
- Around 98% of all forecasts are in the same overall direction as the smoothed Land Registry data. For A category forecasts, this is over 99%.
- Average error at the end of three years for A classified forecasts is +3% for semis and terraced houses, and+5% for detached houses (standard deviation SD of 7.5-8%), while that for A and B classifications together varies between +2% and +6% (SD around 11%). For flats & maisonettes, average error for ‘A’ and ‘B’ classifications was +6.2%, with an SD of 12.4%.
- Overall average error was between +2.5 and +7.3% for houses (SD 12-13.7%), and +8.4% for flats & maisonettes, with a somewhat higher SD of around 17%.
Once again , a ‘-’ sign against an error value would show an underestimate, while positive values show over-estimates. The surprising factor this year is that average correlations over the last three years - which for neighbourhoods are somewhat worse than previously - are significantly improved for towns, at around 0.75 overall.
back to previous
Examples of A, B, C and U category forecasts
Historic forecast accuracy over three years – detailed results for neighbourhoods (postcode sectors)
| Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
42.6% |
43.1% |
12.9% |
1.4% |
100% |
| Average error |
4.0% |
6.7% |
16.6% |
40.0% |
7.3% |
| Standard Deviation of error |
8.3% |
15.5% |
91.0% |
54.6% |
16.4% |
| Proportion positively correlated |
99.5% |
96.2% |
73.9% |
81.2% |
96.7% |
| Average correlation |
0.75 |
0.62 |
0.53 |
0.39 |
0.65 |
| Within 5% after 3 years |
42.4% |
21.5% |
9.8% |
0% |
28.3% |
| Within 10% after 3 years |
71.8% |
42.7% |
19.0% |
1.2% |
51.2% |
| Within 15% after 3 years |
89.7% |
60.0% |
28.7% |
5.9% |
|
|
|
| Semi-Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
46.7% |
43.3% |
9.2% |
0.7% |
100% |
| Average error |
2.7% |
3.9% |
9.9% |
47.3% |
4.2% |
| Standard Deviation of error |
8.0% |
16.1% |
27.9% |
62.3% |
15.6% |
| Proportion positively correlated |
99.1% |
96.9% |
92.3% |
80.8% |
97.4% |
| Average correlation |
0.75 |
0.63 |
0.53 |
0.39 |
0.67 |
| Within 5% after 3 years |
45.3% |
20.8% |
9.8% |
5.8% |
31.1% |
| Within 10% after 3 years |
73.5% |
41.3% |
19.0% |
7.7% |
55.4% |
| Within 15% after 3 years |
91.8% |
61.2% |
28.7% |
9.6% |
72.2% |
|
|
| Terraced houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
64.2% |
30.6% |
4.7% |
0.5% |
100% |
| Average error |
4.0% |
3.5% |
4.8% |
48.4% |
3.9% |
| Standard Deviation of error |
8.0% |
17.0% |
32.1% |
42.9% |
14.1% |
| Proportion positively correlated |
99.1% |
92.5% |
80.0% |
63.6% |
96.0% |
| Average correlation |
0.75 |
0.53 |
0.29 |
0.23 |
0.66 |
| Within 5% after 3 years |
43.3% |
19.1% |
4.2% |
0% |
33.8% |
| Within 10% after 3 years |
73.5% |
33.2% |
7.4% |
0% |
57.6% |
| Within 15% after 3 years |
91.8% |
53.9% |
|
9.1% |
76.2% |
|
|
| Flats & Maisonettes: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
|
40.1% |
16.5% |
5.8% |
100% |
| Average error |
3.7% |
6.1% |
18.4% |
47.9% |
9.6% |
| Standard Deviation of error |
8.4% |
16.7% |
29.0% |
54.9% |
23.8% |
| Proportion positively correlated |
99.1% |
96.9% |
90.7% |
85.7% |
96.0% |
| Average correlation |
0.80 |
0.67 |
0.56 |
0.48 |
0.69 |
| Within 5% after 3 years |
42.1% |
18.3% |
5.2% |
0.7% |
24.1% |
| Within 10% after 3 years |
72.5% |
37.7% |
12.9% |
1.1% |
44.6% |
| Within 15% after 3 years |
89.1% |
56.8% |
20.5% |
1.4% |
59.8% |
back
to previous
Historic forecast accuracy – detailed results for Towns & Cities
| Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
64.1% |
27.4% |
7.9% |
0.7% |
100% |
| Average error |
5.2% |
8.6% |
17.4% |
36.5% |
7.3% |
| Standard Deviation of error |
7.6% |
15.1% |
26.4% |
49.2% |
13.7% |
| Proportion positively correlated |
100% |
96.8% |
89.0% |
66.7% |
98.0% |
| Average correlation |
0.81 |
0.62 |
0.46 |
0.45 |
0.73 |
| Within 5% after 3 years |
43.5% |
18.3% |
11.0% |
16.7% |
33.8% |
| Within 10% after 3 years |
70.8% |
36.9% |
17.8% |
16.7% |
57.0% |
| Within 15% after 3 years |
88.1% |
54.4% |
24.7% |
16.7% |
73.4% |
|
|
| Semi-Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
70.6% |
25.6% |
3.7% |
0.1% |
100% |
| Average error |
3.1% |
6.8% |
3.3% |
59.6% |
4.1% |
| Standard Deviation of error |
7.4% |
17.2% |
25.1% |
0% |
12.0% |
| Proportion positively correlated |
99.7% |
97.0% |
94.1% |
100.0% |
98.8% |
| Average correlation |
0.82 |
0.66 |
0.51 |
0.58 |
0.77 |
| Within 5% after 3 years |
51.1% |
15.3% |
8.8% |
0% |
40.3% |
| Within 10% after 3 years |
79.0% |
33.6% |
29.4% |
0% |
65.5% |
| Within 15% after 3 years |
92.8% |
56.2% |
47.1% |
0% |
81.6% |
|
|
| Terraced houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
65.2% |
29.0% |
5.4% |
|
100% |
| Average error |
2.8% |
0.6% |
8.4% |
6.0% |
2.5% |
| Standard Deviation of error |
7.4% |
15.2% |
32.8% |
53.0% |
13.2% |
| Proportion positively correlated |
99.7% |
97.4% |
88.0% |
33.3% |
98.2% |
| Average correlation |
0.82 |
0.64 |
0.40 |
0.05 |
0.74 |
| Within 5% after 3 years |
49.0% |
18.4% |
12.0% |
33.3% |
38.0% |
| Within 10% after 3 years |
81.5% |
41.2% |
16.0% |
33.3% |
66.1% |
| Within 15% after 3 years |
93.8% |
62.6% |
28.0% |
33.3% |
81.0% |
|
|
| Flats & Maisonettes: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
55.6% |
32.8% |
9.4% |
|
100% |
| Average error |
4.1% |
9.9% |
21.5% |
39.3% |
8.4% |
| Standard Deviation of error |
8.1% |
16.7% |
26.8% |
44.2% |
16.9% |
| Proportion positively correlated |
99.5% |
97.4% |
95.4% |
93.3% |
98.3% |
| Average correlation |
0.85 |
0.70 |
0.65 |
0.63 |
0.78 |
| Within 5% after 3 years |
44.4% |
13.3% |
1.5% |
0% |
29.2% |
| Within 10% after 3 years |
71.5% |
31.4% |
6.2% |
6.67% |
50.8% |
| Within 15% after 3 years |
89.3% |
46.0% |
18.5% |
6.67% |
66.6% |
back to previous
|
|
