 |
|
|
Historic accuracy of forecasts June 2006 - 2009
Obtaining a forecast is all very well – but how accurate is it?
Can one rely on it?
We at www.PropertyForecasts.co.uk are always trying to ensure that,
when you buy a forecast from us, you will be buying one that is as reliable
as current technology can make it. Where we cannot justify a forecast
mathematically, we will not provide one.
Please note: these years include the "boom and bust" period of 2007 to end 2008, so we can expect accuracy to be somewhat reduced, since prices went much higher than expected three years ago, then dropped more dramatically than has been seen in recent years within the experience of the model. It is likely that next year's figures will also be fairly poor. It is worth noting that our four-year forecasts from June 2005 to June 2009 still look pretty accurate - but we are not showing these, as it seems too much like fiddling the figures to look good! We will continue showing the accuracy figures, warts and all.
Annual forecasting accuracy checks
Annually, we carry out a check on our historical forecasting accuracy
performance, across the vast majority of the 8000 postcode sectors in
England & Wales, each on average covering around 2000 dwellings
of up to four property types: detached, semi-detached and terraced houses,
and flats & maisonettes. All series used in our forecasts (including
the background series) are cut back two years, and then forecast to
the current period. The results obtained from the two-year forecasts
are then compared to the latest smoothed Land Registry values, and those
historic forecasts graded according to their accuracy.
Of course, we can never guarantee the future accuracy of any specific
forecast. However, where relationships between historic series are strong,
the forecast tends to continue to be good while that relationship continues.
This implies that, where a forecast has been accurate in the past, that
accuracy tends to continue for a while.
The actual calculation for assessing accuracy of a forecast is highly
complex, as it takes into account not only the accuracy over two years,
but also the match over the two-year period between the forecast and
actual series, and the fluctuations in the two series over the forecast
period.
 |
For example, the “accurate” forecast
in the diagram on the left is not promising a useful forecast for
the future. After all, it is not very helpful to be spot-on accurate
at the end of two years, if over the course of the two years there
has been no match of any kind between forecast and actual! Three
months earlier, and probably also three months later, there is a
wide divergence in values. |
| On the other hand, in the diagram on the right, there
is a difference between the forecast value, and the actual value
achieved at the end of the forecast period. But overall, the match
in shape of the forecast curve with the actual values over the period
is a reasonable confidence-booster that the forecast over the next
period will be much closer to what is actually most likely to happen,
than the exact match in the diagram above. |
 |
 back to previous
Historic forecast accuracy definitions
In summary, the measure of forecast accuracy used by Technical Forecasts
Ltd, and adopted on www.PropertyForecasts.co.uk combines:
- a measure of how accurately the actual and forecast behaviours match
throughout the forecast period (the root mean square (RMS) error)
- the comparative shapes of the curves (volatility; since the volatility
of a data series can render the actual accuracy of a forecast at one
specific time period as a totally misleading indicator of the overall
accuracy of a forecast data set), and
- the ratio between the two (RMS/volatility), measuring accuracy with
respect to the volatility over the period.
- There is no consideration of final accuracy value in the accuracy
classification, as the final point of comparison at the end of the
two years is essentially arbitrary, therefore has no place in the
assessment.
The result of this is a grading system of historic forecast accuracy
of A / B / C / U, where A historically has given the ‘best’
measure of forecast accuracy at any specific point. A ‘U’
classification does not necessarily imply a poor forecast (although
historically it has either been inconsistent, poor, or non-existent
due to lack of data). An additional classification of ‘u’
has been introduced, to identify those series with insufficient data
history to allow a comparative forecast to be made. We cannot assess
the historic accuracy of these series.
Always exercise caution in the use of forecasts for financial planning
– always consult a professional adviser. Remember that past performance
can never guarantee future performance.
Out of interest, the forecast in the first diagram above would probably
be classed as ‘U’, whereas the second would probably be
classed as 'A' or ‘B’.
Historic accuracy figures summary, neighbourhoods ( tested Sept 2009 )
Comparisons are between actual forecasts we made in Aug 2006, from Land Registry data to June 2006, and TFL-filtered Land Registry data up to the end of second quarter 2009.
- For semi-detached and terraced houses, 78% or more of historic forecasts are classified A or B over the three-year period. For detached houses, this has dropped a little to 75%, while for flats & maisonettes, nearly 65% are classified A or B. This is around 10% lower than last year, due mainly to the higher volatility in prices over the 2007-8 period than was forecast in mid-2006.
- For houses, around 15% are classified as A, while for flats and maisonettes this figure stands at over 10%. Again, this is significantly lower than last year, due to the consideration taken by TFL of volatility in the stringent criteria we use for 'A' forecasts.
- More than 50% of A forecasts are within 15% of the smoothed Land Registry values at June 09 (after the three years’ forecast), with around 12% within 5%. This is disappointing, and is not likely to improve significantly until 2011.
- Around one-third of A and B forecasts are within 15%, and about 30% of all forecasts (including U) for houses are within 15% at the end of three years. For flats & maisonettes, this figure has dropped slightly, to around one quarter of all forecasts within 15%.
- In spite of the difficult forecasting period, we still find that less than 3% of all house price forecasts are classified as U, while for flats (the most difficult to forecast, since most series are quite short) this has risen to nealy 9%.
Please remember that ‘u’ forecasts cannot be assessed for accuracy, since there is usually insufficient data remaining when three years’ worth of history has been removed to allow a forecast to be made.
- 'u' forecasts - those which cannot be assessed for accuracy once three years' worth of data has been removed, due to short or erratic data series - account for less than 8% of all houses, but around 15% of flats.
- Average correlation - the measure of how closely the forecast series matches the actual data - has again dropped markedly over the last year, now standing at -0.2 for all properties, reflecting the huge effect of the 'credit crunch' on house prices over the last 2 years, when prices dropped so dramatically after the price bubble.
- Average error at the end of three years for A classified forecasts is around +14% for all property types (standard deviation SD slightly improved, at under 8%). Error for A and B classifications together for all property types is around +20%. This year, errors are predominantly positive, implying that growths over the three years have been over-estimated - again, a reflection of the reductions seen during the credit crunch from positions taken during early days of the 'bubble'. SDs remain broadly similar, at around 13%.
- Overall average error was around +24% for houses, with a SD of around 18%, and +28% for flats & maisonettes, with a SD of around 24%.
A ‘-’ sign in error values shows underestimates while positive values show over-estimates.
In comparison with last year, average correlations in all data series are significantly lower, and the proportion of A and B forecasts within the 15% error band is also significantly lower this year than last (around 33%, as opposed to around 75% previously). Obviously, this is very disappointing, after so many years of improving accuracy, but the severity of the credit crunch was much greater than most predicted at the time.
Results next year may also be disappointing, as the forecasts made in June 2007 were from an inflated base in the 2Q07 Land Registry data - the high point in the market. It might be more indicative if, for a single year next year, we re-introduce a two-year accuracy measure, to allow the greater experience of the model to be reflected in forecasts from the slightly more experienced base-point of 2Q08.
There has been a drop in accuracy of forecasts for flats overall, indicating that the effects of the credit crunch are probably felt even more profoundly in the market for flats and maisonettes than that for houses.
back to previous
Historic accuracy figures summary, Towns & Cities (tested Sept 2009)
As with the Neighbourhoods, comparisons are between forecasts we made in Aug 2006, from Land Registry data to June 2006, and filtered Land Registry data up to the end of second quarter 2009. Averages for Towns & Cities are calculated from postcode sector (neighbourhood) forecasts, using (mainly) the Royal Mail definitions for towns and averaging relevant sector values .
- Overall for houses, around 81-86% of three-year historic forecasts are classified A or B, down around 10% from last year; 25-29% are classified as A, significantly down from last year’s 64-70%.
- Around 38 % of A forecasts for houses are within 15% of the averaged filtered Land Registry values at June 2009, well down on last year, with only around 7% within 5%.
- For houses, 27-28% of A and B forecasts are within 15%, and 25-26% of all forecasts (including U) are within 15% at the end of three years. For flats & maisonettes, this figure has fallen somewhat to 23% of A and B forecasts within 15%, and 20% of all forecasts.
- Less than 2% of all house price forecasts are classified as U; for flats & maisonettes, around 5.3% are classified U.
- Around 26% of all forecasts are in the same overall direction as the smoothed Land Registry data. For A category forecasts, this is 33-37%.
- Average error at the end of three years for A classified forecasts is around +16% (standard deviation SD of 6.5%), while that for A and B classifications together stands at around 21% (SD around 11.4%). For flats & maisonettes, average error for ‘A’ and ‘B’ classifications was +22%, with an SD of 11.7%.
- Overall average error was between +22.8 and +23.7% for houses (SD 15-16%), and +28% for flats & maisonettes, with a somewhat higher SD of around 20%.
Once again , a ‘-’ sign against an error value would show an underestimate, while positive values show over-estimates. All values indicate overestimates, as one would expect in the boom-and-bust climate prevailing over the last three years.
back to previous
Examples of A, B, C and U category forecasts
Historic forecast accuracy over three years – detailed results for neighbourhoods (postcode sectors)
| Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
14.5% |
61.2% |
21.8% |
2.5% |
100% |
| Average error |
13.7% |
21.2% |
34.1% |
69.7% |
24.1% |
| Standard Deviation of error |
7.9% |
13.3% |
22.4% |
76.2% |
18.6% |
| Proportion positively correlated |
38.0% |
23.2% |
21.4% |
17.7% |
24.8% |
| Average correlation |
-0.08 |
-0.23 |
-0.22 |
-0.32 |
-0.21 |
| Within 5% after 3 years |
12.5% |
9.5% |
5.4% |
0% |
8.8% |
| Within 10% after 3 years |
31.3% |
18.6% |
11.1% |
0.6% |
18.3% |
| Within 15% after 3 years |
53.3% |
29.7% |
17.6% |
1.9% |
|
|
|
| Semi-Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
17.5% |
60.8% |
19.0% |
2.8% |
100% |
| Average error |
14.1% |
21.2% |
34.6% |
68.3% |
23.8% |
| Standard Deviation of error |
7.8% |
13.2% |
22.6% |
76.0% |
18.5% |
| Proportion positively correlated |
36.6% |
23.9% |
25.4% |
20.5% |
26.3% |
| Average correlation |
-0.09 |
-0.21 |
-0.20 |
-0.34 |
-0.19 |
| Within 5% after 3 years |
11.3% |
8.3% |
5.7% |
0.5% |
8.1% |
| Within 10% after 3 years |
30.1% |
18.0% |
10.8% |
1.0% |
18.3% |
| Within 15% after 3 years |
51.3% |
29.3% |
17.8% |
2.0% |
30.2% |
|
|
| Terraced houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
15.3% |
62.8% |
20.0% |
1.9% |
100% |
| Average error |
14.1% |
21.5% |
32.5% |
54.6% |
23.2% |
| Standard Deviation of error |
8.0% |
13.5% |
24.4% |
68.9% |
18.1% |
| Proportion positively correlated |
37.9% |
25.0% |
24.9% |
26.4% |
27.0% |
| Average correlation |
-0.08 |
-0.21 |
-0.22 |
-0.23 |
-0.19 |
| Within 5% after 3 years |
12.3% |
8.2% |
6.3% |
2.1% |
8.3% |
| Within 10% after 3 years |
28.0% |
17.9% |
13.5% |
2.1% |
18.3% |
| Within 15% after 3 years |
53.7% |
28.5% |
|
3.6% |
30.2% |
|
|
| Flats & Maisonettes: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
|
54.3% |
26.3% |
8.8% |
100% |
| Average error |
13.8% |
21.8% |
34.3% |
67.0% |
28.2% |
| Standard Deviation of error |
7.8% |
13.4% |
26.0% |
40.7% |
24.8% |
| Proportion positively correlated |
51.2% |
28.7% |
27.5% |
19.3% |
30.0% |
| Average correlation |
0.02 |
-0.17 |
-0.21 |
-0.29 |
-0.17 |
| Within 5% after 3 years |
10.5% |
8.5% |
5.4% |
0.4% |
7.2% |
| Within 10% after 3 years |
28.2% |
17.7% |
12.4% |
1.1% |
16.0% |
| Within 15% after 3 years |
52.4% |
27.5% |
18.4% |
2.4% |
25.5% |
back
to previous
Historic forecast accuracy – detailed results for Towns & Cities
| Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
24.6% |
56.7% |
16.8% |
1.9% |
100% |
| Average error |
15.9% |
23.0% |
30.9% |
62.7% |
23.4% |
| Standard Deviation of error |
6.7% |
11.9% |
24.0% |
38.3% |
16.4% |
| Proportion positively correlated |
32.8% |
22.9% |
24.8% |
11.1% |
25.4% |
| Average correlation |
-0.10 |
-0.23 |
-0.20 |
-0.24 |
-0.2 |
| Within 5% after 3 years |
7.7% |
5.5% |
5.6% |
0% |
6.0% |
| Within 10% after 3 years |
17.5% |
12.2% |
12.4% |
0% |
13.3% |
| Within 15% after 3 years |
38.7% |
21.8% |
18.6% |
5.6% |
25.1% |
|
|
| Semi-Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
29.1% |
57.2% |
12.9% |
0.8% |
100% |
| Average error |
15.8% |
23.3% |
34.1% |
67.8% |
22.8% |
| Standard Deviation of error |
6.5% |
12.4% |
22.3% |
40.1% |
14.9% |
| Proportion positively correlated |
34.4% |
24.1% |
27.5% |
0% |
27.4% |
| Average correlation |
-0.11 |
-0.21 |
-0.21 |
-0.48 |
-0.18 |
| Within 5% after 3 years |
6.7% |
4.7% |
7.5% |
0% |
5.6% |
| Within 10% after 3 years |
18.2% |
12.1% |
10.8% |
0% |
13.6% |
| Within 15% after 3 years |
37.8% |
23.7% |
15.0% |
0% |
26.5% |
|
|
| Terraced houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
25.9% |
58.1% |
14.0% |
|
100% |
| Average error |
16.2% |
23.2% |
34.4% |
62.4% |
23.7% |
| Standard Deviation of error |
6.4% |
12.2% |
23.4% |
37.7% |
16.2% |
| Proportion positively correlated |
32.8% |
22.7% |
23.9% |
26.3% |
25.6% |
| Average correlation |
-0.10 |
-0.22 |
-0.22 |
-0.29 |
-0.19 |
| Within 5% after 3 years |
5.7% |
6.5% |
7.5% |
0% |
6.3% |
| Within 10% after 3 years |
16.2% |
13.4% |
14.2% |
0% |
13.9% |
| Within 15% after 3 years |
38.9% |
23.3% |
18.7% |
0% |
26.2% |
|
|
| Flats & Maisonettes: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
17.0% |
58.0% |
19.7% |
5.3% |
100% |
| Average error |
15.5% |
24.2% |
39.8% |
69.9% |
28.2% |
| Standard Deviation of error |
6.7% |
12.1% |
22.5% |
32.8% |
20.2% |
| Proportion positively correlated |
37.1% |
27.2% |
18.1% |
23.1% |
26.9% |
| Average correlation |
-0.02 |
-0.18 |
-0.29 |
-0.31 |
-0.18 |
| Within 5% after 3 years |
7.3% |
4.7% |
4.9% |
0% |
4.9% |
| Within 10% after 3 years |
17.7% |
12.3% |
10.4% |
0% |
12.2% |
| Within 15% after 3 years |
37.1% |
19.4% |
11.1% |
0% |
19.7% |
back to previous
|
|
