 |
|
|
Historic accuracy of forecasts March 2005 - 2007
Obtaining a forecast is all very well – but how accurate is it?
Can one rely on it?
We at www.PropertyForecasts.co.uk are always trying to ensure that,
when you buy a forecast from us, you will be buying one that is as reliable
as current technology can make it. Where we cannot justify a forecast
mathematically, we will not provide one.
Annual forecasting accuracy checks
Annually, we carry out a check on our historical forecasting accuracy
performance, across the vast majority of the 8000 postcode sectors in
England & Wales, each on average covering around 2000 dwellings
of up to four property types: detached, semi-detached and terraced houses,
and flats & maisonettes. All series used in our forecasts (including
the background series) are cut back two years, and then forecast to
the current period. The results obtained from the two-year forecasts
are then compared to the latest smoothed Land Registry values, and those
historic forecasts graded according to their accuracy.
Of course, we can never guarantee the future accuracy of any specific
forecast. However, where relationships between historic series are strong,
the forecast tends to continue to be good while that relationship continues.
This implies that, where a forecast has been accurate in the past, that
accuracy tends to continue for a while.
The actual calculation for assessing accuracy of a forecast is highly
complex, as it takes into account not only the accuracy over two years,
but also the match over the two-year period between the forecast and
actual series, and the fluctuations in the two series over the forecast
period.
 |
For example, the “accurate” forecast
in the diagram on the left is not promising a useful forecast for
the future. After all, it is not very helpful to be spot-on accurate
at the end of two years, if over the course of the two years there
has been no match of any kind between forecast and actual! Three
months earlier, and probably also three months later, there is a
wide divergence in values. |
| On the other hand, in the diagram on the right, there
is a difference between the forecast value, and the actual value
achieved at the end of the forecast period. But overall, the match
in shape of the forecast curve with the actual values over the period
is a reasonable confidence-booster that the forecast over the next
period will be much closer to what is actually most likely to happen,
than the exact match in the diagram above. |
 |
 back to previous
Historic forecast accuracy definitions
In summary, the measure of forecast accuracy used by Technical Forecasts
Ltd, and adopted on www.PropertyForecasts.co.uk combines:
- a measure of how accurately the actual and forecast behaviours match
throughout the forecast period (the root mean square (RMS) error)
- the comparative shapes of the curves (volatility; since the volatility
of a data series can render the actual accuracy of a forecast at one
specific time period as a totally misleading indicator of the overall
accuracy of a forecast data set), and
- the ratio between the two (RMS/volatility), measuring accuracy with
respect to the volatility over the period.
- There is no consideration of final accuracy value in the accuracy
classification, as the final point of comparison at the end of the
two years is essentially arbitrary, therefore has no place in the
assessment.
The result of this is a grading system of historic forecast accuracy
of A / B / C / U, where A historically has given the ‘best’
measure of forecast accuracy at any specific point. A ‘U’
classification does not necessarily imply a poor forecast (although
historically it has either been inconsistent, poor, or non-existent
due to lack of data). An additional classification of ‘u’
has been introduced, to identify those series with insufficient data
history to allow a comparative forecast to be made. We cannot assess
the historic accuracy of these series.
Always exercise caution in the use of forecasts for financial planning
– always consult a professional adviser. Remember that past performance
can never guarantee future performance.
Out of interest, the forecast in the first diagram above would probably
be classed as ‘U’, whereas the second would probably be
classed as 'A' or ‘B’.
Historic accuracy figures summary, neighbourhoods ( tested July 2007 )
Comparisons are between forecasts from March 2005 and filtered Land Registry data up to the end of first quarter 2007.
- For houses, 92% or more of historic forecasts are classified A or B; more than 55% are classified as A – a significant increase from last year. For flats & maisonettes, nearly 90% are classified A or B, with 50% classed as A.
- Over 93% of A forecasts are within 15% of the smoothed Land Registry values at March 07 (after two years’ forecast), with nearly half within 5%.
- Around 83% of A and B forecasts are within 15%, and nearly 80% of all forecasts (including U) are within 15% at the end of two years. For flats & maisonettes, this figure has dropped slightly, to 76% of A and B forecasts within 15%, and nearly 70% of all forecasts.
- Less than 1% of all price forecasts are classified as U – but remember that ‘u’ forecasts cannot be assessed for accuracy, since there is usually insufficient data remaining when two years’ worth of history has been removed.
- 'u' forecasts - those which cannot be assessed for accuracy once two years' worth of data has been removed, due to short or erratic data series - account for less than 8% of all houses, but around 15% of flats.
- Around 90% of all forecasts are in the same overall direction as the smoothed Land Registry data. For A category forecasts, this has increased to over 95%.
- Average error at the end of two years for A classified forecasts is -0.3% for detached houses, +1% for semis and terraced houses, and +3% for flats (standard deviation SD around 7.5%). Error for A and B classifications together for detached houses is around -0.4% (SD around 10.7%), and for semis and terraced houses error is less than +2%, with a SD of under 11%. For flats & maisonettes, average error for ‘A’ and ‘B’ classifications was +5.7%, with a SD of 11.7%.
- Overall average error was less than 3% for houses, with a SD of less than 13%, and +7.4% for flats & maisonettes, with a SD of around 15%.
The ‘-’ sign shows underestimates while positive values show over-estimates.
In comparison with last year, correlations in A and B data series are significantly higher, and – for smaller houses – the proportion of A and B forecasts within the 10% error band is significantly higher this year than last. There has been a slight drop in accuracy of forecasts for flats overall, although there has been a 20% increase in numbers of sectors forecast for flats this year, implying that many more series which are quite short are being forecast, which inevitably are likely to be less accurate through a shorter history to judge from.
back to previous
Historic accuracy figures summary, Towns & Cities (tested July 2007)
As with the Neighbourhoods, comparisons are between forecasts from March 2005 and filtered Land Registry data up to the end of first quarter 2007. Averages for Towns & Cities are calculated from postcode sector (neighbourhood) forecasts, using (mainly) the Royal Mail definitions for towns.
- Overall for houses, around 93% of historic forecasts are classified A or B; almost 70% are classified as A, significantly up from last year’s 50% or so.
- Over 96% of A forecasts for houses are within 15% of the filtered Land Registry values at March 07 (after two years’ forecast), with around 55% within 5%.
- For houses, 87-90% of A and B forecasts are within 15%, and around 85% of all forecasts (including U) are within 15% at the end of two years. For flats & maisonettes, this figure has fallen somewhat to 78% of A and B forecasts within 15%, and more than 75% of all forecasts.
- Less than 1% of all price forecasts are classified as U.
- Around 95-97% of all forecasts are in the same overall direction as the smoothed Land Registry data. For A category forecasts, this is around 98%.
- Average error at the end of two years for A classified forecasts is +2.3% or less for houses, and +4.8% for flats (standard deviation SD of 6.5-7%), while that for A and B classifications together for houses is around +3.6% (SD around 9%). For flats & maisonettes, average error for ‘A’ and ‘B’ classifications was +7.4%, with an SD of 10.5%.
- Overall average error was around +2.5-4.5% for houses, and +8.2% for flats & maisonettes, with an SD of around 11-12%.
Once again , the ‘-’ sign shows underestimates while positive values show over-estimates. Although the error figures for A-class forecasts have changed little, the proportion of house price forecasts in the A category has risen hugely since last year.
back to previous
Examples of A, B, C and U category forecasts
Historic forecast accuracy – detailed results for neighbourhoods (postcode sectors)
| Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
63.2% |
31.3% |
5.0% |
0.5% |
100% |
| Average error |
-0.3% |
-0.5% |
3.7% |
30.4% |
-0.03% |
| Standard Deviation of error |
7.8% |
15.1% |
24.9% |
46.6% |
12.5% |
| Proportion positively correlated |
95.2% |
79.9% |
73.9% |
66.7% |
89.2% |
| Average correlation |
61.8% |
45.1% |
29.1% |
21.2% |
53.2% |
| Within 5% after 2 years |
48.6% |
19.3% |
21.7% |
0% |
37.9% |
| Within 10% after 2 years |
78.9% |
39.4% |
25.0% |
0% |
63.5% |
| Within 15% after 2 years |
93.9% |
59.0% |
38.0% |
0% |
79.7% |
|
|
| Semi-Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
55.6% |
36.6% |
7.5% |
0.4% |
100% |
| Average error |
1.0% |
3.5% |
13.0% |
37.6% |
2.9% |
| Standard Deviation of error |
7.4% |
14.6% |
22.5% |
39.1% |
12.9% |
| Proportion positively correlated |
97.6% |
92.4% |
84.4% |
46.4% |
94.5% |
| Average correlation |
64.9% |
51.1% |
36.2% |
1.9% |
57.5% |
| Within 5% after 2 years |
48.6% |
22.9% |
13.6% |
3.6% |
36.4% |
| Within 10% after 2 years |
81.8% |
44.9% |
27.5% |
3.6% |
63.9% |
| Within 15% after 2 years |
95.7% |
65.2% |
39.7% |
7.1% |
80.0% |
|
|
| Terraced houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
55.5% |
36.2% |
8.0% |
0.4% |
100% |
| Average error |
1.1% |
2.5% |
5.4% |
21.4% |
2.0% |
| Standard Deviation of error |
7.3% |
14.6% |
24.1% |
38.8% |
12.8% |
| Proportion positively correlated |
97.4% |
91.8% |
81.5% |
54.8% |
93.9% |
| Average correlation |
65.3% |
49.8% |
33.3% |
9.5% |
56.9% |
| Within 5% after 2 years |
49.6% |
22.6% |
16.1% |
3.2% |
37.0% |
| Within 10% after 2 years |
82.7% |
44.5% |
29.6% |
12.9% |
64.4% |
| Within 15% after 2 years |
95.5% |
64.6% |
40.9% |
16.1% |
79.7% |
|
|
| Flats & Maisonettes: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
50.1% |
38.9% |
10.1% |
1.0% |
100% |
| Average error |
2.9% |
9.3% |
18.2% |
47.7% |
7.4% |
| Standard Deviation of error |
7.7% |
14.7% |
24.3% |
42.9% |
15.1% |
| Proportion positively correlated |
98.1% |
92.7% |
77.0% |
44.0% |
93.4% |
| Average correlation |
69.8% |
53.8% |
29.8% |
-1.4% |
58.9% |
| Within 5% after 2 years |
45.4% |
18.4% |
10.8% |
6.0% |
31.0% |
| Within 10% after 2 years |
78.0% |
35.7% |
19.9% |
8.0% |
55.1% |
| Within 15% after 2 years |
93.5% |
54.3% |
28.1% |
14.0% |
70.1% |
back
to previous
Historic forecast accuracy – detailed results for Towns & Cities
| Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
68.2% |
24.8% |
6.3% |
0.7% |
100% |
| Average error |
1.7% |
3.4% |
3.1% |
54.6% |
2.58% |
| Standard Deviation of error |
6.6% |
13.7% |
22.5% |
20.3% |
11.5% |
| Proportion positively correlated |
97.7% |
91.4% |
83.1% |
71.4% |
95.0% |
| Average correlation |
73.0% |
48.0% |
34.0% |
0.3% |
63.8% |
| Within 5% after 2 years |
57.3% |
24.9% |
30.5% |
0% |
47.1% |
| Within 10% after 2 years |
85.3% |
48.9% |
44.1% |
0% |
73.1% |
| Within 15% after 2 years |
96.7% |
70.8% |
55.9% |
0% |
87.0% |
|
|
| Semi-Detached houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
69.3% |
23.7% |
6.6% |
0.4% |
100% |
| Average error |
2.3% |
7.4% |
16.5% |
29.7% |
4.6% |
| Standard Deviation of error |
6.7% |
13.8% |
10.9% |
32.4% |
11.2% |
| Proportion positively correlated |
99.1% |
95.2% |
79.7% |
50.0% |
96.7% |
| Average correlation |
76.7% |
55.6% |
34.7% |
30.0% |
68.7% |
| Within 5% after 2 years |
53.9% |
21.8% |
18.8% |
0% |
43.7% |
| Within 10% after 2 years |
82.8% |
40.2% |
37.5% |
0% |
69.4% |
| Within 15% after 2 years |
96.7% |
60.3% |
42.2% |
0% |
84.1% |
|
|
| Terraced houses: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
70.3% |
23.5% |
5.7% |
0.5% |
100% |
| Average error |
2.1% |
5.7% |
14.7% |
30.9% |
3.8% |
| Standard Deviation of error |
6.8% |
14.0% |
21.6% |
23.9% |
11.0% |
| Proportion positively correlated |
98.5% |
92.5% |
78.2% |
80% |
95.9% |
| Average correlation |
75.9% |
53.2% |
31.2% |
12.8% |
67.7% |
| Within 5% after 2 years |
54.6% |
26.9% |
18.2% |
0% |
45.7% |
| Within 10% after 2 years |
82.4% |
41.9% |
32.7% |
0% |
69.6% |
| Within 15% after 2 years |
96.0% |
61.2% |
34.6% |
20.0% |
84.0% |
|
|
| Flats & Maisonettes: |
'A' |
'B' |
'C' |
'U' |
ALL |
| Proportion of dataset (%) |
63.2% |
31.1% |
5.7% |
0% |
100% |
| Average error |
4.8% |
12.8% |
22.2% |
|
8.3% |
| Standard Deviation of error |
7.0% |
13.9% |
21.1% |
|
11.9% |
| Proportion positively correlated |
98.9% |
92.6% |
83.3% |
|
96.1% |
| Average correlation |
79.4% |
53.2% |
31.2% |
|
69.7% |
| Within 5% after 2 years |
43.0% |
11.3% |
9.5% |
|
31.2% |
| Within 10% after 2 years |
73.1% |
25.7% |
23.8% |
|
55.5% |
| Within 15% after 2 years |
93.8% |
44.8% |
33.3% |
|
75.1% |
back to previous
|
|
