EdgeWare FastBreak Pro Version 6.5 - Manual (Page 69)

69
single fund system has the system with the best OS result but is also the system with the
worst OS result. What is interesting is that the “hold 3” systems actually has the best
average performance, 32% on average, and the worst system had a very respectable
24%/year return.
OS Testing Period
How much data should be reserved for Out-of-Sample (OS) testing, and how often should
a system be re-optimized? There is not a good general rule. One always wants to use as
much recent data as possible in building systems, but this must be weighed against
reserving enough data for OS testing. We recommend that at least one year be reserved
for the OS testing. The following chart was made from a study to try to determine how
allowing IS data to go “stale” will affect performance. Four FastBreak optimization runs
were made using the Fidelity Select family from the first study. All runs used an IS start
date of 5/12/1989 but with different IS end dates.
The runs ended with a date of
9/30/1998, 3/30/1998, 9/30/1997, and the final run with a date of 9/30/1996. The
generations with the best OS results were saved into DFT files. The DFT files were
loaded into FastBreak with a start date of 5/12/1989 and an end date of 9/30/1999. The
equity curves from all 40 strategies (4 time periods, 10 best strategies per period) were
looked at in FastTrack for the period of 9/30/1998 to 9/30/1999. The following table was
made to show how the systems results were affected for this one year of OS returns as the
data became more out of date:
"Stale"
months OS
Return,
%/year
0 61.3
6 61.5
12 65.8
24 48.6
The “Stale” column is the time period from the end of the IS optimization period until the
start of the OS test period (9/30/1998 in this case). The results are uniformly excellent,
and there is no effect of the IS period ending at least 12 months in the past. Actually, the
results improve slightly for the data that is 12 months old. It is not until the IS data is two
years out of date that we see a decline.
The 9/30/1998 date is near the low of a severe market sell off. We wanted to look at an
additional six months prior to this time period that captured the market sell off. A second
test was examined looking at an 18 month period (as opposed to the 12 month period in
the first example) from 6/30/1998 to 9/30/1999. This required dropping one of the test
cases (the case which had the IS period ending on 9/30/1998). Here are the average
return results for the top ten systems for each of the optimization runs: