Back Testing for Turbo Cycles module
edition March, 2012
In the first part of this article you will find the short explanation how to perform Walk Forward Analysis for Turbo Cycles module. In other words, we discuss here the approach that allows to find the optimal cyclical model for any financial instrument. After that you will find more detailed explanation regarding this issue.
Here is step-by-step guide how to find such a model:
- Step #1: downloading price history
We recommend to download 2K price bars at least, better 10K price bars or more. After downloading the price history, follow this:
- Step #2: backtesting
Simply click this button:
and wait while the program performs the analysis.
- Step #3: analyzing the results
After the calculation you get the list like this one:
Here are ranked Turbo Cycles models, i.e. the cyclical models that provide the best (in terms of forecasting abilities) projection line. We are proud to inform you that Timing Solution performs this work fantastically fast (Our first backtesting modules worked not minutes like this one, they worked days and weeks).
We recommend to check the dozen of the top models. In order to do that highlight these models one by one and click this button:
The program runs Turbo Cycle module with appropriate backtested parameters. Watch how this model looks, try to vary LBC position to see how this model works under different conditions.
If this model works OK, leave it; otherwise close this Turbo Cycle module, go back to backtesting module, highlight the next Turbo Cycles model and watch how it works. We recommend to choose 1-3 of the best models.
Very often the cyclical models are tend to inversions, they are marked this way:
In this case Turbo Cycles module displays two projection lines: a regular one and inverted.
- Step #4: how to interpret the results: the model should provide a "good" projection line on both Test #1, Test #2 and Test #3 intervals
When you run this module, the program sets the optimal parameters automatically, so you need just click button and wait while the computer will do this job.
Here is the algorithm of verification of the projection line. It creates three testing pieces of price history (Test #1, Test #2 and Test #3):
These intervals are used to verify the projection line. Let's consider this line:
[Forecast 50] - means that this model allows to forecast 50 bars ahead. If you have downloaded the daily chart, it will be 50 trading days forecast, if you use 1min. chart it will be 50 minutes forecast.
[Test #3 54.3% (+89/-75)] - means that on Test #3 interval this model proved 87 "good" forecast versus 75 "bad" forecast, i.e. in 54.3% of the case we have "good" forecast. The similar information for Test #2 and Test #1 intervals is presented.
The model should provide a "good" projection line on all three intervals, Test #1, Test #2 and Test #3.
You can find the information about these intervals here:
In other words in our particular case the program uses the price history prior 1991 to calculate the turbo cycle models. The rest of 21 years of price history was spitted on three intervals this way:
"Does any life exist after purchasing the software?" -Back Testing is the answer.
Before going into anything, I need to make a statement: I do not know actually how cyclic models work in the stock market. I prefer to tell this clear, straightforward, without any secrets or any hidden agenda. Meanwhile I know how to develop the best cyclic models. It may sound strange, though it is how it is: I can do these models, I can make forecasts with them, - and I do not know how and why they work. I do not think that anybody knows that...
I have spent a lot of time developing/analyzing/back testing procedure for the cycles based models. This is a big part of my life. This is not my hobby, and I am not the only one involved: Ben Price, Mikhail Pavlov, Alexey Kirienko and his fund have helped me on different stages of that quest. In the years 2007-2008, 3-4 computers worked 24 hours a day doing the back testing for different cyclic models. All day along, day and night, day after day... And as a result, I have figured out that I know about the cycles not more than I knew 3-5 years ago. Though, of course, I can provide very nice looking projection lines and equity curves, I still cannot provide the models that I really could trust. I mean the models that will work long after their demonstration. I want you to understand me precisely here: it is one thing to design a model based on some sort of a cycle (and I can do that well enough), and it is a totally different thing to know it all about those models - why and how they work, what happens to them with the time, for what reason/reasons they stop to be in tune with reality, as well as many other things. It is just a technical thing - to program a model, to find the best model from a set, to make a forecast with the model that looks very impressive, to make a good presentation of it. But - the real understanding will show us to what extent we can trust the cycle based model. Without it, the Truth still will be out there.
That is how I get to the idea of back testing. As so far, I believe that it is the only tool that can help us to gain some knowledge of a real forecasting ability of cycle based models.
There are two notes that I would like to make regarding the back testing itself.
First: Very often people use the word "back testing" absolutely incorrectly. Usually they take some model and check it once or twice on a set of past data; then, seeing a very nice looking projection line, they claim some unbelievable performance of their "wonderful" model. It is quite possible to make some money with such a model - if you know when to stop using it. However, it is not what a real back testing is about. It is not enough to have one, or two, ... or ten good examples. To be able to make any significant statement about your model, you need to do much more tests. For real back testing, normally I use 1000 examples as a minimum. It means that to figure out how any model works I compare the projection line created by that model to real price data 1000 times at least; for intraday it is up to 10.000 times. This parameter is called in the program (and in statistics) "a sample size". And this is not all what we have in Back testing module. When testing just one model, the program analyzes simultaneously 360 different modifications of that model (we slightly change the model's parameters). Thus, typically we analyze 36K - 3.6M of projection lines. This is the most complicated task that I ever had encountered during my scientific and software vendor career.
Second: It should be non future leaks back testing. There are thousands ways to calculate impressive projection lines and superb equity curves based on those impressive projection lines. You should know the difference between regular optimization and back testing. Some info regarding this issue can be found here: http://www.timingsolution.com/TS/Articles/bt_no_future_leaks/index.htm
Trend mode versus Cyclic mode versus Quantum mode
So far I am inclined to think (though I do not know that exactly) that the stock market can be in three different states: in trend mode, cyclic mode or in quantum mode. What are they?
Trend mode: The stock market in a trend mode steadily grows or drops. It is looking for a new equilibrium point. The classical economics theory of the supply - demand balance is a good example; speaking about the trend mode, we think here in terms "shares are cheap or too expansive". The most obvious way to analyze trends is to analyze the price chart in respect to the extended price history assuming that the trend is just a fraction of some much more longer term cycle. In this case you should download more price history for the analysis. As an example, here I have downloaded 14 years of weekly price chart, run spectrum module and have revealed two strongest cycles (one of them, 41-months cycle, is the classical Kitchen cycle); then I dragged and dropped these cycles to the Main screen:
Try to use different combinations of cycles. I recommend always keep an eye on major economical cycles: 39-41 months Kitchen and 9-11 years Juglar cycles; I would suggest to include them into your model even if they are not so strong on the spectrogram (i.e. they are not the highest peaks). Though be cautious: this approach does not work always, the economical cycles have their own evolution, and this fact can smash the whole picture. I think fundamental factors (like official reports, investors mood, etc.) play the main role here. Do not forget the Annual cycle as well.
Cyclic mode: Here the equilibrium point has been found, and the price oscillates around it. Cyclic mode sees the stock market movement as a superposition of harmonic waves, this is a smooth movement. We have to catch the right wave here and follow this wave. The main issue here is to reveal the upcoming cycle as early as possible. Turbo Cycles Back testing module has been developed especially for that purpose. Also Wavelet module is helpful here as it allows to visualize the process of "cycle hunting".
One cycle and multi cycle cyclic mode: According to John Ehlers, the stock market is in cyclic mode approximately 30% of time, i.e. only 30% of time we can apply cyclic based models. The remaining 70% of time are related to the market being in trend mode, and cyclic models do not work here. In Timing Solution we use multi cycles model (John Ehlers used one cycle approach), this approach allows to cover more time than 30%. Look at the wavelet diagram below; there I have marked the periods when cycles are active (and we can use them to forecast the price). As you see these cycles cover much more than 30% of time:
The only thing we need to do is to reveal the cycles' appearance as early as it is possible. This is actually what Back testing module does: it searches the cyclic model that reveals new cycles maximum early.
Cyclic mode and fundamental factors: It seems to me fundamental factors play a role of initial push force here, they trigger the appearance of a new cycle. After that, for some period of time, the price oscillates due to its internal rules.
Quantum mode: Quantum models sees the stock market as a process of constructing the stock market "building". We have to see here when a new brick in this building appears, this is the time to make some decision. This is not a trend and not an oscillation, this is a process of creating something new. I think quantum mode is the period when the stock market changes its structure. It is very difficult to find any order here, we just watch how new bricks are add to stock market building. The short period between laying down two bricks is the only time when we can try to forecast the stock market behavior. You can create and back test Quantum models using Trading Strategy Constructor module.
These are just my thoughts in brief. They came to me as a result of several years of research. I back tested different models, watched them, thought about them. As so far, I am sure that these three modes of the stock market do exist, and they somehow interact, though I do not know yet how. There is definitely some missing link there, and this whole issue needs more research.
In this article we deal with cycles only, we try to reveal new cycles as early as possible and make a forecast based on these cycles.
"Digital fortress" - what Back Testing does
In general, the procedure of creating a projection line based on some cycles consists of these steps:
a) Data normalization - usually, before doing any research, analysis, or models, we have to remove the effect of trend. We do it just because cyclic models work better with non trend data. In most cases we use some kind of a relative price oscillator. Here is an example:
A recommended parameter to vary here is the period of the oscillator.
b) Revealing cycles. There is no secret here, it is a classical math problem: we calculate a periodogram. The peaks on the diagram correspond to the most powerful cycles, to the strongest cycles. These cycles are a base of the model that we are creating. For the example above, this is how the periodogram looks like:
We see here 8 peaks. It means that our model will use these 8 cycles - either one by one, or all of them, or any their combination.
c) Extracting the most important cycles among all available cycles. In our example we apply "three cycles model", i.e. the program picks up three of the most influential cycles (3 of 8):
How does the program define what cycles are more influential than others? It is not that simple as taking the highest peaks. To select the most powerful cycles, special algorithms have been developed (my 18 years of scientific experience in the Institute of Nuclear Research in Russia were extremely important for that).
d) Calculating a projection line based on the selected cycles. This is a standard procedure in the program. Here is the projection line based on three cycles above:
e) Comparing this projection line with the real price data.
The main problem is that we have so many variables in this process. We can perform these steps in plenty of different ways: there are many ways to normalize the price, there are a lot of variants to calculate a periodogram, the selection of cycles for the model may be different, etc. What back testing really does is that: it tries different combinations, different models with different parameters - totally different models as well as very similar models with slightly different parameters. And as a result we find the best combination of parameters.
(What I have said about the back testing is just one sentence. In reality it turns out to be a lot, just A LOT of calculations. Just imagine: in 2007 we have started what we called "total back testing" using four computers; these computers worked 24 hours a day non stop. Initially we thought that after one year of this total back testing we would have a set of reliable models. Instead, we have spent more than a year. Many times we had to stop tests as some artifacts were found. And then we did necessary corrections and started again. The artifacts (things that corrupt the result of research) became subjects of many articles that I published on TS website these years. Also, now, when I see some descriptions of "wonderful" forecasting models, I am able to recognize traces of those artifacts.)
To give you understanding of how much research has to be done, consider this: we can obtain one projection line in 10 sec. (Actually, TS Neural Network calculates one projection line for a spectrum model within 30-60 seconds, therefore four computers generate one projection line every 7-15 seconds). It means that within one year we are able to get 3 millions of projection lines! And this is very slow! this is really slow. I did some research and have found that to get the full cyclic portrait of some financial instrument we need to calculate 3-4 millions of projection lines (intraday data). This is a whole year for four computers working non stop! For a small business, this is practically irresolvable problem, it needs applying cloud computing technology which is a very expensive service.
This is how I came to the idea of Turbo Cycles Back Testing module; it uses very smart and very fast algorithms that speed up backtesting procedure drastically - in 700 times! To show you that I am not kidding, please look at this information panel from one real intraday back testing (it has been done for 100K price bars):
The program analyses 17 projection lines per second; so, to get a whole cyclic portrait, it requires less than 3 days of calculations.
For daily data (60 years price history, 15K price bars) the program performs all calculations within 42 minutes and generates more than 100 projection lines per second:
I was very proud of myself when I have developed this module in summer 2008.
How to conduct backtesting
Step #1, fast start: Now I will show how to find the best cyclic model for any financial instrument. I tried to develop this module as simple as possible, its goal is to get the answer quickly, so all details are hidden.
You need to download the price history data for your financial instrument. For intraday data. I recommend to download 50K price bars at least, for daily data it should be at least 3K bars (12 years of price history).
After downloading the price history, follow these steps:
This module is "ready for use", i.e. the program automatically sets all necessary parameters. All you need to do is simply clicking "Run" button:
To verify a workability of cyclic models, the program uses three different intervals: Validate #1, Validate #2 and Testing intervals. Here they are:
In this particular case the program uses last 9000 price bars to verify cyclic models. Last 3000 bars are called "Testing interval", the program checks there how the program forecasts the future. 3000 bars prior Testing interval are called "Validate #2 interval". And other 3000 bars prior Validate #2 interval are called "Validate #1 interval". The idea of back testing procedure is very simple: tested models should work well on both these intervals. This is important - your model should show good performance on both intervals Validate #1 and Validate #2, and it should show good results on Testing interval as well. It is how we lessen the random error.
After the calculations, check "Back Testing Results" tab:
The program shows the ranked list of models and their performance on Validate #1, Validate #2 and Testing intervals. In our example the program informs that our model provides 62% of "good" projection lines on Validate #1 interval, 64% on Validate #2, and 55% on Testing interval. Please remember that 50% means that the model does not work (no correlation found).
When back testing is finished, you can run Turbo Cycles module that corresponds to a highlighted model this way:
You will get projection line for chosen model.
I would recommend to save this model into "Turbo Cycles models library": a) highlight the model you want to save; b) fill out the information form; and c) click "->" button:
After that you can download this model through Turbo Cycles module clicking this button:
Thus you can separate back testing procedure and trading. You can perform back testing when you are not trading. When trading, you will be able to download previously back tested models fast and easy.
Step #2, Inversions: For a long time I could not accept the fact that in many cases INVERTED models provide better solution. I thought that this is some random statistical effect. Finally my friends who helped me with back testing proved me that this is not random, but a fundamental phenomenon.
Very often you will deal with models like this one:
You see, this model provides 58% good projection lines on Validate #1 interval, 55% on Validate #2 interval while only 42% good models on Testing interval. It means that INVERTED model on Testing interval works better.
When back testing procedure reveals inversions, the program displays the projection line this way:
I.e. it shows together the original and inverted projection lines.
Step #3, Parameters: The setting of initial parameters in this module is extremely important issue, so better use these parameters as they are. "By default" the program considers 360 different cyclic models. It means that it covers practically all possible cycles based models. The first versions of this module took into consideration several thousands of models. Then, step by step, I removed close related models. I believe this is important - otherwise we can face with the over-training effect. The models that are left cover a very wide range of cyclic models: MESA, multiframe and classical spectrum. The kind of spectrum is defined by stock market memory parameter, here it is:
When this parameter is low (i.e. stock market memory is equal to 3), we deal with a variation of Auto regression and MESA model, the value 7 corresponds to multiframe spectrum, the big value (12) is related to classical spectrum.
The only parameter I recommend to vary is the length of max cycle:
If you set it to 100-20,0 the program will be more concentrated on searching short term cycles models. I did not research this issue yet.
While the program performs back testing, you can browse preliminary results clicking this button:
Step #4, Models preview: Before using back tested model in your trading, I would recommend to preview it. To do that:
a) run Turbo Cycles module;
b) download from "Turbo Cycles models library" a back tested model;
c) follow menu "Back Testing" -> "Set Random LBC":
d) when you click "OK" button, the program sets Learning Border Cursor (LBC) randomly, and Turbo Cycles module recalculates itself automatically. As only the price history prior LBC is used to calculate a projection line, we can estimate the real workability of this model comparing the projection line and the price after LBC, this is a testing interval (a red zone):
Click this button several times to figure out how this models works.