Recursive formula bias

20230804: new info added at the bottom to highlight how crucial this bias can be

The term “lookahead bias,” which refers to the situation when your bot is looking at future data at some point during a backtest and it affects its entry and/or exit decisions, should be familiar to the majority of you. But what is a “Recursive loop bias”?

An example makes it simpler to explain. Imagine that I have a very basic indicator called steps. The first row’s value is always 0, while the following rows’ values are equal to the value of the previous row’s plus 1. Let’s imagine I want to backtest a strategy that I have running on the 1d timeframe for the 20230101-20230201 timerange. My steps value at the 20230101 candle is therefore 0, and my steps value at the 20230131 candle is 30, respectively.

What if, however, my backtest range is 20230115-20230201? My steps value at the 20230115 candle is therefore 0, and my steps value at the 20230131 candle will be 16. The backtest results for the two timeranges may differ slightly because of the different step values for one same candle. I refer to this as Recursive loop bias.

Since the first row always changes after the arrival of new candles, this bias might also effect your dry/live run. As a result of this bias, you may see entry/exit signals on certain candles that weren’t there before, or vice versa.

Example of indicator that has this bias is this Ehler’s Super Smoother Filter in pandas-ta. This function even has worse issue because the value for the first 3 rows are taken from future datas (lookahead bias). So this function has both lookahead and recursive loop biases. This is the problematic rows (might be fixed in the future)

for i in range(0, m):
    ssf.iloc[i] = c1 * close.iloc[i] + c2 * ssf.iloc[i - 1] + c3 * ssf.iloc[i - 2] + c4 * ssf.iloc[i - 3]

for i in range(0, m):
    ssf.iloc[i] = c1 * close.iloc[i] + b1 * ssf.iloc[i - 1] + a1 * ssf.iloc[i - 2]

There isn’t a viable solution to this problem. The only options are to either not utilize this indicator or, if you must compare two results, ensure that you are using the same start time for both. At least the indicator’s value will be the same in both backtests.

Test results

So I have done a simple recursive test to see how much the difference gonna be given different start time. How I do the test is quite simple

• Do a full backtest with supplied timerange and use in-strat defined startup candle count
• After the full backtest done, do several smaller backtest using a very small timerange (same end time, but the start time is just 1 candle before end time), but with varying startup candles (200, 500, 1000, 2000). These backtests gonna simulate what happen in dry/live run

So this is the result.

Start checking for recursive bias
Comparing last row of 2021-09-01T00:00:00-2023-08-04T01:07:54 backtest
vs 2023-07-30T21:07:54-2023-08-04T01:07:54 which mimic 200 startup candle on dry/live
=> found difference in indicator ema_25, with difference of -0.00000014%
=> found difference in indicator ema_50, with difference of 0.00064649%
=> found difference in indicator ema_100, with difference of 0.02790815%
=> found difference in indicator ema_200, with difference of nan%
=> found difference in indicator tema_25, with difference of 0.00002832%
=> found difference in indicator dema_25, with difference of -0.00000109%
=> found difference in indicator tema_50, with difference of -0.02984285%
=> found difference in indicator dema_50, with difference of -0.00438440%
=> found difference in indicator tema_100, with difference of nan%
=> found difference in indicator dema_100, with difference of nan%
=> found difference in indicator tema_200, with difference of nan%
=> found difference in indicator dema_200, with difference of nan%
=> found difference in indicator ewo_50_200, with difference of nan%
=> found difference in indicator hma_50, with difference of 0.00040163%
=> found difference in indicator sma_9, with difference of -0.00000000%
=> found difference in indicator sma_50, with difference of 0.00000000%
=> found difference in indicator sma_100, with difference of 0.00000000%
=> found difference in indicator sma_200, with difference of nan%
=> found difference in indicator rsi_14, with difference of 0.00022551%
=> found difference in indicator rsi_20, with difference of 0.00837835%
=> found difference in indicator rsi_45, with difference of 0.36517816%
=> found difference in indicator mfi_14, with difference of 0.00000000%
=> found difference in indicator mfi_45, with difference of -0.00000000%
=> found difference in indicator bb20_2_low, with difference of -0.00000000%
=> found difference in indicator bb20_2_mid, with difference of -0.00000000%
=> found difference in indicator bb20_2_upp, with difference of 0.00000000%
=> found difference in indicator bb40_2_low, with difference of -0.00000000%
=> found difference in indicator bb40_2_delta, with difference of 0.00000010%
=> found difference in indicator bb40_2_tail, with difference of 0.00000010%
=> found difference in indicator r_480, with difference of nan%
=> found difference in indicator tsi, with difference of -0.00293043%
=> found difference in indicator tsi_signal, with difference of -0.00337236%

Comparing last row of 2021-09-01T00:00:00-2023-08-04T01:07:54 backtest
vs 2023-07-26T17:07:54-2023-08-04T01:07:54 which mimic 400 startup candle on dry/live
=> found difference in indicator ema_25, with difference of 0.00000000%
=> found difference in indicator ema_50, with difference of 0.00000028%
=> found difference in indicator ema_100, with difference of 0.00041264%
=> found difference in indicator ema_200, with difference of -0.00580720%
=> found difference in indicator tema_25, with difference of 0.00000000%
=> found difference in indicator dema_25, with difference of -0.00000000%
=> found difference in indicator tema_50, with difference of 0.00001458%
=> found difference in indicator dema_50, with difference of -0.00000338%
=> found difference in indicator tema_100, with difference of 0.00630592%
=> found difference in indicator dema_100, with difference of -0.00146290%
=> found difference in indicator tema_200, with difference of nan%
=> found difference in indicator dema_200, with difference of nan%
=> found difference in indicator ewo_50_200, with difference of -2.53638995%
=> found difference in indicator hma_50, with difference of 0.00000005%
=> found difference in indicator sma_9, with difference of -0.00000000%
=> found difference in indicator sma_50, with difference of 0.00000000%
=> found difference in indicator sma_100, with difference of 0.00000000%
=> found difference in indicator sma_200, with difference of 0.00000000%
=> found difference in indicator close_9_mean, with difference of 0.00000000%
=> found difference in indicator rsi_14, with difference of 0.00000000%
=> found difference in indicator rsi_20, with difference of 0.00000017%
=> found difference in indicator rsi_45, with difference of 0.00405776%
=> found difference in indicator mfi_14, with difference of 0.00000000%
=> found difference in indicator mfi_45, with difference of -0.00000000%
=> found difference in indicator bb20_2_low, with difference of -0.00000000%
=> found difference in indicator bb20_2_upp, with difference of 0.00000000%
=> found difference in indicator bb40_2_low, with difference of -0.00000000%
=> found difference in indicator bb40_2_delta, with difference of 0.00000010%
=> found difference in indicator bb40_2_tail, with difference of 0.00000010%
=> found difference in indicator r_480, with difference of nan%
=> found difference in indicator tsi, with difference of -0.00000000%
=> found difference in indicator tsi_signal, with difference of -0.00000000%

Comparing last row of 2021-09-01T00:00:00-2023-08-04T01:07:54 backtest
vs 2023-07-24T15:07:54-2023-08-04T01:07:54 which mimic 500 startup candle on dry/live
=> found difference in indicator ema_50, with difference of -0.00000001%
=> found difference in indicator ema_100, with difference of -0.00018238%
=> found difference in indicator ema_200, with difference of -0.02155496%
=> found difference in indicator tema_25, with difference of -0.00000000%
=> found difference in indicator tema_50, with difference of -0.00000155%
=> found difference in indicator dema_50, with difference of 0.00000019%
=> found difference in indicator tema_100, with difference of -0.00380078%
=> found difference in indicator dema_100, with difference of 0.00132336%
=> found difference in indicator tema_200, with difference of nan%
=> found difference in indicator dema_200, with difference of 0.05605471%
=> found difference in indicator ewo_50_200, with difference of -9.41403005%
=> found difference in indicator hma_50, with difference of 0.00000001%
=> found difference in indicator sma_9, with difference of -0.00000000%
=> found difference in indicator sma_50, with difference of 0.00000000%
=> found difference in indicator sma_200, with difference of 0.00000000%
=> found difference in indicator close_9_mean, with difference of 0.00000000%
=> found difference in indicator rsi_14, with difference of 0.00000000%
=> found difference in indicator rsi_20, with difference of 0.00000000%
=> found difference in indicator rsi_45, with difference of 0.00090360%
=> found difference in indicator mfi_14, with difference of 0.00000000%
=> found difference in indicator mfi_45, with difference of -0.00000000%
=> found difference in indicator bb20_2_low, with difference of -0.00000000%
=> found difference in indicator bb20_2_mid, with difference of -0.00000000%
=> found difference in indicator bb20_2_upp, with difference of 0.00000000%
=> found difference in indicator bb40_2_low, with difference of -0.00000000%
=> found difference in indicator bb40_2_mid, with difference of -0.00000000%
=> found difference in indicator bb40_2_delta, with difference of 0.00000010%
=> found difference in indicator bb40_2_tail, with difference of 0.00000010%
=> found difference in indicator r_480, with difference of nan%
=> found difference in indicator tsi, with difference of -0.00000000%
=> found difference in indicator tsi_signal, with difference of -0.00000000%

Comparing last row of 2021-09-01T00:00:00-2023-08-04T01:07:54 backtest
vs 2023-07-14T05:07:54-2023-08-04T01:07:54 which mimic 1000 startup candle on dry/live
=> found difference in indicator ema_100, with difference of 0.00000001%
=> found difference in indicator ema_200, with difference of 0.00008003%
=> found difference in indicator tema_50, with difference of -0.00000000%
=> found difference in indicator dema_50, with difference of -0.00000000%
=> found difference in indicator tema_100, with difference of 0.00000070%
=> found difference in indicator dema_100, with difference of -0.00000010%
=> found difference in indicator tema_200, with difference of 0.00166532%
=> found difference in indicator dema_200, with difference of -0.00045546%
=> found difference in indicator ewo_50_200, with difference of 0.03495172%
=> found difference in indicator sma_9, with difference of -0.00000000%
=> found difference in indicator sma_50, with difference of 0.00000000%
=> found difference in indicator sma_100, with difference of -0.00000000%
=> found difference in indicator sma_200, with difference of 0.00000000%
=> found difference in indicator close_9_mean, with difference of 0.00000000%
=> found difference in indicator rsi_45, with difference of -0.00000003%
=> found difference in indicator mfi_14, with difference of 0.00000000%
=> found difference in indicator mfi_45, with difference of -0.00000000%
=> found difference in indicator bb20_2_low, with difference of -0.00000000%
=> found difference in indicator bb20_2_upp, with difference of 0.00000000%
=> found difference in indicator bb40_2_low, with difference of -0.00000000%
=> found difference in indicator bb40_2_delta, with difference of 0.00000010%
=> found difference in indicator bb40_2_tail, with difference of 0.00000011%

Comparing last row of 2021-09-01T00:00:00-2023-08-04T01:07:54 backtest
vs 2023-06-23T09:07:54-2023-08-04T01:07:54 which mimic 2000 startup candle on dry/live
=> found difference in indicator ema_200, with difference of 0.00000002%
=> found difference in indicator tema_100, with difference of 0.00000000%
=> found difference in indicator tema_200, with difference of 0.00000277%
=> found difference in indicator dema_200, with difference of -0.00000034%
=> found difference in indicator ewo_50_200, with difference of 0.00000788%
=> found difference in indicator sma_9, with difference of -0.00000000%
=> found difference in indicator sma_50, with difference of 0.00000000%
=> found difference in indicator sma_100, with difference of -0.00000000%
=> found difference in indicator sma_200, with difference of 0.00000000%
=> found difference in indicator close_9_mean, with difference of 0.00000000%
=> found difference in indicator mfi_14, with difference of 0.00000000%
=> found difference in indicator mfi_45, with difference of -0.00000000%
=> found difference in indicator bb20_2_low, with difference of -0.00000000%
=> found difference in indicator bb20_2_upp, with difference of 0.00000000%
=> found difference in indicator bb40_2_low, with difference of -0.00000000%
=> found difference in indicator bb40_2_delta, with difference of 0.00000010%
=> found difference in indicator bb40_2_tail, with difference of 0.00000010%
Checking recursive bias of indicators of test_recursive.py took 14 seconds.

Some things to be noted

• If you see nan%, that means that indicator return null value, which means supplied data isn’t enough for its calculation. So for example, for startup candle of 200, any 200 moving average can’t be calculated.
• If you are using EMA and want to use long length, it is advisable to use startup candle that is three to four times the EMA length.
• If you use any indicators that use EMA as part of its calculation, you might need longer startup candle. For TEMA for example, you might need to have startup candles equal to eight times the TEMA length. For EWO, you would need ten times the slow EMA length.

You can do your own test using recursive-analysis command. This is the strategy I used to get the result above. The command to launch the recursive test is freqtrade recursive-analysis --config <your config.json> --strategy <your strategy> --timerange=<any timerange you usually use for backtest> -p <one pair>

Things to note

• It can only check one strategy at a time
• Since it don’t care about any trades, to make the process faster, it’s recommended to turn off the entry and exit trend just for the recursive test
• And to make it even faster, use just one pair for the test. Any extra pairs apart from the first pair will be ignored.