[gekko 0.6] API change for running a backtest

[tommiehansen]

Another thing I am thinking about: When you run a backtest this is basically the work that needs to be done:

1. load 1min candles from disk.
2. convert them into 1h candles (assuming candle size is 60).
3. calculate indicators
4. run the strategy
5. simulate trades
6. calculate performance

I have a feeling the first 2 items take up a lot of time. If you build a GA tool that tries to figure out the best MACD params (example) there is little reason to do step 1 and 2 on every iteration (if you have enough memory to store them). If you want to test strategies with different tresholds (keeping all indicator params intact) we can even include step 3 here. What about some new endpoints like this:

Code:

POST prepareBacktestData { watch, candleSize, etc. } -> returns some ID
POST backtest { the id you just got } -> returns backtest result

After the first call Gekko has stored all of these candles and keeps them in memory to very quickly feed them into a backtest. This might be able to speed up the backtests by a ton since a lot of heavy lifting will just be done once.

I have the feeling that loading all 1 min candles when running e.g. 3 minutes (or 5.. or 10) is slighly degrading to performance? Why not actually load the candles that are needed straight away; no need to convert if SQL since the query could simply be done in one single query? This combined query is ofc slower then a "SELECT *" but step 2 isn't needed then (so potentially faster) and this whole query could easily be cached temporarily somewhere:

Code:

SELECT * FROM candles_BTC_USD
WHERE ID % 60 = 0

Returns each 60th candle which in this case would be an analog for using a candle size of 60 minutes.

Also the idea that it takes a long time to do the query that requests the 1 min candles is, sorry to say, slightly false? On average a SELECT * FROM candles_xxx_xxx that contain a total of 2 years and 11 months of data only take 243-438ms (0.2-0.4s). Do note that e.g. SQLite actually caches queries so a properly setup environment for it should at least cut that time in half. Do also note that measurement was not taken using Gekko but another external tool so if Gekko has poor optimizations or uses a poor library for SQLite that may affect things.