Does gekko store data in memory or database?

[gsxr]

I had run --import to load initial data of over 5000 mintues using CLI.

I found the postgresql database loaded with some records successfully (with 6718 records).

Live paper trader was successfully run with slow moving average indicator loaded with 1000 x 5 min candles.

I did make sure config.candleWriter = { enabled: true }

I supposed new candle data will be inserted to database for all the subsequent ticker update. However, I found the number of records did not increased.

After a day of running, I restarted gekko live trader using the same config file. Datasets were not stitched with following log...

---- DEBUG LOG ----
The trading method requests 5000 minutes of historic data. Checking availablity..
Available local data:
from: 5000 minutes ago
to: 1176 minutes ago
Usable local data available, trying to match with exchange data..
Preventing Gekko from requesting 1191 minutes of history.
Fetching exchange data since 240 minutes ago
Available exchange data:
from: 240 minutes ago
to: 180 minutes ago
Unable to stitch datasets.
Not seeding locally available data to the trading method.
The exchange does not return enough data. 0 minutes are still missing.
---- END OF LOG ----

It seems like that 1176 minutes running of live paper trader did not save candle data into database but only in memory. Therefore that period of data was lost after gekko restarted.

I also don't understand why gekko fetch exchange data from 240 min to 180 min? I have seen this happened many times before.

Since it doesn't make sense to wait for a 5000 minutes warmup, my workaround is to run --import using CLI again to load the data after every gekko restart.

Does anyone know if this is a constraint of did I miss anything in the configuration for gekko to save live data into the database?

Thanks for advice.

[askmike]

It seems like that 1176 minutes running of live paper trader did not save candle data into database but only in memory. Therefore that period of data was lost after gekko restarted.

Very weird, with the candleWriter set to true this should definitely work, can you upload the full config (without API keys) and post a link here?

I also don't understand why gekko fetch exchange data from 240 min to 180 min? I have seen this happened many times before.

The 240 comes from your required history, however there is a limit (for performance as well as reasons related to exchange APIs) to not fetch too much data at once. Which is why it gets shifted to 180.

[mark.sch]

Hello gsxr, I had exactly the same problem you described here. On one machine I had no problems, on another one big problems and most times no new candles in db. I have rewritten some stuff, you may check if it also solves your problem: https://github.com/mark-sch/gekko

[askmike]

@mark.sch

Awesome stuff man! Would love to port this over to Gekko, the current postgresql is a mess. Do you mind if I pull this one?

[mark.sch]

@mark.sch

Awesome stuff man! Would love to port this over to Gekko, the current postgresql is a mess. Do you mind if I pull this one?

Hi Mike, of course, this is the way open source should work, yeah. Let me sort out one thing, it has a prob when the table for a new candle pair is not already existing, it is exiting with error. I will fix this and let you know by pull request.

[mark.sch]

Fix is commited, see PM and pull request.