Live Gekko Not Using Imported Data

Live Gekko Not Using Imported Data - Printable Version

+- Gekko Forum (https://forum.gekko.wizb.it)
+-- Forum: Gekko (https://forum.gekko.wizb.it/forum-13.html)
+--- Forum: General Discussion (https://forum.gekko.wizb.it/forum-14.html)
+--- Thread: Live Gekko Not Using Imported Data (/thread-57397.html)

Pages: 1 2

Live Gekko Not Using Imported Data - faldor - 07-19-2018

When running a Market watcher/Strat runner, it seems to get the data from fresh when I have already imported data for the same trading pair and exchange up till now. I would expect the Live Gekkos to piece together the imported data with the fresh data to make up the "Warmup Period" that my Live Gekko needs. When running a live gekko on 12 hour candles and a warmup period of 90 (using EMA(90)) in my strat, it would take over a month of fresh data if it doesn't utilize the imported data I have already got. Is this how Gekko is supposed to work? If so, could I easily get it to use imported data and piece together any data inbetween to get my live gekkos making calls instantly?

Thanks!

RE: Live Gekko Not Using Imported Data - mark.sch - 07-19-2018

This is a limitation of the Web UI version. The CLI version works as you expect it to work - with db history.

RE: Live Gekko Not Using Imported Data - faldor - 07-20-2018

Oooooh, I didn't even realize there is a non-Web UI version. Great information, thank you!

RE: Live Gekko Not Using Imported Data - gabbath - 09-29-2018

Apologies for resurrecting an old thread, but this ties into a question I also have about the web version:

It appears that web-launched live gekkos get their data from web-launched market watchers. Does this mean the gekko would trade as if it were launched when the market watcher started? I'm asking because sometimes gekkos crash and need to be relaunched. Would the behavior be the same as if it hadn't stopped at all? (assuming, of course, that the bot wouldn't have made any trades during the downtime)

Thanks in advance!

RE: Live Gekko Not Using Imported Data - askmike - 09-30-2018

Quote:It appears that web-launched live gekkos get their data from web-launched market watchers. Does this mean the gekko would trade as if it were launched when the market watcher started?

So when you start a live trading in the UI you basically run 2 low level gekkos:

- the market watcher responsible for pulling market data and storing it in the DB
- a leecher that pulls this out of the DB

If you now start another gekko (paper trader for example) on the same market Gekko reuses the first watcher.

Quote:I'm asking because sometimes gekkos crash and need to be relaunched. Would the behavior be the same as if it hadn't stopped at all? (assuming, of course, that the bot wouldn't have made any trades during the downtime)

The answer on that question depends on which low level Gekko crashed. If the market watcher crashed it will crash the other one (so you don't have a live trader trying to pull new market data from a database that won't update).

Whichever the case: the proper solution is that when a leecher starts it doesn't simply check whether a current market watcher is running and when it started (current behavior) - but actually looks into the database to see if we have more data (if you stop a market watcher after a week and start it again you should have good market data of a week).

RE: Live Gekko Not Using Imported Data - gabbath - 09-30-2018

Very insightful, thank you so much!

Yes, it would be awesome if web-launched Gekkos (both watchers and leechers) looked into the database in a newer release. I don't see that as critical path though, since there's always the option of launching from the CLI.

So, just to confirm that I understood your answer: assuming the watcher is healthy and only the leecher crashes (e.g. due to some trading error) and I need to recreate it, the new leecher would make the same predictions as the old one would have if it hadn't crashed. Correct?

RE: Live Gekko Not Using Imported Data - askmike - 09-30-2018

Quote:the new leecher would make the same predictions as the old one would have if it hadn't crashed. Correct?

Generally it should yes, but this does depend on the quality of the strategy. Specifically if it has a configured warmup period that matches the indicators, this is a hard question though. See this for example: https://quant.stackexchange.com/questions/3174/what-is-the-minimum-history-data-size-to-get-an-accurate-ema-macd-for-latest-hi

Also if your strategy has some internal state it might not be the same, all of Gekko sample strategies don't really (except for persistence).

RE: Live Gekko Not Using Imported Data - gabbath - 09-30-2018

(09-30-2018, 07:22 AM)askmike Wrote:
Quote:the new leecher would make the same predictions as the old one would have if it hadn't crashed. Correct?

Generally it should yes, but this does depend on the quality of the strategy. Specifically if it has a configured warmup period that matches the indicators, this is a hard question though. See this for example: https://quant.stackexchange.com/questions/3174/what-is-the-minimum-history-data-size-to-get-an-accurate-ema-macd-for-latest-hi

Also if your strategy has some internal state it might not be the same, all of Gekko sample strategies don't really (except for persistence).

Sounds like it should be fine then. The main reason I was asking is that for some reason strats tend to work better when started on the hour mark, so I started my watcher on the hour mark and hoped that the leecher would follow, but from your answer it seems it does.

Thanks again for your helpful and prompt replies!

RE: Live Gekko Not Using Imported Data - Piff_Sara - 10-05-2018

(07-19-2018, 11:30 PM)mark.sch Wrote: This is a limitation of the Web UI version. The CLI version works as you expect it to work - with db histor

Does the CLI also download automatically time windows that are not present in the db? For example, I started to run a live trading today (5-10-2018) with a strategy that needs a warm up of 10 days, using 1Day candles. In my database i only got data from 1-09-2018 till 1-10-2018 (hence 4 days are missing). Does the CLI automatically download the 4 missing days up to the 5th of October?

RE: Live Gekko Not Using Imported Data - gabbath - 10-05-2018

(10-05-2018, 04:51 AM)Piff_Sara Wrote:
(07-19-2018, 11:30 PM)mark.sch Wrote: This is a limitation of the Web UI version. The CLI version works as you expect it to work - with db histor
Does the CLI also download automatically time windows that are not present in the db? For example, I started to run a live trading today (5-10-2018) with a strategy that needs a warm up of 10 days, using 1Day candles. In my database i only got data from 1-09-2018 till 1-10-2018 (hence 4 days are missing). Does the CLI automatically download the 4 missing days up to the 5th of October?

If it's just a few minutes before, I know it fetches them on the fly and stitches the datasets together. Haven't tried it with more than that. What you should do instead is:

1) In your config, under the importer settings, modify the dates so that you have at least the previous 10 days up to now. Note that times are in UTC, so you need to make sure the hours on the end date are properly offset compared to your local date (for instance, since I'm UTC+3, I need to offset by subtracting 3 hours from my local time).
2) Make sure, in your config, that the candleWriter is set to "enabled: true". Otherwise, it will fail when trying to stitch the datasets. Not sure why this is, but it's useful overall, since the setting allows it to record history while it's trading.
3) Run the import first:

Code:
node gekko --config your-config.js --import

4) Run it live:

Code:
node gekko --config your-config.js

Some extra tips:
- Because you're writing the candles, you're essentially importing when trading, so if you ever need to restart the process, you'll already have the db up to date.
- If you're running on a Linux machine through an SSH session, note that node will stop when you exit the terminal session. To make your command run outside of any sessions, run it with nohup like this:

Code:
nohup node gekko --config your-config.js &

The output will be saved in a file called nohup.out, which you can then check to see how Gekko is doing.
- To stop a nohup process, first check for node processes like this:

Code:
ps aux | grep node

You will see your command there with a PID next to it, like "22104". To kill it, run:

Code:
kill 22104

Or, to kill all node processes:

Code:
killall -9 node

Hope that helps!