Gekko 0.6 released

[mark.sch]

@mark.sch I've summarized everything in a doc file, see here: https://github.com/askmike/gekko/blob/pr...per_api.md

@Henry151 I fixed this ~40 minutes ago, see here: https://github.com/askmike/gekko/commit/...d45fcbbcbe are you running the latest commit on that branch?

@aslmike awesome, I will take a closer look to your detailed response. Right now I am close to finalize hitbtc exchange+import into 0.5. Candle writing is working in realtime mode to db and also in import mode. Trading is implemented and needs testing - using the ccxt lib. When db is prefilled with imports and gekko starts with stitching in next realtime run, it fetches new candles but never write it back to db. The same happens here with kraken too. Do you maybe have a reported issue in mind, which can get me into direction?

@henry151 I remember the same binance complaint yesterday also on v0.6. Out of my head it was a
 missing library. npm install binance and npm install poloniex.js fixed this error when testing on binance and poloniex. However my v0.6 then got stuck with lodash version dependency probs.

[askmike]

Awesome! Would LOVE hitbtc support in there, the difference between 0.5 and 0.6 is minimal. So after 0.5 we can easily port to 0.6.

ccxt

For WIPs this is fine but unfortunately for complicated reasons we cannot have ccxt in a main release of Gekko. I will go through the effort of refactoring out usage of ccxt once the rest is working.

Do you maybe have a reported issue in mind, which can get me into direction?

Yes reports have been coming up since last week. Under what branch are you experiencing this? Are you sure the candleWriter plugin is enabled?

However my v0.6 then got stuck with lodash version dependency probs.

Lodash usage is a mess after a few years of relying on a very old version, this is how it should work in theory:

- Gekko Broker (and thus the new wrappers) should use a very new version of lodash (4.17.5), this means some methods like _.bindAll are not possible anymore. Thew new wrappers can simply use functions with arrow notation to automatically bind (this ES6 feature did not exist when Gekko started).
- The rest of the codebase still relies on an old version (2.x) mostly because of the _.bindAll reliance which should be refactored out.

In order to properly install both lodash versions it SHOULD be as simple as:

Code:

cd gekko
npm install # installs main gekko deps + old lodash
cd exchange
npm install # installs gekko broker deps + new lodash, automatically used by gekko broker

Let me know if the above is not working.

[Henry151]

@mark.sch I've summarized everything in a doc file, see here: https://github.com/askmike/gekko/blob/pr...per_api.md

@Henry151 I fixed this ~40 minutes ago, see here: https://github.com/askmike/gekko/commit/...d45fcbbcbe are you running the latest commit on that branch?

Thanks, latest commit, no problem. However I think Binance is down right now, so I'm getting "Error:

Gekko tried to retrieve data since 2018-06-26 03:31:00, however
        binance did not return any trades."

But I think that's just a binance problem, not a gekko problem.

I compiled node from source to enable static probe points so that I can use bcc-tools assorted node-observing tools to watch what the program is doing "under-the-hood" to try to learn more about how it works. Very excited to see what I can learn! Unfortunately i don't know how much i'll understand, but the whole process has been fascinating.

[askmike]

Yep Binance is down right now, for the last ~6 hours they have been under maintenance.

I compiled node from source to enable static probe points so that I can use bcc-tools assorted node-observing tools to watch what the program is doing "under-the-hood" to try to learn more about how it works. Very excited to see what I can learn! Unfortunately i don't know how much i'll understand, but the whole process has been fascinating.

That's very low level, and not just under Gekko's hood but under node/v8/libuv as well. In the majority of Gekko's development I am not working on this level. With a good understanding of javascript and the nodejs event loop you can get very far in creating real software. It's a great abstraction similar to something like Java and the JVM (v8 was actually architected by one of the leads of the JVM).

[mark.sch]

However my v0.6 then got stuck with lodash version dependency probs.

In order to properly install both lodash versions it SHOULD be as simple as:

Code:

cd gekko
npm install # installs main gekko deps + old lodash
cd exchange
npm install # installs gekko broker deps + new lodash, automatically used by gekko broker

Let me know if the above is not working.

ahh, I see. 

find -name package.json -not -path "*/node_modules/*"
./web/vue/package.json
./exchange/package.json
./package.json


There are 3 different package.json files maintained. This did the trick ;-) Great. So v0.6 is running and waiting for Binance to get back. This is the only complete exchange right now? Poloniex is throwing an error, ok, and kraken needs to be converted to new Broker model.

Are you planning to refactor the code to use one package.json file? Maybe it is worth in this early 0.6 stage, also to get rid of these npm audit warning and critical package messages. And maybe this background knowledge could be a show stopper to get started with Gekko for beginners.

A lazy coder is giving up after
git clone ...
npm install
npm start

;-D

[askmike]

This is the only complete exchange right now? Poloniex is throwing an error, ok, and kraken needs to be converted to new Broker model.

See this list: https://github.com/askmike/gekko/blob/pr....md#status

So Poloniex should work, what error are you seeing? I am actually trading right now on poloniex using Gekko Broker.

EDIT: I just pushhed a fix to Gekko's integration with poloniex (outside of Gekko Broker). Please pull and try again

get rid of these npm audit warning and critical package messages.

Should be as simple as updating some outdated dependencies, this is not high on my list since I don't think Gekko can be easily exploited because of them.

Are you planning to refactor the code to use one package.json file? Maybe it is worth in this early 0.6 stage

From 0.6 onwards I want Gekko Broker (order execution) to be a separate library from Gekko. A lot of people building bots have been struggling to find proper order execution (advanced order types), especially people building systems that trade on multiple exchanges. As such I do really want to keep Gekko Broker's dependencies isolated in it's own package.json. If installing becomes even more complex I might include an installation script of sorts.

> ./web/vue/package.json

The UI is a vue frontend project, when you download a stable release you'll get a precompiled vue frontend (via webpack), in the majority of the cases you don't need this package file. Only if you want to develop the are you in need of this file. See here for more info: https://gekko.wizb.it/docs/internals/gekko_ui.html

maybe this background knowledge could be a show stopper to get started with Gekko for beginners.

I've updated the installation docs, once 0.6 is live these will be visible on the website, see here: https://github.com/askmike/gekko/commit/...0223a8c355

The codebase has already grown out of proportions. Splitting up major parts of the codebase (such as Gekko Broker) takes priority over easing installation. Note that if you are working on strategy development in simulated environments (when you are only backtesting) you only need market data and do not need Gekko Broker.

[mark.sch]

Thanks for your background info on this. The ability to run the broker (out of process) probably makes also sense for your GekkoPlus project.

With your latest commit the poloniex start error is fixed, it runs now fine in watcher mode. For the --import there is something missing in core directory:

Gekko v0.6.0-alpha.0
I'm gonna make you rich, Bud Fox.


2018-06-26 18:58:16 (INFO): Setting up Gekko in importer mode
2018-06-26 18:58:16 (INFO):
2018-06-26 18:58:16 (INFO): Setting up:
2018-06-26 18:58:16 (INFO): Candle writer
2018-06-26 18:58:16 (INFO): Store candles in a database
2018-06-26 18:58:16 (INFO):

2018-06-26 18:58:16 (DEBUG): Check database exists: binance
2018-06-26 18:58:16 (DEBUG): Database exists: binance
2018-06-26 18:58:16 (DEBUG): Postgres connected to binance
2018-06-26 18:58:16 (WARN): The plugin Trading Advisor does not support the mode importer. It has been disabled.
2018-06-26 18:58:16 (WARN): The plugin Paper Trader does not support the mode importer. It has been disabled.
2018-06-26 18:58:16 (WARN): The plugin Performance Analyzer does not support the mode importer. It has been disabled.
2018-06-26 18:58:16 (WARN): The plugin Event logger does not support the mode importer. It has been disabled.
2018-06-26 18:58:16 (WARN): The plugin Backtest result export does not support the mode importer. It has been disabled.
2018-06-26 18:58:16 (DEBUG): No end date specified for importing, setting to 2018-06-26 16:58:16
module.js:549
throw err;
^

Error: Cannot find module '/git/gekko06/core/../core/exchangeChecker'
at Function.Module._resolveFilename (module.js:547:15)
at Function.Module._load (module.js:474:25)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/git/gekko06/core/markets/importer.js:31:23)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)

[mark.sch]

Hi Mike, I am still testing the candlewriting (0.5). You wrote:

Under what branch are you experiencing this? Are you sure the candleWriter plugin is enabled?

Yes, I have checked this, candleWriter Plugin is enabled and running on latest development branch. For debugging I use kraken, but other exchanges have the same prob. When I start with --import all data is written into postgresql fine. If I start with an empty db in realtime/watcher mode, candles are also written. If I start with candles inside db already, but missing 30 minutes, it is perfoming watcher.getTrades() but these candles never get written into db.

select to_timestamp(start),* from candles_eur_eth order by start desc limit 100

With this query I am monitoring and no new candle is written to postgres

Output:

2018-06-26 20:34:54 (INFO): Setting up Gekko in realtime mode
2018-06-26 20:34:54 (INFO):
2018-06-26 20:34:54 (INFO): Setting up:
2018-06-26 20:34:54 (INFO): Candle writer
2018-06-26 20:34:54 (INFO): Store candles in a database
2018-06-26 20:34:54 (INFO):

2018-06-26 20:34:54 (DEBUG): Check database exists: kraken
2018-06-26 20:34:54 (DEBUG): Database exists: kraken
2018-06-26 20:34:54 (DEBUG): Postgres connected to kraken
2018-06-26 20:34:54 (INFO): Setting up:
2018-06-26 20:34:54 (INFO): Trading Advisor
2018-06-26 20:34:54 (INFO): Calculate trading advice
2018-06-26 20:34:54 (INFO): Using the strategy: Think5eth
2018-06-26 20:34:54 (INFO): The trading method requests 600 minutes of historic data. Checking availablity..
2018-06-26 20:34:54 (DEBUG): endTime: 1530038040 - idealStartTime: 1530002040
2018-06-26 20:34:54 (INFO):

2018-06-26 20:34:54 (DEBUG): DB data available from 1530002040 to 1530037860
2018-06-26 20:34:54 (DEBUG): Available local data:
2018-06-26 20:34:54 (DEBUG): from: 600 minutes ago
2018-06-26 20:34:54 (DEBUG): to: 3 minutes ago
2018-06-26 20:34:54 (INFO): Usable local data available, trying to match with exchange data..
2018-06-26 20:34:54 (DEBUG): Fetching exchange data since 18 minutes ago
2018-06-26 20:34:54 (DEBUG): checkExchangeTrades
2018-06-26 20:34:54 (DEBUG): checkExchangeTrades, calling getTrades now, since: 1530036960000
2018-06-26 20:34:55 (DEBUG): Available exchange data:
2018-06-26 20:34:55 (DEBUG): from: 18 minutes ago
2018-06-26 20:34:55 (DEBUG): to: 1 minutes ago
2018-06-26 20:34:55 (DEBUG): Stitching datasets
2018-06-26 20:34:55 (INFO): Full history locally available. Seeding the trading method with all required historical candles.
2018-06-26 20:34:55 (DEBUG): Seeding with:
2018-06-26 20:34:55 (DEBUG): from: 600 minutes ago
2018-06-26 20:34:55 (DEBUG): to: 19 minutes ago
2018-06-26 20:34:55 (INFO): Setting up:
2018-06-26 20:34:55 (INFO): Mailer
2018-06-26 20:34:55 (INFO): Sends you an email everytime Gekko has new advice.
2018-06-26 20:34:55 (DEBUG): Setup email adviser.
2018-06-26 20:34:55 (INFO):

2018-06-26 20:34:56 (INFO): Send advice via email.
2018-06-26 20:34:56 (INFO): Setting up:
2018-06-26 20:34:56 (INFO): Advice logger
2018-06-26 20:34:56 (INFO):
2018-06-26 20:34:56 (INFO): Setting up:
2018-06-26 20:34:56 (INFO): Paper Trader
2018-06-26 20:34:56 (INFO): Paper trader that simulates fake trades.
2018-06-26 20:34:56 (INFO):

2018-06-26 20:34:56 (INFO): Setting up:
2018-06-26 20:34:56 (INFO): Performance Analyzer
2018-06-26 20:34:56 (INFO): Analyzes performances of trades
2018-06-26 20:34:56 (INFO):

2018-06-26 20:34:56 (INFO): Starting to watch the market: Kraken ETH/EUR
2018-06-26 20:34:56 (DEBUG): scheduling ticks
2018-06-26 20:34:56 (DEBUG): Requested ETH/EUR trade data from Kraken ...
2018-06-26 20:34:56 (DEBUG): Processing 71 new trades. From 2018-06-26 18:16:34 UTC to 2018-06-26 18:33:46 UTC. (17 minutes)
2018-06-26 20:35:16 (DEBUG): Requested ETH/EUR trade data from Kraken ...
2018-06-26 20:35:16 (DEBUG): Processing 2 new trades. From 2018-06-26 18:35:23 UTC to 2018-06-26 18:35:35 UTC. (a few seconds)
2018-06-26 20:35:36 (DEBUG): Requested ETH/EUR trade data from Kraken ...

[Henry151]

Heya,

I'm trying to backtest from command line on v0.6 and getting some errors; however i've never done backtests from command line on older versions so I may just be doing something wrong; anyway it looks like this:

Code:

2018-06-27 17:14:51 (INFO):     Setting up Gekko in backtest mode
2018-06-27 17:14:51 (INFO):
2018-06-27 17:14:51 (WARN):     The plugin Candle writer does not support the mode backtest. It has been disabled.
2018-06-27 17:14:51 (INFO):     Setting up:
2018-06-27 17:14:51 (INFO):              Trading Advisor
2018-06-27 17:14:51 (INFO):              Calculate trading advice
2018-06-27 17:14:51 (INFO):              Using the strategy: buyatsellat
2018-06-27 17:14:51 (INFO):

2018-06-27 17:14:51 (INFO):     Setting up:
2018-06-27 17:14:51 (INFO):              Paper Trader
2018-06-27 17:14:51 (INFO):              Paper trader that simulates fake trades.
2018-06-27 17:14:51 (INFO):

2018-06-27 17:14:51 (INFO):     Setting up:
2018-06-27 17:14:51 (INFO):              Performance Analyzer
2018-06-27 17:14:51 (INFO):              Analyzes performancesof trades
2018-06-27 17:14:51 (INFO):


2018-06-27 17:14:51 (INFO):             WARNING: BACKTESTING FEATURE NEEDS PROPER TESTING
2018-06-27 17:14:51 (INFO):             WARNING: ACT ON THESE NUMBERS AT YOUR OWN RISK!

/home/henry/gekko/node_modules/sqlite3/lib/trace.js:27
                   throw err;
                   ^

ReferenceError: amount is not defined
   at PaperTrader.processAdvice (/home/henry/gekko/plugins/paperTrader/paperTrader.js:125:5)
   at Actor.bound (/home/henry/gekko/node_modules/lodash/dist/lodash.js:729:21)
   at emitOne (events.js:116:13)
   at Actor.emit (events.js:211:7)
   at Actor.bound [as emit] (/home/henry/gekko/node_modules/lodash/dist/lodash.js:729:21)
   at Actor.GekkoEventEmitter.broadcastDeferredEmit (/home/henry/gekko/core/emitter.js:30:8)
   at Actor.bound [as broadcastDeferredEmit] (/home/henry/gekko/node_modules/lodash/dist/lodash.js:729:21)
   at _.find.producer (/home/henry/gekko/core/gekkoStream.js:74:26)
   at Function.find (/home/henry/gekko/node_modules/lodash/dist/lodash.js:3217:15)
   at Gekko.flushDefferedEvents (/home/henry/gekko/core/gekkoStream.js:72:25)
--> in Database#all('\n    SELECT * from candles_USDT_BTC\nWHERE start <= 1503017399 AND start >= 1503014400\n    ORDER BY start ASC\n  ', [Function])
   at Reader.get (/home/henry/gekko/plugins/sqlite/reader.js:98:11)
   at Reader.bound [as get] (/home/henry/gekko/node_modules/lodash/dist/lodash.js:729:21)
   at Market.get (/home/henry/gekko/core/markets/backtest.js:61:15)
   at Market.bound [as get] (/home/henry/gekko/node_modules/lodash/dist/lodash.js:729:21)
   at Market.<anonymous> (/home/henry/gekko/core/markets/backtest.js:52:8)
   at Market._read (/home/henry/gekko/node_modules/lodash/dist/lodash.js:5605:23)
   at Market.bound [as _read] (/home/henry/gekko/node_modules/lodash/dist/lodash.js:729:21)
   at Market.Readable.read (_stream_readable.js:442:10)
   at Market.bound [as read] (/home/henry/gekko/node_modules/lodash/dist/lodash.js:729:21)

[askmike]

@mark.sch very strange, is this also happening with the sqlite adapter?

@henry that issue was fixed yesterday here, please pull the branch again!