Gekko 0.6 released

[mark.sch]

@mark.sch very strange, is this also happening with the sqlite adapter?

@henry that issue was fixed yesterday here, please pull the branch again!

Yes, both on postgres and sqlite unfortunately. What I figured out so far, it seems to be an GMT+2 and UTC issue. When you run an --import, then 2 hours of data are missing, e.g. we have 16:30h local time, then data is only written until 14:40h. The realtime run is unable to write the gap on kraken.

var from = moment.utc(daterange.from);

if(daterange.to) {
  var to = moment.utc(daterange.to);
} else {
  var to = moment().utc();
  log.debug(
    'No end date specified for importing, setting to',
    to.format('YYYY-MM-DD HH:mms')
  );
}

This code is from import.js. When I change to:

var from = moment(daterange.from);

if(daterange.to) {
  var to = moment(daterange.to);
} else {
  var to = moment();
  log.debug(
    'No end date specified for importing, setting to',
    to.format('YYYY-MM-DD HH:mms')
  );
}

and the config this way:

config.importer = {
  daterange: {
    // NOTE: these dates are in UTC
    from: "2018-06-27T00:00:00Z"
    //from: "2018-06-27 00:00:00"
    }
}

I get all import data until 16:30h local time, except a couple of minutes missing. However the next real run is not adding any data to the db anymore. With blank db it is adding data.

[mark.sch]

Things turn out to get a lot more complex with the database issue. The --import gap was possible to fix with changed utc handling. But there are situations where no data is written to DB. On another machine this was not happening. First showstopper was a version problem of node postgres 6.x package. After updating to 7.x package it is working - but only from time to time. Very hard to reproduce. I added several debug logs and it seems to be a race condition during db insert (it is not a locked transaction right now) and db connection handling. At least I was able to get this error message:

2018-06-28 15:56:05 (DEBUG): Processing 350 new trades. From 2018-06-28 12:53:07 UTC to 2018-06-28 13:56:08 UTC. (an hour)
2018-06-28 15:56:05 (DEBUG): Writing candles to DB!
2018-06-28 15:56:05 (DEBUG): Error: This socket has been ended by the other party
at Socket.writeAfterFIN [as write] (net.js:364:12)
at Connection.query (/git/gekko/node_modules/pg/lib/connection.js:205:15)
at Query.submit (/git/gekko/node_modules/pg/lib/query.js:164:16)
at Client._pulseQueryQueue (/git/gekko/node_modules/pg/lib/client.js:356:24)
at Client.query (/git/gekko/node_modules/pg/lib/client.js:393:8)
at _.each.candle (/git/gekko/plugins/postgresql/writer.js:60:13)
at Function.forEach (/git/gekko/node_modules/lodash/dist/lodash.js:3298:15)
at Store.writeCandles (/git/gekko/plugins/postgresql/writer.js:47:5)
at Store.bound [as writeCandles] (/git/gekko/node_modules/lodash/dist/lodash.js:729:21)
at Store.processCandle (/git/gekko/plugins/postgresql/writer.js:75:10)

Another indication that it probably is a transactional race condition problem: When changing inside the getTrades method to callback(undefined, parsedTrades.reverse()); and forcing to write the lastest (definately not existing candles) first, the problem with a not updating database does not occur.

I think it is necessary to prevent client.end() and release the db connection to a connection pool instead. I will also test the new postgres feature "ON CONFLICT DO NOTHING" for transactional upsert handling instead of checking the insert against a query of existing candles with a certain start time.

[mark.sch]

Finally, the database errors are gone, puh. After hours of debugging, it worked one time and another time no candles were written into db.

The above error lead me into the direction to introduce a connection pool usage. This will also prevent to have a single, unrecoverable connection inside handle.js and speeds up db usage. If for some reason this connection previously dropped, the process died; a connection pool can handle this.
Second change was to move from the prepared statement for candle upserts into a transactional upsert query.

I will do more testing, but after several --import and realtime runs, there was no candle missing in the db anymore. If you want to take a closer look or merge I can create a pull request.

[mark.sch]

Ok, back on track with v0.6 testing. I get missing exchangeChecker. @Mike, is --import implemented already and should run?

2018-06-30 16:36:22 (DEBUG): No end date specified for importing, setting to 2018-06-30 14:36:22
module.js:549
throw err;
^

Error: Cannot find module '/git/gekko06/core/../core/exchangeChecker'
at Function.Module._resolveFilename (module.js:547:15)
at Function.Module._load (module.js:474:25)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/git/gekko06/core/markets/importer.js:31:23)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)

[Kris191]

Will Gekko plus support high frequency trading ?

[askmike]

Will Gekko plus support high frequency trading ?

Unfortunately not! The scope will not change, see this doc for details.

0.6 is now out! I will write a full report asap If you encounter any issue definitely let me know!

[Kris191]

Installed 0.6v and getting this error

Code:

/root/gekko/exchange/wrappers/gdax.js:377
     log.debug('Scanning back in the history needed...');
     ^

ReferenceError: log is not defined
   at Trader.getTrades (/root/gekko/exchange/wrappers/gdax.js:377:7)
   at Stitcher.checkExchangeTrades (/root/gekko/core/tools/dataStitcher.js:199:11)
   at Stitcher.<anonymous> (/root/gekko/core/tools/dataStitcher.js:108:10)
   at Statement.<anonymous> (/root/gekko/plugins/sqlite/reader.js:72:12)

can you help?

[ankasem]

thank you
Ubuntu desktop version 18.04 Gekko v0.6.0 was installed.
history + strategies + config strategies copied from old version

but Local Data
Start a new import market

sellective buttons is not active (Exchange+Currency+Asset)

[donkykong017]

thank you
Ubuntu desktop version 18.04 Gekko v0.6.0 was installed.
history + strategies + config strategies copied from old version

but Local Data
Start a new import market

sellective buttons is not active (Exchange+Currency+Asset)

you have to install broker from Exchange folder. look at the documentation website.

one question though. intalled everything and tested a littel bit yesterday. i did not seem to get warmup period from history data for strat runners in UI. 
i have to admit i did no read anywhere about it in the changelog an releasnotes. so i guess its not implemented yet, is it? just want to confirm this.

thanks for the huge update mike!

cheers DK

[askmike]

@anksem & @Kris191

I thought I went through all importers, but apparently I did not. I just pushed a new minor release (0.6.1) that includes fixes for importing from Kraken, Gdax & Bitfinex, see here: https://github.com/askmike/gekko/releases