Things turn out to get a lot more complex with the database issue. The --import gap was possible to fix with changed utc handling. But there are situations where no data is written to DB. On another machine this was not happening. First showstopper was a version problem of node postgres 6.x package. After updating to 7.x package it is working - but only from time to time. Very hard to reproduce. I added several debug logs and it seems to be a race condition during db insert (it is not a locked transaction right now) and db connection handling. At least I was able to get this error message:
2018-06-28 15:56:05 (DEBUG): Processing 350 new trades. From 2018-06-28 12:53:07 UTC to 2018-06-28 13:56:08 UTC. (an hour)
2018-06-28 15:56:05 (DEBUG): Writing candles to DB!
2018-06-28 15:56:05 (DEBUG): Error: This socket has been ended by the other party
at Socket.writeAfterFIN [as write] (net.js:364:12)
at Connection.query (/git/gekko/node_modules/pg/lib/connection.js:205:15)
at Query.submit (/git/gekko/node_modules/pg/lib/query.js:164:16)
at Client._pulseQueryQueue (/git/gekko/node_modules/pg/lib/client.js:356:24)
at Client.query (/git/gekko/node_modules/pg/lib/client.js:393:8)
at _.each.candle (/git/gekko/plugins/postgresql/writer.js:60:13)
at Function.forEach (/git/gekko/node_modules/lodash/dist/lodash.js:3298:15)
at Store.writeCandles (/git/gekko/plugins/postgresql/writer.js:47:5)
at Store.bound [as writeCandles] (/git/gekko/node_modules/lodash/dist/lodash.js:729:21)
at Store.processCandle (/git/gekko/plugins/postgresql/writer.js:75:10)
Another indication that it probably is a transactional race condition problem: When changing inside the getTrades method to callback(undefined, parsedTrades.reverse()); and forcing to write the lastest (definately not existing candles) first, the problem with a not updating database does not occur.
I think it is necessary to prevent client.end() and release the db connection to a connection pool instead. I will also test the new postgres feature "ON CONFLICT DO NOTHING" for transactional upsert handling instead of checking the insert against a query of existing candles with a certain start time.
2018-06-28 15:56:05 (DEBUG): Processing 350 new trades. From 2018-06-28 12:53:07 UTC to 2018-06-28 13:56:08 UTC. (an hour)
2018-06-28 15:56:05 (DEBUG): Writing candles to DB!
2018-06-28 15:56:05 (DEBUG): Error: This socket has been ended by the other party
at Socket.writeAfterFIN [as write] (net.js:364:12)
at Connection.query (/git/gekko/node_modules/pg/lib/connection.js:205:15)
at Query.submit (/git/gekko/node_modules/pg/lib/query.js:164:16)
at Client._pulseQueryQueue (/git/gekko/node_modules/pg/lib/client.js:356:24)
at Client.query (/git/gekko/node_modules/pg/lib/client.js:393:8)
at _.each.candle (/git/gekko/plugins/postgresql/writer.js:60:13)
at Function.forEach (/git/gekko/node_modules/lodash/dist/lodash.js:3298:15)
at Store.writeCandles (/git/gekko/plugins/postgresql/writer.js:47:5)
at Store.bound [as writeCandles] (/git/gekko/node_modules/lodash/dist/lodash.js:729:21)
at Store.processCandle (/git/gekko/plugins/postgresql/writer.js:75:10)
Another indication that it probably is a transactional race condition problem: When changing inside the getTrades method to callback(undefined, parsedTrades.reverse()); and forcing to write the lastest (definately not existing candles) first, the problem with a not updating database does not occur.
I think it is necessary to prevent client.end() and release the db connection to a connection pool instead. I will also test the new postgres feature "ON CONFLICT DO NOTHING" for transactional upsert handling instead of checking the insert against a query of existing candles with a certain start time.