Exchange import not supported?

[mark.sch]

Today I added HuobiPro and OKEX support to Gekko, watching the market goes fine but during history --import implementation it seems, the API is not exposing trade history. Or at least not more than the last 2000 trades. Am I missing something?

https://gekko.wizb.it/docs/introduction/...anges.html

On the supported exchange list there a several exchanges listed without an --import possibility. Did anyone implemented an universal importer taking history data from coinmarketcap for examble? Or how do you bridge the history gap for backtesting an exchange without import?

[askmike]

Gekko does not work with any normalized functions to import data because all exchanges have very different implementations. So if you add an exchange normally users are only able to:

- watch the realtime function (gekko will poll getTrades and build candles)
- do live trading (by creating and managing orders)

importing goes via an "importer" implementation found here: https://github.com/askmike/gekko/tree/de.../exchanges

Did anyone implemented an universal importer taking history data from coinmarketcap for examble?

Very open to this, we just have to make sure that we can get all data Gekko already stores as part of candles, they are:

- open / high / low / close (standard candle)
- vwp
- # trades

Today I added HuobiPro and OKEX support to Gekko, watching the market goes fine but during history --import implementation it seems, the API is not exposing trade history. Or at least not more than the last 2000 trades. Am I missing something?

I am not 100% sure what you mean. The API of the exchange? In that case unfortunately Gekko won't import on that exchange.

[mark.sch]

Gekko does not work with any normalized functions to import data because all exchanges have very different implementations. So if you add an exchange normally users are only able to:

- watch the realtime function (gekko will poll getTrades and build candles)
- do live trading (by creating and managing orders)

Yes, I have seen this in code. Many importers use the fetcher.getTrades method of the wrapper and set a from param.

Did anyone implemented an universal importer taking history data from coinmarketcap for examble?

Very open to this, we just have to make sure that we can get all data Gekko already stores as part of candles, they are:

- open / high / low / close (standard candle)
- vwp
- # trades

Coinmarketcap is very quick for fetching data. However, it has some limitations. The candle size for a long term history period is 15 minutes. Benefit is, you get it all at once. If the daterange is <=24h, Coimarketcap returns 5 Min. candles, but you can of course loop the day fetching.


It contains: 

- Unix style epoch date
- price in usd
- volume in usd

I did not follow the complete Gekko backtest code yet, how will Gekko behave if you set open / high / low / close to the same usd value?

Today I added HuobiPro and OKEX support to Gekko, watching the market goes fine but during history --import implementation it seems, the API is not exposing trade history. Or at least not more than the last 2000 trades. Am I missing something?

I am not 100% sure what you mean. The API of the exchange? In that case unfortunately Gekko won't import on that exchange.

Ah, sorry, yes I mean the exchange api, huobipro for example. The REST api does not offer much history data. Ccxt works fine with unified data access, but it can of course not expose, what the underlying REST api is not offering. Or I missed s.th., at least I did not see history data in both huobipro and okex REST api. This lead me to the idea of coinmarketcap - since there a several supported exchanges in Gekko without importer functionality.

One idea is a new --fastimport switch, to prefill the db in seconds with coinmaketcap data, to get a quick backtest direction. This would bridge the gap for all exchanges where no "--import" is available. For those exchages supporting the Gekko --import switch, will it update and overwrite existing candle data later on with more accurate data or will it skip (when running --import after --fastimport)?

[askmike]

Ccxt works fine

Note that I don't want CCXT to be a dependency of Gekko (broker).

Coinmarketcap is very quick for fetching data. However, it has some limitations. The candle size for a long term history period is 15 minutes. Benefit is, you get it all at once. If the daterange is <=24h, Coimarketcap returns 5 Min. candles, but you can of course loop the day fetching.

Dealing with 5 or 15 minute candles is going to be very bothersome for Gekko: Gekko stores 1 minute candles. If users use any candle size that's not modulus 5 or 15 they are not running strats over actual market data but a weird averaged candle (that includes nasty things like lookahead bias).

[mark.sch]

Note that I don't want CCXT to be a dependency of Gekko (broker).

Yes, any of the three new exchange implementations (HitBTC, HuobiPro, OKEX) do not need any core changes. It is just adding the js file to the own gekko exchange installation folder - like any plugin/exchange. Under the hood they use CCXT but they do not necessarily be maintained inside your main development branch, if you do not want to.

Coinmarketcap is very quick for fetching data. However, it has some limitations. The candle size for a long term history period is 15 minutes. Benefit is, you get it all at once. If the daterange is <=24h, Coimarketcap returns 5 Min. candles, but you can of course loop the day fetching.

Dealing with 5 or 15 minute candles is going to be very bothersome for Gekko: Gekko stores 1 minute candles. If users use any candle size that's not modulus 5 or 15 they are not running strats over actual market data but a weird averaged candle (that includes nasty things like lookahead bias).

I proceeded implementing and testing this. For HuobiPro and OKEX I made coinmarketcap the default importer - because there is no data from the exchange. This way it works like any other --import, without new command line switches or core changes. It catches single days from cmc as necessary for the given import range and returns 5 min. candles. After looping and expanding them to one minute candles, they are emitted for processing and db writing. After this, the db stores 1 minute candles like any other import does. For most of my backtest scenarios this works fine. Even the limitations, it helps at least to backtest certain altcoins only supported by these exchanges, e.g. Cortex CTXT/USDT on HuobiPro.

Instead of using the fetch ticker, I implemented requests with async/await - data gets fetched from coinmarketcap pretty fast, most times 2 complete days of history data within 1 second.





New exchanges can be found here: https://github.com/mark-sch/gekko/tree/d...e/wrappers  (and in the exchanges import folder)