04-17-2018, 09:23 AM
The importer is such a major suck in its current state to be frank.
It's so bad that i've considered many times to simply rewrite it.
This should be done
1. Split the sets into actual files, this since SQLite db's can be finicky and prone to corruption. A single exchange/set corrupting is a zero-problem. An entire db with 10+ sets corruption less so. This would also make it around 5 billion times % easier to share sets, manage them, convert to other DB's ... etc etc ... etc.
2. Use free filesharing sites such as transfer.sh to act as a proxy so that not every single user has to spam the exchanges for the same data over and over again.
-
Flow
This is a flow that could work:
a) download gekko_history.txt (basically an index file) that contains e.g. this CSV data:
b) Check if user request contains the date that is within the csv data.. if so download from "link", untar it etc.
-
Any sort of file service could be used that allows for simple transfers and scripting transfer. transfer.sh is just used as an example since it's one of the simplest that i found myself for this sort of stuff.
It's so bad that i've considered many times to simply rewrite it.
This should be done
1. Split the sets into actual files, this since SQLite db's can be finicky and prone to corruption. A single exchange/set corrupting is a zero-problem. An entire db with 10+ sets corruption less so. This would also make it around 5 billion times % easier to share sets, manage them, convert to other DB's ... etc etc ... etc.
2. Use free filesharing sites such as transfer.sh to act as a proxy so that not every single user has to spam the exchanges for the same data over and over again.
-
Flow
This is a flow that could work:
a) download gekko_history.txt (basically an index file) that contains e.g. this CSV data:
Code:
EXCHANGE,ASSET,CURRENCY,FROM,TO,FILE
poloniex,NEO,USDT,2016-01-01,2018-04-01,https://transfer.sh/XJjVl/poloniex_neo_usdt_2016-01-01--2018-04-01.tar.gzb) Check if user request contains the date that is within the csv data.. if so download from "link", untar it etc.
-
Any sort of file service could be used that allows for simple transfers and scripting transfer. transfer.sh is just used as an example since it's one of the simplest that i found myself for this sort of stuff.