02-19-2018, 05:14 AM
> 4. Don't store the data as pure db's but rather store it as e.g. csv files so that these can get gzipped.
You probably want to store it as binary json as opposed to utf8 (csv, json), that gzips even better in my experience.
> Most exchanges got a TOS that prevent data from being shared or make it easily available.
Are you sure? I haven't heard this before, there are even some online services that are selling raw data for $$.
------
I am already storing a lot of different markets and as part of my Gekko service will make this available. I hope to make it available to everyone for free (I can maybe use cloudflare), but I'm not sure how many people are looking for so much data.
You probably want to store it as binary json as opposed to utf8 (csv, json), that gzips even better in my experience.
> Most exchanges got a TOS that prevent data from being shared or make it easily available.
Are you sure? I haven't heard this before, there are even some online services that are selling raw data for $$.
------
I am already storing a lot of different markets and as part of my Gekko service will make this available. I hope to make it available to everyone for free (I can maybe use cloudflare), but I'm not sure how many people are looking for so much data.