02-21-2018, 10:05 AM
(02-19-2018, 05:14 AM)askmike Wrote: > 4. Don't store the data as pure db's but rather store it as e.g. csv files so that these can get gzipped.
You probably want to store it as binary json as opposed to utf8 (csv, json), that gzips even better in my experience.
> Most exchanges got a TOS that prevent data from being shared or make it easily available.
Are you sure? I haven't heard this before, there are even some online services that are selling raw data for $$.
------
I am already storing a lot of different markets and as part of my Gekko service will make this available. I hope to make it available to everyone for free (I can maybe use cloudflare), but I'm not sure how many people are looking for so much data.
Thank you for taking the time to answer, I know you're occupied etc so great.
Yes, bson would be even better. The storing method is the simplest part though.
If one go to the exchanges they all got TOS's, but it sort of depend on what defines 'shares'. A format that is very specific for something and not RESOLD shouldn't be a problem. As said -- the data is already used by several other sites so doesn't seem to be a problem. This was merely a heads up.
--
Backtesting requires as much data as possible. This for obvious reasons: One want as many different sort of periods as possible. This since more data reaffirms something to a much greater extent then less data.
This is true for all backtesting. That some seem to believe that a 3m period is enough to confirm that a strategy works is quite insane tbh. 'Anyone' can easily create a strategy that will only work for such a short period.