[SHARE] GAB - Gekko Automated Backtests

[tommiehansen]

Great that you got it running!

1. There are details on this in the config.sample.php file. But yes -- all your Gekko servers must be in sync meaning they must have the same strategies and history file(s). How you sync your servers is up to you. But rsync is one way to do it -- yes.

2. No requirement, but basically one can count 1 thread / 1 physical core needed. So if you have a Quad-core processor you would max out around 4 threads. What mostly happen if you use more threads then your core count is that your system will begin to que threads which is a bad thing since it will increase the time taken to finish each run and you may hit timeouts after a while. Often one can use core count + 1 though so a Quad-core computer can use 5 threads. Do note that 5 threads on a quad core will nearly use 100% of your CPU so for very long runs (10+ hours) i would not recommend that you push your hardware in such a way.

These limits doesn't got anything to do with GAB but is more about how Gekko API works; unfortunately Gekko API spawn's 1x nodejs instance per thread and doesn't make use of any sort of caching mechanism for repeated requests. To solve this Gekko itself would have to be modified.

--

3. Note that the multi-server mode is sort of an advanced use of GAB. It of course can do wonders for the max number of threads one can run but it requires quite a bit more from the user (like syncing everything etc). But -- it also is very recommended to run such a thing with the MySQL option. This due to the fact that any high number of threads can cause a lot of WRITE operations which SQLite just isn't great for and that can cause the SQLite databases to become corrupt. With the MySQL option this simply isn't a problem since MySQL easily handles a lot more concurrent read/write then GAB will ever use.

[simpsus]

I want to filter the best results a little. For example, 2/3 of the top results are with candle size 70 minutes and 1/3 significantly lower, so I want to create an average that only takes the 70 minutes into account. Or I want to blacklist a few strategies with very little (or far too many) trades.
I downloaded the results db file and looked at it in a database browser. all cool, but the strategies in the results table are a blob.
Is there any way to get the strat information off the db?
I had a look at the view.php and class.gab.php but could not pinpoint where this is done.
Thanks for any hint on this!

[Ualas]

Tommiehansen,

I've just upgraded, run the Compatibility Fix, and the SanityCheck is okay but I'm getting this error at view.php

Code:

Notice: Undefined variable: oldMySQL in /var/www/html/gab/view.php on line 145

Edit:
I've switched to MySQL but now I have the warning:

Code:

Warning! You are using an old version of MySQL, please upgrade to 5.6+ that came out in 2013...

But my MySQL version is 5.7.21

[tommiehansen]

Tommiehansen,

I've just upgraded, run the Compatibility Fix, and the SanityCheck is okay but I'm getting this error at view.php

Code:

Notice: Undefined variable: oldMySQL in /var/www/html/gab/view.php on line 145

Edit:
I've switched to MySQL but now I have the warning:

Code:

Warning! You are using an old version of MySQL, please upgrade to 5.6+ that came out in 2013...

But my MySQL version is  5.7.21

That error message is obviously a bit 50/50. You're hitting the $oldMySQL because the first query to the MySQL db did not work which is usually the case for older db's and then it seems to error out. This most likely because your MySQL db cannot be connected to or there is some error with it.

Do you even got any runs to view?

[tommiehansen]

I want to filter the best results a little. For example, 2/3 of the top results are with candle size 70 minutes and 1/3 significantly lower, so I want to create an average that only takes the 70 minutes into account. Or I want to blacklist a few strategies with very little (or far too many) trades.
I downloaded the results db file and looked at it in a database browser. all cool, but the strategies in the results table are a blob.
Is there any way to get the strat information off the db?
I had a look at the view.php and class.gab.php but could not pinpoint where this is done.
Thanks for any hint on this!

Yes, it's a db and the blobs are gzipped (try running 10 000 runs for 2+ years of data and check the db size) and thus not that easily readable through ordinary db management software.

You could however use the db as is and then write a script that does everything you're talking about. It's a simple db, it doesn't care how you use it or from where. :-)

[Ualas]

Sorry, forgot to mention when I've switched from SQLlite to MySQL the oldMySQL error dissapeared. Currently I do have the MySQL version warning message but the system works properly.

I maybe found a bug: all my Runs Last Change are 1982-02-08 01:00.

Can you help me with a question? Do we have different warmups periods when we backtest and when we are live (the length is the same but the market is different) ? Since the strategy is "calibrated" at the warmup period, not only the market trades (present and future) should be our concerns, but the warmup period too (past).

[tommiehansen]

Sorry, forgot to mention when I've switched from SQLlite to MySQL the oldMySQL error dissapeared. Currently I do have the MySQL version warning message but the system works properly.

I maybe found a bug: all my Runs Last Change are 1982-02-08 01:00.

Can you help me with a question? Do we have different warmups periods when we backtest and when we are live (the length is the same but the market is different) ? Since the strategy is "calibrated" at the warmup period, not only the market trades (present and future) should be our concerns, but the warmup period too (past).

1982-02-08 is not a bug, it's my birthday. It's a fake value since with the older MySQL cannot lookup the InnoDB-tables since they do not exist under older versions. Hence the query for that data fails in the 'View' of GAB and it instead tries to query the db using an alternate query.

But this query cannot return the so called 'last_update' for each database so hence there is a fake value set. 1982-02-08 was set strictly for fun and since it's obvious that it's a date that is wrong. The error message could be clearer though and notify of that thing.

However the fake value is only visible if you also see the error message. It makes no sense that you would not see the error message and at the same time see the 1982-02-08 dates since these two are under the same condition. So in order to see 1982-xx without seeing the error message you would have to hack the code or have some version of GAB that is very outdated by now where such a thing was possible.

Warmup periods etc are about Gekko and not about GAB, please use any of the other threads (or create a new thread) for such questinons, else this thread would be filled with Gekko-specific questions which would become slighly off-topic.

[simpsus]

Yes, it's a db and the blobs are gzipped (try running 10 000 runs for 2+ years of data and check the db size) and thus not that easily readable through ordinary db management software.

You could however use the db as is and then write a script that does everything you're talking about. It's a simple db, it doesn't care how you use it or from where. :-)

Thank you very much for your reply.
I have to admit I don't get how I could use the db "as-is". Could you point me to the code in GAB that uses the db so I can see how the blobs are taken out of the db? Or any other general pointer as to how I would go about doing that? Thank you very much.

Edit: I tried to do this with 

Code:

import sqlite3
import zlib

db = sqlite3.connect('gab_kraken.db')
rows = db.execute('select * from results')

but python tries to convert to text and fails horribly:

Code:

sqlite3.OperationalError: Could not decode to UTF-8 column 'strat' with text '�'

google did not turn up anything useful...

[22178]

Hi, 

I'm not very knowledgeable about this. 

system: bash on Ubuntu on windows 10, database: mysql, php 7.1 , php admin, php localhost:3030, mysql:127.0.0.1:3030  Gekko localhost:3000 

I have completed the Pre-Conditions (required).  Rename the gab-master folder to gab. Copy the file to var\www directory. So var\www\gab. PHP connection point localhost:8080. I am using mysql socket 127.0.0.1:3030. How do I change the usinconfig.php settings? 

http://localhost:3030/gab  I am getting a "Not Found"

Gekko localhost:3000 

Where do i make mistakes and what should i do?
I'm sorry for my bad english
Thanks.

[tommiehansen]

Yes, it's a db and the blobs are gzipped (try running 10 000 runs for 2+ years of data and check the db size) and thus not that easily readable through ordinary db management software.

You could however use the db as is and then write a script that does everything you're talking about. It's a simple db, it doesn't care how you use it or from where. :-)

Thank you very much for your reply.
I have to admit I don't get how I could use the db "as-is". Could you point me to the code in GAB that uses the db so I can see how the blobs are taken out of the db? Or any other general pointer as to how I would go about doing that? Thank you very much.

Edit: I tried to do this with 

Code:

import sqlite3
import zlib

db = sqlite3.connect('gab_kraken.db')
rows = db.execute('select * from results')

but python tries to convert to text and fails horribly:

Code:

sqlite3.OperationalError: Could not decode to UTF-8 column 'strat' with text '�'

google did not turn up anything useful...

You would need to gzip decode it as well since there is not "UTF-8" in this case (it's binary data).