[SHARE] GAB - Gekko Automated Backtests

[tommiehansen]

GAB
Gekko Automated Backtets

*** THIS IS BETA ***
If something does not work, it simply does not work since time hasn't existed yet to get it working.

Updated 14 april 2018 with MySQL as new option.

---

Why

I needed a way to run backtests for Gekko in a 'brute-force' manor automated and with multi-threading.
I also needed a way to compare all these runs and get extra data such as win percent etc.

-

Prerequisites (required)

1. Gekko installed and working
2. Webserver + PHP (preferrably 7.1+) with cURL and PDO SQLite or PDO MySQL/MariaDB
3. A user with write access

If you do not understand any of this it will be very hard to run this tool.
You can try googling for things such as install apache php 7.2.

-

How-to

0. Copy+Paste system/user.config.sample.php and rename the new file to user.config.php
1. Open user.config.php and make sure the $server variable is pointing at your Gekko install e.g. http://localhost:3000
2. Go to your-localhost/gab/ and see if it works.
3. Run something..
4. After you got some runs click 'View runs' in the menu and check results

--

Multi-server mode
If you're a nerd you can also use multi-server mode by specifying an array of servers instead of just one in your user.config.php like this:

$server = [
  'http://localhost:3000',
  'http://my-other-server:3000',
  'http://my-third-server:3000',
];

If your Gekko-files are not in sync on all these servers it will fail miserably though.
Setting an array will randomly use one of the servers to run a strategy.

In this case it's 3 so setting threads to e.g. 6 will usually mean 2 threads will run on each of these servers.
Do note that e.g. Chrome has a max concurrent connections @ 6 so in order to increase max-threads (and have it actually run in paralell) you would need to either force higher max concurrent connections or randomize/create subdomains for post.php (and remove any CORS restrictions).

-

Dynamic parameters
GAB uses dynamic parameters, these work for all strategy paramers. Example, RSI BULL/BEAR:

Code:

# SMA
SMA_long = 100:1000,100
SMA_short = 10:90,10

# BULL
BULL_RSI = 5:20,5
BULL_RSI_high = 70:90,5
BULL_RSI_low = 40:60,5

# BEAR
BEAR_RSI = 5:20,5
BEAR_RSI_high = 40:60,5
BEAR_RSI_low = 10:30,5

The format is <FROM>:<TO>,<STEPPING>.
This means that e.g. 10:20,5 will generate the range: 10,15,20
The params are inclusive meaning that odd stepping e.g. 5:15,10 will become 5,10,15 - the first and last of range is always kept.

-

Screens

Runner (select.php): https://i.imgur.com/kcXXMdW.png
Results (view.php): https://i.imgur.com/CPhvDEl.png

-

Donate

People keeps asking about this so i'll just leave it here for future reference:

BTC: 15cZUi7VvmCQJLbPXYYWChHF3JpsBaYDtH
ETH: 0xe03c5eb9DF93360e3Bcfcd13012B7DeebbED6923

-

Download

Go to: https://github.com/tommiehansen/gab/
or just clone: https://github.com/tommiehansen/gab.git

[donkykong017]

awesome! thanks for sharing this.

so this is bruteforce and will just run all possible combinations.
would it make any sense to implement something like GA that tends to find better results in fewer runs?

[tommiehansen]

awesome! thanks for sharing this.

so this is bruteforce and will just run all possible combinations.
would it make any sense to implement something like GA that tends to find better results in fewer runs?

Yes.
Yes, but the problem is -- when to optimize? Some datasets could have 2 years of data so optimizing after X time never makes sense and 1000 runs could take quite a while. It's simpler to just do 1000x runs, check the averages and optimize manually. There's a reason there's a Strategy average in the view.php screen and there's even more work going into it in the future.

Otherwise you could simply use the GA-stuff that's already available.

[donkykong017]

awesome! thanks for sharing this.

so this is bruteforce and will just run all possible combinations.
would it make any sense to implement something like GA that tends to find better results in fewer runs?

Yes.
Yes, but the problem is -- when to optimize? Some datasets could have 2 years of data so optimizing after X time never makes sense and 1000 runs could take quite a while. It's simpler to just do 1000x runs, check the averages and optimize manually. There's a reason there's a Strategy average in the view.php screen.

Otherwise you could simply use the GA-stuff that's already available.

thx for your reply.

i just thought over it and im getting what you mean.
i will have a look at your work later.
thx again

[donkykong017]

i just set up your GAB tool and started testing it.
it is a beautiful tool. very well thought through.
it seems that everything is working great.
except one thing and i cant seem to debug it

somtimes it does not get data from the test it says in the output

15:14:33 Runner.php ERROR: Get from curl_post() did not return data, something is wrong
15:14:19 Bad! Performed 264% worse then market
15:14:05 Bad! Performed 304% worse then market
15:13:36 Runner.php ERROR: Get from curl_post() did not return data, something is wrong
15:12:36 Runner.php ERROR: Get from curl_post() did not return data, something is wrong
15:12:33 Bad! Performed 280% worse then market
15:12:20 Success! Performed 566% better

i guess it has something to do with gekko having too much threads, but gekko does not give any error and lowering the threads also does not work. only restarting it made it a little bit better. after some time the runner errors get more an more.

[tommiehansen]

This can be due to several reasons:

1. The curl_post() function simply died since your PHP is out of memory or something similar
2. The request took way too long time (over 60 minutes) so your curl request timed out
3. You have some settings in your server that overrides all settings that GAB tries to set and thus curl times a bit too quickly (and thus returns error)
4. You run with too many threads which creates a long que of requests since your CPU simply cannot run all the requests at the same time.
Thus you hit the AJAX timeout --OR-- your servers max execution time and thus curl_post() returns nothing.

Try not using over 5 threads and check your CPU-usage to get an idea what the max threads you actually can run at the same time is before your CPU spikes and uses 90-100% of it.

Setting threads to 10+ requires quite a beefy CPU and if your CPU is at 100% (or close) when using just 3-4 threads it means that the rest
of the runs will be qued up (it simply won't run any faster; you've hit your limit).

In future versions i'm considering being able to have an array of servers instead of just one but that's such a super-nerdy thing that few would use so it isn't a top priority and it comes with a plethora of problems (like the different servers needing the exakt same datasets, strategies etc -- so all of them have to be in 100% sync and this should ofc also be checked etc....).
Also real error messages and not stuff like "something is wrong" will be added.

Thanks for helping out to test the tool.

[simpsus]

Now that I got it to work I can start asking content related questions.

Any Strategy is only as good as the past data you test it with. Or in other words, we are fitting strategies to past data. The success depends on the correlation between past and future. Nobody knows the future, so we try to make the correlation as robust as possible. 

Is that the reason for averaging?
Do you plan to automate the fitting further? i.e. Automatically iterating one parameter. At the moment I just make a 50 to a 40:60,10 and so on.

If so, then you could also iterate over strategies to detect fundamental changes in the market to switch to a better fitting strategy. Bigger scale trend changes, increases in volatility

[tommiehansen]

Now that I got it to work I can start asking content related questions.

Any Strategy is only as good as the past data you test it with. Or in other words, we are fitting strategies to past data. The success depends on the correlation between past and future. Nobody knows the future, so we try to make the correlation as robust as possible. 

Is that the reason for averaging?
Do you plan to automate the fitting further? i.e. Automatically iterating one parameter. At the moment I just make a 50 to a 40:60,10 and so on.

If so, then you could also iterate over strategies to detect fundamental changes in the market to switch to a better fitting strategy. Bigger scale trend changes, increases in volatility

Good questions,

Past vs Future

Yes, thus we should not read to much into a single backtest. People that post insane result for a single backtest is quite annoying since they do not understand that a single backtest is worth next to nothing. I could post results that yields 2+ TRILLION percent profit but simply don't since some users somehow expects such insane results is doable in the future. Thus it only leads to a lot of silly questions that i simply don't have time with (i do appreciate good questions though!).

But... if you got 1000+ runs with a larger spread of values that gives you an "OK -- this shit actually was profitable" you should, statistically, have something that is much better then just a single run. It's the same with sciences where 1 test yielding a positive outcome isn't considered valid since we just got 1 data-point with 1 setup that has proved it. It's commonly called "Sigma" and is sort of the value of certainty.

This is a very scientific topic though so i won't go to deep, for further reading see e.g:
http://www.physics.org/article-questions.asp?id=103

This is also the reason why one can sort by other parameters and not just "Most profitable" since "Most profitable" just mean most profitable for that specific combination of parameters and for those specific points in time. I consider the "most profitable" to be the cieling of what is possible, not as something that comes close to "these params will yield this result" since that would, statistically, be a very uneducated stance (short: it would be insane).

Please do note though that if you too quickly optimize parameters you might also be overfitting the strategy (Google 'backtest overfitting of data').

This is a typhical example of overfitting:
1. Use the tool to do 100 runs (way too few)
2. Stop it and check the most profitable runs parameters
3. Use that as a baseline and do +/-10 on the params
4. All runs profit!

Good way:
long = 100:2000,100
short = 10:90,10

The bad way:
long = 100:2000,1
short = 10:90,1

Good: The good version of this uses large stepping which is good since it gives us an idea, a "in the ballpark" value, which is what we are after.
Bad: The bad version tries to find the most profitable only and doesn't care for the "in the ballpark" value (we don't know the future so 100% best backtest matters less). It will also yield a lot of similar results and thus the average of "this works" will be worse.

-

Averaging the results

Yes -- this is the reason the average is interesting. Since a single backtest does not matter one optimally wants an average for at least 1000x runs. That average will most probably be more future proof since if it's an average of say 1000x runs it is something that has proven to work 1000x (and not just 1 single time).

If we get a little bit more scientific i would even say that one would need at least 10 000 runs in order to have some level of certainty.
You might now understand why i needed a tool to do this automatically, imagine doing this in the regular Gekko UI... there are nicer ways to commit suicide.
1000x runs should get one in the ballpark though since it starts painting a pretty clear picture when it comes to what works and not.

Here's an example of using the RBB ADX strategy with a XMR/USDT pair:
https://i.imgur.com/RCKYm0k.png


So 1,4 trillion profit?!
No -- It's safe to say that the 1,4 trillion result is basically garbage because -- see the rest of the 19 runs? All those, together, are much more important.The 1,4T+ result is an outlier, an odd set of params that happend to yield something the best profit in this specific case. It should be treated as such.

The average part of the 'view' will also be expanded upon.
I'm rewriting the core at the moment so expect the next version of the tool to have breaking changes (which mean that old results will not work since stuff such as the database columns etc have changed and thus the parsing will change to reflect that etc).

-

Iterate-per-param

No -- no plan exist to iterate over 1x parameter and just changing it one after the other since i don't see the point. If you just want to change a single parameter just make that specific parameter dynamic (A:B,C) and leave the rest?

-

Auto-change strategy

This is outside of the tool since it actually doesn't do any of the testing, it just sends params for Gekko to test.
If one wants such functions one would have to code a strategy that works in such a way. There have been examples of someone doing that so it should be possible.

[donkykong017]

This can be due to several reasons:

1. The curl_post() function simply died since your PHP is out of memory or something similar
2. The request took way too long time (over 60 minutes) so your curl request timed out
3. You have some settings in your server that overrides all settings that GAB tries to set and thus curl times a bit too quickly (and thus returns error)
4. You run with too many threads which creates a long que of requests since your CPU simply cannot run all the requests at the same time.
Thus you hit the AJAX timeout --OR-- your servers max execution time and thus curl_post() returns nothing.

Try not using over 5 threads and check your CPU-usage to get an idea what the max threads you actually can run at the same time is before your CPU spikes and uses 90-100% of it.

Setting threads to 10+ requires quite a beefy CPU and if your CPU is at 100% (or close) when using just 3-4 threads it means that the rest
of the runs will be qued up (it simply won't run any faster; you've hit your limit).

In future versions i'm considering being able to have an array of servers instead of just one but that's such a super-nerdy thing that few would use so it isn't a top priority and it comes with a plethora of problems (like the different servers needing the exakt same datasets, strategies etc -- so all of them have to be in 100% sync and this should ofc also be checked etc....).
Also real error messages and not stuff like "something is wrong" will be added.

Thanks for helping out to test the tool.

thx man for the reply.
i tried to look into it a little further.

4. i have a pretty new machine that should handle up to 8 backtests and changing the settings there did not solve it. it just slows the rate the errors occure.
having said that:
1. i suspect it having something to do with that. as it occurs more often after some time. i will have a look into this. i sadly have no clue about apache and PHP.
3. i did not change anything in my server settings. i just changed the root path for application access to the path where GAB is saved.
2. there could also be something with this. as now when i start GAB it shows 4-5 of those errors instantly as if there were some backtests stuck in the "pipeline" and apon starting give no results anymore.

but this is not a big issue at the moment as it gives actual results that can be used further. the only question with this is. if i restart the testing on a already used dataset were actual results exist will GAB know which tests with which params were already used or will it start from scratch just populate the result file even more. this could lead to the exact same Backtest results in the file and when restarted more and more often screw (a little bit) with the results. although i believe it would not have much on an impact when you consider you should run 1000+ tests.

there were some other errors i just don't have anymore in the log output. about dividing through zero and one other i can't remember just now. will report back when they occur again.
edit: just popped up

12:43:44
Warning:  max(): Array must contain at least one element in /home/david/gab/runner.php on line 288

Warning:  min(): Array must contain at least one element in /home/david/gab/runner.php on line 289

Warning:  Division by zero in /home/david/gab/runner.php on line 299

Warning:  Division by zero in /home/david/gab/runner.php on line 304

edit 2:
just had a look at the code and it has to do with best and worst trade and averages of trades:

PHP Code:

// best and worst trade
            $trades['best'] = number_format(max($trades['win_avg']), 2);
            $trades['worst'] = number_format(min($trades['lose_avg']), 2); 


PHP Code:

// calc averages for win and losing trades
            $count = count($trades['win_avg']);
            $total = 0;
            foreach( $trades['win_avg'] as $num ){ $total += $num; }
            $trades['win_avg'] = number_format($total/$count, 2);

            $count = count($trades['lose_avg']);
            $total = 0;
            foreach( $trades['lose_avg'] as $num ){ $total += $num; }
            $trades['lose_avg'] = number_format($total/$count, 2); 


another thing that came to my mind was that i updated my dataset with the candles of the last days and started testing on that one. GAB consider that as a completely new test and saved a new output file. wouldn't it be a good option if we could extend that result file with further test on newly downloaded candles over time? i'm not an expert but i think it would make the strat even saver, right? i think i would be best to test it on a dataset as long as possible to avoid overfitting. that would generate a strong strat for a long period of time. 
otherwise one could overfit a strat on a short period of time to have a strat that is good to be used on special movements of the marked. e.g. creating a strat for bullish/bearish markets. what would you say to that?

also looking forward to the next update of GAB
thanks again for sharing that. this tool is awesome. also thanks for your detailed explanation. no offense man, but you are crazy...in a good way 


cheers DK

[tommiehansen]

1. Make sure you run a recent PHP version. A problem could be that you're running something old; goto your terminal and write php -v. Don't run versions prior to PHP 7.
2. "pretty recent" doesn't mean much, if you want to know your max check your cpu-usage when running 1, 2, 3.. threads.
3. The errors you're seeing is because it could not add the trades to the $trades array. Thus you also see 'division by zero'. I don't know how to solve that since it makes no logical sense since there should be values there. This could be because of old php version that does things differently.
4. No appending results to a date range would be terrible since different date ranges always yield different results and thus SET A shouldn't be compared to B since they would live in completely different universes backtesting-wise.

Btw -- if my answers seem a bit short it's simply because i got a lot of stuff to do.