|
Re: Lapsa's very own thread
[Re: Lapsa]
#485405
03/07/22 22:15
03/07/22 22:15
|
Joined: Aug 2017
Posts: 294 Netherlands
Grant
Member
|
Member
Joined: Aug 2017
Posts: 294
Netherlands
|
But that's in-sample (right?), which makes this return completely irrelevant. Why not reserve some history for out-of-sample testing? This will speed up your dev process.
Last edited by Grant; 03/07/22 23:37.
|
|
|
Re: Lapsa's very own thread
[Re: Lapsa]
#485409
03/08/22 06:31
03/08/22 06:31
|
Joined: Aug 2021
Posts: 237
Lapsa
OP
Member
|
OP
Member
Joined: Aug 2021
Posts: 237
|
What do you mean by in-sample?
That's the whole history there is. MATIC is around for couple years. I'm just ignoring the very initial stage when it had no traction at all.
What's the point of keeping out-of-sample history if I'm gonna test it anyway? Doesn't that sort of make it in-sample too?
I don't get your point.
----
> But that's in-sample (right?)
I didn't fine tune it for a single week and then extrapolated expected returns. I fine tuned it for the (!) whole meaningful history I got. And those are the numbers that come up.
Last edited by Lapsa; 03/08/22 07:14.
|
|
|
Re: Lapsa's very own thread
[Re: Lapsa]
#485410
03/08/22 13:37
03/08/22 13:37
|
Joined: Aug 2017
Posts: 294 Netherlands
Grant
Member
|
Member
Joined: Aug 2017
Posts: 294
Netherlands
|
Let's compare strategy development cycles in a nut shell..
Yours: take the full data set (in-sample) for tuning > skip out-of-sample testing > test it right away on live market data > weeks go on and now you realize that it performs poorly (most cases) or you have an unicorn.
The traditional way: take 75-90% from your data for tuning > test it out-of-sample on the rest of your data set > you realize almost right away that it performs poorly (most cases) or you have an unicorn > once you have that unicorn, you run it on live market data for a final test ride (better safe than sorry).
Advantages of the traditional method: you save much time and you can compare out-of-sample results from multiple strategies / tune-settings (very important).
Your strategy has much potential when I look at the in-sample results, but your tuning method leads to over-fitting. That's the main weakness that you need to fix.
Last edited by Grant; 03/08/22 15:08.
|
|
|
Re: Lapsa's very own thread
[Re: Lapsa]
#485416
03/08/22 18:47
03/08/22 18:47
|
Joined: Aug 2017
Posts: 294 Netherlands
Grant
Member
|
Member
Joined: Aug 2017
Posts: 294
Netherlands
|
I can appreciate your honesty. Setting up successful algo trading isn't like just baking a cake There's no golden standard for robustness/over-fitting (both are -IMO- pretty much the same), so I can only speak for myself. I look for decent OOS statistics, enough trades (say 50+), combined with a decent PF (say 1.25+). More precise would be to look at the performance offsets between your in- and out-of-sample stats. To read it from a different perspective, check this interesting blog article https://quantlane.com/blog/avoid-overfitting-trading-strategies/Believe me, even with 10 years of data with 1000+ trades you can have over-fitting, so a large in-sample set is not a guarantee for robustness. This largely depends on the complexity of your approach, the more complex, the higher the likelihood of over-fitting.
|
|
|
Re: Lapsa's very own thread
[Re: Lapsa]
#485418
03/08/22 22:24
03/08/22 22:24
|
Joined: Aug 2017
Posts: 294 Netherlands
Grant
Member
|
Member
Joined: Aug 2017
Posts: 294
Netherlands
|
sadly - I find that blog article empty. nicely written, good theory overview, near zero practical value
It provides basic guidelines. Don't expect effective 'recipes' in this secretive area. > enough trades (say 50+)
that's like couple days
to my mind - tells nothing
That's just a number to give an indication, just like reserving 10-25% from your data set for OOS testing. > not a guarantee for robustness
I don't believe there is any
True, but I provide you some guidelines to increase the likelihood. Up to you what do with that, it's your broker account. > This largely depends on the complexity of your approach, the more complex, the higher the likelihood of over-fitting.
already mentioned - think it's much more important when working with machine learning
> Believe me, even with 10 years of data with 1000+ trades you can have over-fitting
I know it's there. just don't think it's that easy to pick it out by delaying tests on some particular data
Yes, this is esp true with ML, but you can basically over-fit any method. I don't know about the complexity of your approach, but those in-sample stats are way too optimistic.
|
|
|
Re: Lapsa's very own thread
[Re: Grant]
#485419
03/09/22 06:02
03/09/22 06:02
|
Joined: Aug 2021
Posts: 237
Lapsa
OP
Member
|
OP
Member
Joined: Aug 2021
Posts: 237
|
but those in-sample stats are way too optimistic.
well... those are just numbers my expectations are slightly lower - I expect it to be profitable and that's all
Last edited by Lapsa; 03/09/22 09:29.
|
|
|
|