But that's in-sample (right?), which makes this return completely irrelevant. Why not reserve some history for out-of-sample testing? This will speed up your dev process.

What do you mean by in-sample?

That's the whole history there is. MATIC is around for couple years.
I'm just ignoring the very initial stage when it had no traction at all.

What's the point of keeping out-of-sample history if I'm gonna test it anyway?
Doesn't that sort of make it in-sample too?

I don't get your point.

----

> But that's in-sample (right?)

I didn't fine tune it for a single week and then extrapolated expected returns.
I fine tuned it for the (!) whole meaningful history I got.
And those are the numbers that come up.

Let's compare strategy development cycles in a nut shell..

Yours: take the full data set (in-sample) for tuning > skip out-of-sample testing > test it right away on live market data > weeks go on and now you realize that it performs poorly (most cases) or you have an unicorn.

The traditional way: take 75-90% from your data for tuning > test it out-of-sample on the rest of your data set > you realize almost right away that it performs poorly (most cases) or you have an unicorn > once you have that unicorn, you run it on live market data for a final test ride (better safe than sorry).

Advantages of the traditional method: you save much time and you can compare out-of-sample results from multiple strategies / tune-settings (very important).

Your strategy has much potential when I look at the in-sample results, but your tuning method leads to over-fitting. That's the main weakness that you need to fix.

Yeah, but you wait a month, test it again and you got your 10-25% out-of-sample data.

I really don't see a benefit.

What if we pick January 2022 as our out-of-sample data?
Why such data should be allowed to trump everything else just because it got ignored while fine-tuning?

Once your OOS results are good (which is far from easy), you only need another month or so once. Sure, you can skip that second test phase, but that would be foolish.

It's not about 'trump everything', it's about finding out right away how robust your strategy is. In most cases there's over-fitting, so you need to OOS test, adapt, OOS test, adapt, etc. over and over again before your strategy is tradeable.

I just fail to see it.

I mean, I do see some benefit in such approach.
Some rules indeed work only temporarily.

But I also feel it raises bunch of new concerns and uncertainties.

Just wanted to highlight one of them - actually picking OOS data.
No matter what's your approach on picking up robustness measurements - they would be based in OOS data.

I think my biggest problem is with the definition of over-fitting itself.
I don't know how to measure it (!) precisely.

Have seen and made algos that are obviously over-fitted and fail immediately on any different data.

But when it's such heaps of data and thousands of trades - I don't really believe it's that easy to over-fit.
More and more the results legitimizes themselves as The Unicorn.

--------

Btw, on actual performance - you might argue it's not half bad.
Sort of break even-ish month. 1 week was great (dunno, +30% or something) - flattened out by others.

Given the circumstances we currently live in - it's not really that much out of the line.
In 2021 - it shows about 3 flat months in a row. Ulcer's might not like it, but hey - that's the path I chose!

--------

Much of the frustration comes from the fact how hard it actually is to live trade.

You need bravery - sending out bunch of money and putting it on the line isn't exactly that easy.
Foolishness can help too (and backstab you later on).

You need resilience - even after hitting stop loss like 5 times in a row, that may or may not tell anything.

You need patience - those hours go by slowly. Even worse when you get bad gut feel predictions for days / weeks.

And then the stuff automagically happens and you are TOO LATE.
Either it failed or you are left with constant reminders that success ain't given freely and may very well disappear next week.

Ridiculous of me initially thinking that sound alerts are a good idea.

I can appreciate your honesty. Setting up successful algo trading isn't like just baking a cake

There's no golden standard for robustness/over-fitting (both are -IMO- pretty much the same), so I can only speak for myself.
I look for decent OOS statistics, enough trades (say 50+), combined with a decent PF (say 1.25+).
More precise would be to look at the performance offsets between your in- and out-of-sample stats.

To read it from a different perspective, check this interesting blog article https://quantlane.com/blog/avoid-overfitting-trading-strategies/

Believe me, even with 10 years of data with 1000+ trades you can have over-fitting, so a large in-sample set is not a guarantee for robustness.
This largely depends on the complexity of your approach, the more complex, the higher the likelihood of over-fitting.

sadly - I find that blog article empty.
nicely written, good theory overview, near zero practical value

> enough trades (say 50+)

that's like couple days

to my mind - tells nothing

> not a guarantee for robustness

I don't believe there is any

> This largely depends on the complexity of your approach, the more complex, the higher the likelihood of over-fitting.

already mentioned - think it's much more important when working with machine learning

> Believe me, even with 10 years of data with 1000+ trades you can have over-fitting

I know it's there. just don't think it's that easy to pick it out by delaying tests on some particular data

It provides basic guidelines. Don't expect effective 'recipes' in this secretive area.



That's just a number to give an indication, just like reserving 10-25% from your data set for OOS testing.



True, but I provide you some guidelines to increase the likelihood. Up to you what do with that, it's your broker account.



Yes, this is esp true with ML, but you can basically over-fit any method. I don't know about the complexity of your approach, but those in-sample stats are way too optimistic.

well... those are just numbers

my expectations are slightly lower - I expect it to be profitable and that's all