Out of Sample

Posted By: danatrader

Out of Sample - 05/30/20 12:05

What is more significant?

Leave some Data out of training and run test against previous unseen data, or run test 1000x against DETREND = SHUFFLE?;

Posted By: AndrewAMD

Re: Out of Sample - 05/30/20 13:06

Why not both?

Posted By: danatrader

Re: Out of Sample - 05/30/20 20:02

Time.
How decide on contraindication?

Posted By: AndrewAMD

Re: Out of Sample - 05/30/20 21:12

Reject when either says it’s bad.

Posted By: danatrader

Re: Out of Sample - 05/30/20 22:05

Probably, better safe than sorry.

Run 1000 times against the artifical price curve of the previous unseen data, or against the artificial curve of the previously used train data.

Or again both?

Posted By: AndrewAMD

Re: Out of Sample - 06/01/20 01:30

Originally Posted by danatrader

Run 1000 times against the artifical price curve of the previous unseen data, or against the artificial curve of the previously used train data.

Or again both?

You should think about what you're testing, and why.

If you optimize to dataset A, and you shuffle dataset A 1000 times, your optimized unshuffled configuration will almost necessarily reign king. This is not useful at all.

Whereas if you optimize to dataset A, then check it against an OOS dataset B, then shuffle dataset B 1000 times, the results are actually useful. (You want the first OOS test to outperform random data with statistical significance.)