Index | Recent Threads | Who's Online | Search

Posts: 42    Pages: 5    Prev 1 2 3 4 5 Next
Last Post
New Thread
This topic has been viewed 1644 times and has 41 replies
abwillingham
Re: Are we over fitting?

5. Exclude top performers
P123 enables you to exclude the stocks which perform best in a simulation. If a model's performance crashes down after removing its best performers, odds are it was "lucky" (a.k.a. over-fitting).


How does one exclude top performers?

Jan 9, 2022 1:41:31 PM       
danparquette
Re: Are we over fitting?

Run the simulation. Go to the Transactions, Realized tab. Click on the 'Pct' column so the top performers on at the top of the list. Make a list of the tickers of the top performers. If you want a large number of tickers like the top 20, then download the list of Realized Transactions so you can copy the top 20 tickers rather then typing them.

Click Re-Run and go to the Periods and Restrictions tab. Paste the ticker list in the Restricted Buy List. Those tickers will not be bought when the sim is rerun.

Another option is using Buy rule like Random<=.5 to eliminate 50% of the tickers in the universe at random. Then run the simulation several times. The Optimizer works well for this. Just set it up to run the same combination many times and each one will apply the Random rule. If the returns are consistent, then that is a good sign that the system is not over optimized.

Dan

Jan 9, 2022 3:56:52 PM       
abwillingham
Re: Are we over fitting?

Thanks Dan. That is an interesting approach to anti-curve-fitting.
I have been using Yuval's suggestion of creating multiple (5 in this case) randomized universes like this:
Mod (StockID,5) = 1
Mod (StockID,5) = 2
Mod (StockID,5) = etc...

But to be honest I am not sure I am using it effectively.
If I have 5 universes, I will test against 3 or 4 of them, and then try the fourth/fifth at the end, which is always much worse, and proves I am fitting, big time. This leaves me scratching my head about what to change in my model.
I think what this means is that the out of sample universe that is tested at the end is what will be closest to my actual future results.

Using your approach it seems that if you run enough iterations, you will end up with the same over fitting problem.
If I do it 1MM times, isn't that essentially the same as not splitting up the universe at all?
And isn't testing all about running as many iterations as possible, using as many different combinations of factors and weights as possible?
I think this may be my biggest problem in quant modeling.

Jan 10, 2022 12:57:28 PM       
Edit 3 times, last edit by abwillingham at Jan 10, 2022 1:01:20 PM
yuvaltaylor
Re: Are we over fitting?

Thanks Dan. That is an interesting approach to anti-curve-fitting.
I have been using Yuval's suggestion of creating multiple (5 in this case) randomized universes like this:
Mod (StockID,5) = 1
Mod (StockID,5) = 2
Mod (StockID,5) = etc...

But to be honest I am not sure I am using it effectively.
If I have 5 universes, I will test against 3 or 4 of them, and then try the fourth/fifth at the end, which is always much worse, and proves I am fitting, big time. This leaves me scratching my head about what to change in my model.
I think what this means is that the out of sample universe that is tested at the end is what will be closest to my actual future results.

Using your approach it seems that if you run enough iterations, you will end up with the same over fitting problem.
If I do it 1MM times, isn't that essentially the same as not splitting up the universe at all?
And isn't testing all about running as many iterations as possible, using as many different combinations of factors and weights as possible?
I think this may be my biggest problem in quant modeling.


It is indeed a problem. Can overfitting be entirely avoided? I don't think so.

My approach is a little different. I optimize (somewhat roughly) for each of my five subuniverses and then combine those optimized systems for my final system, which will (obviously) backtest very well. There might be significant differences between the optimized systems but they get averaged out. So, in the end, I don't test on hold-out data. That could be a bad thing. My problem is this: if I were to reserve some data as hold-out data that isn't tested, what will my threshold be for performance? I'm sure that it'll be worse. How much worse is acceptable? What will I get from doing this?

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Jan 10, 2022 2:47:45 PM       
danparquette
Re: Are we over fitting?

Hi Tony,
Regarding the approach I mentioned where I use Random in the buy rules... Looking at the average return from the set of sims is not meaningful for the reason you gave. You want to look to see if the returns are consistent. Look at the returns from the worst performing sim in the group - would you be satisfied if your live port had the same performance?

Dan

Jan 10, 2022 3:11:05 PM       
abwillingham
Re: Are we over fitting?

Yuval, that is exactly my problem: what do to with hold-out data results that look crappy/good/whatever compared to my in sample data.

Let me restate how I interpret what you said. You have 5 universes and you have a separate, optimized RS for each one.
And you average those 5 RS into one RS. How does one average ranking systems together?

Dan, I think I understand your approach now. Back to my original question. Is there not a more automatic way to filter out extreme over-performance stocks?

On a general note, lots of very smart people have been doing quant modeling for a long time. Seems like an established best-practice method that everyone uses would have emerged by now.

Jan 10, 2022 3:31:54 PM       
Edit 2 times, last edit by abwillingham at Jan 10, 2022 3:34:41 PM
Jrinne
Re: Are we over fitting?

I don't test on hold-out data.


I just want to add that while ideal, a holdout test set is difficult in practice.

If you look at value factors for the last 5 years, don't you already have an idea of what you will find? There is no true holdout test set for factors you have used before.

Also, it is human nature to peek at the data before you are done. Or do a little tweaking later--after you have already seen the results in the test set.

For some pricing data--where one does not have some idea of what to expect--it is easier.

Jim

Great theory, "and yet it moves."
-Quote attributed to Galileo Galilei (1564-1642) gets my personal award for the best real-world use of an indirect proof or reductio ad absurdum.
`

Jan 10, 2022 3:38:39 PM       
Edit 2 times, last edit by Jrinne at Jan 10, 2022 4:39:28 PM
abwillingham
Re: Are we over fitting?

Jim, what method do you use to combat over optimizing?

Jan 10, 2022 4:02:11 PM       
Jrinne
Re: Are we over fitting?

Hi Tony,

So, I have no problem with what Yuval and Dan have said. And I was just trying to support the idea that holdout test sets are difficult. The other problem with holdout test sets is it takes more data to be able to use a test set.

But as far as what I do: I frequently use criss-cross validation that I found in a text book a long time ago. Validation is different than a holdout test set, BTW.

So as an example I recently trained some data (for a random forest classifier) from 2006 until 2013. I then validated it on data from 2014 till now. Perhaps that is the "criss" in the nomenclature. Maybe this is the most reliable data as the training data occurs before the validation in time.

But there is no downside to looking at the "cross." In other words training on 2014 until now and then validating on 2006 to 2013.

The other thing that is possibly significant in what I do is that I don't really tune the hyperparameters much anymore (much of the time anyway). For random forests as an example, I just use the default (again much of the time). This reduces overfitting.

When there is not tuning or the hyperparameters the difference be between a test set and validation set gets a litter blurred. But if you have ever done anything like what you are doing now with the same data you have looked at the data before. If you are honest with yourself you will not pretend it is a test set.

Obviously, I do different things much of the time (e.g., I do not always use random forests). But the criss-cross is easy for a lot of methods and sometime spending too much time tuning the hyperparameters is nothing more than overfitting. This goes for most methods

Hope that is at least somewhat clear if not helpful.

Best,

Jim

Great theory, "and yet it moves."
-Quote attributed to Galileo Galilei (1564-1642) gets my personal award for the best real-world use of an indirect proof or reductio ad absurdum.
`

Jan 10, 2022 4:55:50 PM       
Edit 5 times, last edit by Jrinne at Jan 10, 2022 5:37:55 PM
abwillingham
Re: Are we over fitting?

Jim, I have no idea what you just said but it sounds impressive.
Maybe I need to read the book you referenced.

Tony

Jan 10, 2022 5:00:51 PM       
Posts: 42    Pages: 5    Prev 1 2 3 4 5 Next
 Last Post