Index | Recent Threads | Who's Online | Search

Posts: 19    Pages: 2    1 2 Next
Last Post
New Thread
This topic has been viewed 623 times and has 18 replies
Jrinne
Overfitting: A visual example

ANN is an artificial neural network. She (Ann) is like I was when I first started at P123. She has one purpose: To continuously optimize and improve the in-sample data she is given.

She does this well. The top graph shows continuous improvement as she works on optimizing the data--lower is better.

For a while her out-of-sample results improve too (second image). But after a while (about epoch 30 on the x-axis) she starts to overfit. Her in-sample results continue to improve but the out-of-sample results relentlessly and continuously worsen.

This is just the best graphical example of overfitting I have encountered. The reference to Ann is incidental. Ann just keeps track of her results better than I have. This is just something that helps me understand my own overfitting a little better.

-Jim

Attachment ANN fits.png (34108 bytes) (Download count: 112)


Attachment ANN Overfits.png (57392 bytes) (Download count: 111)


From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Jan 16, 2020 5:50:53 AM       
Edit 6 times, last edit by Jrinne at Jan 16, 2020 6:03:56 AM
philjoe
Re: Overfitting: A visual example

What does the y axis represent?

Jan 16, 2020 6:03:16 AM       
Jrinne
Re: Overfitting: A visual example

Philip,

Jargon alert (DEFINITELY recommend skipping this): This is the loss or cost function. In this case, the loss function is logcosh which deals well with outliers and non-normal distributions.

Less jargon but you may have had this in class: It can be though of as no different than a linear regression which minimizes (makes as small as possible) the root mean squared error.

So, an error of the predicted returns compared to the actual returns of a stock—many error functions available.

To the point and accurate: tells you how far off the model is on the predictions of the best stocks to buy. And, I think, is therefore applicable to what we do.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Jan 16, 2020 6:12:28 AM       
Edit 7 times, last edit by Jrinne at Jan 16, 2020 7:54:44 AM
RTNL
Re: Overfitting: A visual example

Thanks for trying to enlighten us.

As an ignoramus in neural nets, can you please help me understand, if 30 is datapoints or variables? (variables would be equiv to factors, datapoints would be performance)

Jan 16, 2020 8:06:20 AM       
Jrinne
Re: Overfitting: A visual example

Thanks for trying to enlighten us.

As an ignoramus in neural nets, can you please help me understand, if 30 is datapoints or variables? (variables would be equiv to factors, datapoints would be performance)

LOL;-)

Epochs are directly equivalent to each rank performance test you run. This is the simple answer to your question.

So to continue the example, initially when you add factors and adjust the weights your (and Ann’s) in-sample and out-of-sample results both improve. Improve on the epochs and the rank performance tests you run.

Some time before you add data on the business school that the CEO graduated from it is only the in-sample data that is improving. You are unaware that you are harming your out-of-sample results.

But ANN can tell exactly when this harm starts happening with her holdout data. We have to wait a few years for our out-of-sample data (assuming we have kept good enough records).

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Jan 16, 2020 8:19:52 AM       
Edit 1 times, last edit by Jrinne at Jan 16, 2020 8:22:50 AM
RTNL
Re: Overfitting: A visual example

thanks Jim

Jan 16, 2020 11:50:44 AM       
Schm1347
Re: Overfitting: A visual example

I saw an example where you have scatter plots and you try to do a line of best fit. As you add more variables the line turns into a curve then a squiggly. When you overfit the contorted line eventually intersects each point. But the problem is by that point you have tweaked the best fit line to also fit the noise.

The market is apparently noisy. That doesn’t mean the noise is necessarily random but this noise can appear random. We need to first develop sound models in theory and then test those. However there is always some refinement. It’s inevitable. The point is that we actually want to ignore that noise because we can’t explain it. If it can’t be explained it is dangerous to model it. So the key is understanding what actually improves our model through causality and not just correlation. Correlation without grounding in causal theory is noise fitting. I inquired about noise injection because it is one tool to break a model. If a model is curve fit to noise it will break very easily when noise is added or subtracted in the mix. Is this an end all. Absolutely not. Models have to be grounded in sound theory of why the securities being picked are mispriced and have potential to outperform.

Jeff

Jan 16, 2020 8:21:20 PM       
Schm1347
Re: Overfitting: A visual example

As far as ANN goes I question how helpful it might be though. If you are fitting in sample data and keep optimizing and then ANN tells you are are now overfitting that is relative. How do you know you weren’t overfit to the in sample and out of sample data before ANN flagged there was a problem? You don’t. What prevents you from then trying to fit more after ANN tells you a change was bad? Perhaps I am not understanding or misunderstanding.

I actually think that out of sample testing doesn’t really work for a few reasons. The first problem is that most models even good models won’t outperform every year. If you choose a really good in sample period and really bad out of sample period you might think you have a bad model when you actually have a good model. The reverse is also true. The other problem is if your out of sample data stinks do you just forget your model? No. Usually people will try to improve to get better out of sample data. Now your out of sample data is in sample. The other problem is even if you arrive at both good in sample and out of sample data how do you still not know that your model is overfit? You don’t again.

Marc is right. It comes down to sound financial theory. Even a sound model can stop outperforming or even underperform for a period of time. What worries me most about models most is not lacking outperformance but simply falling apart when I start trading it.

Jeff

Jan 16, 2020 8:34:29 PM       
Jrinne
Re: Overfitting: A visual example

I saw an example where you have scatter plots and you try to do a line of best fit. As you add more variables the line turns into a curve then a squiggly. When you overfit the contorted line eventually intersects each point. But the problem is by that point you have tweaked the best fit line to also fit the noise

Jeff

Jeff,

You are EXACTLY RIGHT about this. In fact, IMHO, you could not be more right.

That is why I defended you so strongly on you ideas about adding noise to the data.

If you are interested I will show you Ann 2.0 who uses some of your ideas. Your ideas have been studied by numerous Ph.D.s for decades now and your ideas have been expanded upon just a bit.

Anyway, Ann 2.0 does not overfit. Ann 2.0 does not overfit because you were right.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Jan 17, 2020 12:48:30 AM       
Edit 1 times, last edit by Jrinne at Jan 17, 2020 12:55:44 AM
Schm1347
Re: Overfitting: A visual example

Jim,

Yeah I’d be curious to see it.

Thanks,

Jeff

Jan 17, 2020 12:51:02 AM       
Posts: 19    Pages: 2    1 2 Next
 Last Post