Index | Recent Threads | Who's Online | Search

Posts: 42    Pages: 5    1 2 3 4 5 Next
Last Post
New Thread
This topic has been viewed 1649 times and has 41 replies
tobiasberr
Are we over fitting?

I strongly recommend reading this presentation
http://www.davidhbailey.com/dhbtalks/dhb-london-quant.pdf

or this presentation
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2606462

or this paper http://www.ams.org/notices/201405/rnoti-p458.pdf

Then think about it and ask yourself "Am I over fitting?"

And even if I am not over fitting, am I perhaps being fooled by intragenerational or intergenerational over fitting?
(meaning I am basing my systems on publications of people how have done over fitting or how have themselves been victims of intra/intergenerational overfitting)

This post is an appeal to everyone to engage on serious soul searching about their own sims and work so far.
Just reflect on it and your own approach.

Do not go off in anger and start posting your defense, why your ports are not over fitted.....

P123 is the best! Many Thanks to Marco and the crew!!!
Anyone near Augsburg/Munich, Germany wanting to meet up, just drop me a message.

Sep 9, 2016 2:02:53 AM       
InspectorSector
Re: Are we over fitting?

"Over-fitting" unto itself is not bad provided that the investor understands what he/she is doing and why. What is bad is when that person tries to market the results as performance. For example, if I were to make claims that my market timer will produce an equity curve straight up to the heavens as in Market Timer: Summary Of Ranking System Factors, then that would be bad. But laying out clearly what was done / assumptions made and letting the reader decide what if anything is of interest to them, isn't bad, even if it is over-fit.

Your second link provides some insight:

"Paradoxically, some of the best hedge funds are math-driven:
–Financial firms can conduct research in terms analogous to Scientific laboratories. E.g., deploy an execution algorithm and experiment with alternative configurations (market interaction).
–Financial firms can control for the increased probability of false positives that results from multiple testing. Their research protocols can legally enforce the accounting of the results from all trials carried out by employees.
–In the Industry, out-of-sample testing isthe peer-review. If you don’t make money, you are out of business. Corollary: Backtestcarefully or die.
–Financial firms do not necessarily report their empirical discoveries, thus discovered effects are more likely to persist.
Unless this state of affairs changes, true discoveries in Empirical Finance are more likely to come from the Industry than Academia."

Now I honestly don't believe that counting the number of iterations is practical (certainly not on P123) and it leads to a case of false analytics. It just moves up the uncounted iterations to a higher level. i.e. do your test, count the iterations, calculate the deflated Sharpe Ratio, Pass or Fail the results (likely fail), then repeat until you pass. You will always have uncounted iterations.

Take care
Steve


Take care
Steve

Sep 9, 2016 6:51:28 AM       
Cyberjoe
Re: Are we over fitting?

My statistics/econometrics education has become a bit rusty, but in general I try to pay attention to the following rules to avoid over-fitting:

1. Limit the number of independent variables and rules
The more independent variables (such as cash flow per share or sales growth) and rules (such as SMA50>SMA100) you introduce, the higher the chances of over-fitting.

2. Do not run too many iterations
The more iterations you run, the higher the chances of arriving at a "great" model by chance.

3. Have a large number of transactions
Assuming key return/risk results are the same, I would rather invest my money into a stock simulation with 1000 trades over 15 years than an ETF sector rotation model that trades 10 times in 15 years.

4. Check parameter stability
I stay away from a model that falls apart when changing one of my (few!) parameters a little. For example, my universe consists of the top 10% EPS growing S&P500 companies and delivers 25% p.a., but including the top 15% S&P500 companies results in 10% p.a.

5. Exclude top performers
P123 enables you to exclude the stocks which perform best in a simulation. If a model's performance crashes down after removing its best performers, odds are it was "lucky" (a.k.a. over-fitting).

Sep 9, 2016 7:37:14 AM       
Jrinne
Re: Are we over fitting?

Tobias,
Thanks for the links to these articles.

All,
The last article makes a point that I have been making for a while: some types of optimization are more harmful than others. This is best illustrated by the images below from the article. In the first image the out-of-sample results are not as good as the in-sample results. But the out-of sample results are not made worse--on average--by following the strategy compared to no strategy at all. Illustrated by the horizontal regression line. In the second image the strategy causes you to do worse than you would have if you followed no strategy (on average). Illustrated by the downward-slopping regression line.

So what factors are more harmful than others? I'm not sure I have a good answer to this but I think it is important and I will try to start the discussion.

I will really have to think about what factors are not harmful or not as harmful. Maybe they are all harmful. But the second image has an answer as to what is almost certainly harmful. It talks about serial correlation. This usual refers to a time-series. So market timing and using get-series too much could cause harmful over-optimization.

BTW, I am more and more convinced that CyberJoe is right about all of that. More right with some factors than others.

-Jim

Attachment Screenshot 2016-09-09 08.54.05.png (525770 bytes) (Download count: 176)


Attachment Screenshot 2016-09-09 08.54.23.png (552511 bytes) (Download count: 182)


Great theory, "and yet it moves."
-Quote attributed to Galileo Galilei (1564-1642) gets my personal award for the best real-world use of an indirect proof or reductio ad absurdum.
`

Sep 9, 2016 8:19:56 AM       
Edit 7 times, last edit by Jrinne at Sep 9, 2016 8:46:43 AM
InspectorSector
Re: Are we over fitting?

So market timing and using get-series too much could be harmful.

Jim et All - There are problems with such analysis. They are making generalizations out of selective (and sometimes silly) cases. It isn't difficult to reference Fibonacci or astrology then throw everything out (including the baby) with the bath water. For example, there are market timing strategies that have worked for decades, such as New NYSE 3 Month Hi/Los.

Steve

Sep 9, 2016 8:48:22 AM       
Jrinne
Re: Are we over fitting?

Steve,

I do not think the article said testing and optimization of any factor is always bad. The goal would be to avoid OVER optimization.

I certainly do not have all of the answers on how to do this. If the authors do they have not given out all of their secrets in this short article. Actually, I think CyberJoe has a good start on this topic: how to avoid the overuse of potentially harmful factors.

-Jim

Great theory, "and yet it moves."
-Quote attributed to Galileo Galilei (1564-1642) gets my personal award for the best real-world use of an indirect proof or reductio ad absurdum.
`

Sep 9, 2016 8:55:53 AM       
sgmd01
Re: Are we over fitting?

Some of the more successful/profitable hedge funds are currently avidly hiring quants (scientists, mathematicians, and programmers) to mine the data for profitable trading/investing strategies. They are looking for the optimization "sweet spot" and likely employing the ideas in Bailey's latest paper: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2819847

Scott

Sep 9, 2016 9:11:02 AM       
InspectorSector
Re: Are we over fitting?

Jim - I appreciate the distinction CyberJoe and you are making. However, I have doubts that we can discern the difference between optimization and over-optimization. Think about it for a minute. The first author says that the markets are effectively made efficient due to HFTs and arbitration strategies. With markets apparently this difficult, how does one propose a theory, test it with extremely few iterations, and come to the conclusion that it "works"? In my opinion, this is never going to happen and not a practical suggestion. These authors are blind to the fact that one has to optimize/over-optimize at some level, if not at the backtest level, then at the theory proposal level. In other words, somewhere along the line, they are not counting iterations.

If I read you correctly, you were suggesting that market timing was detrimental. Hence my last response.

Steve

Sep 9, 2016 9:25:35 AM       
yuvaltaylor
Re: Are we over fitting?

None of the three papers consider the differences between technical analysis and fundamental analysis, which are huge. Garbage in, garbage out needs to be the first principle of backtesting. If you're trying to optimize a strategy based on stuff that makes no financial sense--stop losses, Sharpe ratios, best days of the week, trends, and so on--you're going to be ensnared by all sorts of traps. Overfitting when backtesting using fundamentals can happen too, but there are lots of ways to make backtests more robust. The answer isn't to backtest less, but to examine every measure to make sure it makes logical financial sense. If it doesn't, either throw it out or improve it. Now matter how good the data is, you can't stand by a measure that you can't explain. Personally, I have found that including measures that relate today's price to a price anytime in the past have produced much more unreliable results than sticking with fundamentals. I would bet that if the authors of these papers compared backtesting using past prices to backtesting using fundamentals, and if they stopped relying on the Sharpe ratio (which has almost no predictive power) and relied instead on a simple CAGR, they'd find that their conclusions need to be altered.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Sep 9, 2016 9:26:15 AM       
InspectorSector
Re: Are we over fitting?

Yuval - everything you say is right on, but there are some issues with "examine every measure to make sure it makes logical financial sense". The problem is that what makes financial sense doesn't consider market behavior, what makes sense may still result in significant losses. That is why we try to capture market behavior the best we can through optimization.

Steve

Sep 9, 2016 9:36:21 AM       
Posts: 42    Pages: 5    1 2 3 4 5 Next
 Last Post