The 4 sources of excess returns.

According to Patrick O’Shaughnessy in a recent podcast, there are 4 sources of excess returns.,

!) “Behavioral.” Other investors making systematic mistakes that you can capitalize upon.

  1. “Informational edge.” You have information (or data sets) that others do not posses. David (Primus) is a dedicated student of this source of excess returns.

3)" Analytical." You analyze every stock in your universe, in a systematic way with P123’s tools while others are using Excel spreadsheets with data from Yahoo looking at a few stocks that are being talked about on CNBC. But there areother methods besides the excellent methods P123 has to offer.

4)" Trading or Structural." Your algo at IB is better than others.

It would be a lot to expect one person to posses all of the wisdom in in the areas of behavioral and analytical.

By definition there are only a few people that have an informational edge—probably never shared at P123.

Trading, Perhaps it is most obvious with this that one would need to expand their reading beyond the P123 forum to get much of an edge here.

But there are cults you can follow—on the forum and elsewhere that can give you the felling that you posses all knowledge beginning to end.[b]" The Alpha and the Omega."

We are told that you can just look at the pretty buckets. Maybe we can make a feature request for some different color schemes. That is all you need according to one cult.

Who could argue with such an advanced analytical technique? Strange that Patrick O’Shaughnessy did not mention this one.[/b]

Sadly, Marc has lost the spirit for an intervention. While I was not an exclusive follower of his ideas either, at least Marc had some connection to mainstream ideas and he had a course in Econometrics with an understanding of what a p-value meant. And a degree to prove it.

[b]And recently, Marc has been open to multiple ideas, especially behavioral ideas. HE HAS SAID THE FINANCIAL FACTORS WORK BECAUSE OF MISPRICINGS CAUSED BY BEHAVIORAL FACTORS. THE SAME THING O’SHAUGHNESSY IS SAYING!!!

Marc does not post much now and we quickly get a final word on whether he is right or not from one of the other members.

I miss Marc. But you can find his ideas on Patrick O’Shaughnessy’s podcasts.[/b]

-Jim

For those who do not follow the forum closely and miss my point.

Completely ignoring any of the 4 sources of excess revenue that Patrick O’Shaughnessy mentions. Patrick would note that the factors themselves are not so important as the behavioral mispricing that surrounds the factor. Something that Marc understands.

But we do have a final answer.

I miss Marc but we can still get his ideas by listening to Patrick O’Shaughnessy.

-Jim

I can’t find the thread, but I think I’ve been harping on the idea that barriers-to-entry play a fundamental role in the ability to generate excess returns for a while now.

In order to beat the market, one must be able to do at least one of the following:

  1. Know something that others don’t (or know someone that knows something that other’s don’t) (informational edge)
  2. Be somewhere that other’s can’t (structural edge)
  3. Take an approach that others don’t (analytical edge)
  4. Take risks that other’s won’t (risk-asymmetrical / behavioral edge)
  5. Execute with greater precision and concision than other do (professional edge)

Because institutional quality fundamental data is so ubiquitous now (and because the pedigree gap is closing), I think we should rule out informational edge as a source going forward. However, I still think there’s some opportunity to chip away at 3, 4, and 5.

I think a recent post where people gave their optimization techniques (including David and Yuval) is all you need to understand. This is NOT meant as a criticism. I do this too. The technique is so simple that nothing could be added by me.

NO FINANCE, MATH, OR MACHINE LEARNING DEGREES REQUIRED FOR WHAT WE DO NOW.

We do HEAVY OPTIMIZATION.

What I object to is are the claims that some weird analysis is the secret of success.

Then in the next sentence the CORRECT statement that all we are doing is looking at the pretty buckets. That is what we do, indeed.

I am fine with some sort of blockade on mainstream ideas as long as we do not get something weird in its place.

I can optimize like anyone else. What I can do without is the hyped claims for Omega, or k-means clustering. If that means no printout of the information ratio, I am fine with that.

[b]I’m just surprise that Marc is okay with what P123 has become: pure optimization with no academic foundation whatsoever.

No mainstream ideas or degrees required. No degree in any related discipline required.

In fact, the less published it is, the further away from the mainstream it is and the more mathematical assumptions it violates the more likely it is to get hyped as the magical answer.

In truth, it is just a story added at the end of a series of heavy optimizations and looking at the pretty buckets.[/b]

-Jim

Somebody misses me? Aw shucks . . .

I’m surprised too. When did I ever become OK with that? I am open to machine learning and am personally still in the “learning” stage when it comes to this sort of thing. I’m always open to learning about something new, although I’m very comfortable ultimately turning thumbs down if what I learn is that it can’t really do what it’s supposed to do (as is the case with optimization, when I eventually learned how much a prisoner it is of its specifically defined sample).

Perhaps a better way to explain myself might be to temporarily drop financial talks and switch to another field undergoing a quant revolution; sports, where the word used is “analytics.” It’s become so prevalent, that baseball managers, once all powerful iconic figures, have seen their stature reduced to the point where they are little more than glorified functionaries stationed in the dugout to implement directives from the GM and the analytics department that feeds him information. But today, its no longer confined to baseball. It’s penetrated the NBA in a big way, and the NFL as well.

The NBA’s Houston Rockets are an interesting case study. Darrel Moray, their analytics-obsessed GM, has managed to build a team that wins a lot, a heck of a lot, and is a championship contender year in and year out. But lately, they’ve developed an alarming tendency to crap out very badly in the playoffs under scenarios when they entered as favorites, to if not win outright, at least fight to the bitter end.

One sports media personality who I like a lot (who often uses stock market language to express opinions including a weekly segment during football season called “Buy, Hold Sell”) Colin Cowherd, presented an interesting, and I think very persuasive take on the situation and it’s somewhat analogous to what I keep saying here about past and future.

Moray does, is, indeed, very successful in using analytics to solve the problem as he defines it – winning games. But he has not succeed in recognizing the difference between winning games and winning a championship. Moray’s modeling presumes that success in one will confer success in the other. But that is not so.

The 82 game regular season is a prolonged stretch in which the Rockets play everybody, good teams, middle of the road teams and bad teams. No single game is do or die (except maybe here and there towards the end as teams jockey for final playoff positioning). But players can pace themselves (and rest where needed within games and even by sitting some out). And no team really gets an opportunity to adapt in the moment to any other team because they don;t play the same team in consecutive games.

The playoffs are a completely different animal and all of Moray’s algorithms, that succeed so well over the 82, need to be trashed; either he needs a completely different model or would be better off using no model. In playoffs, you only play good teams, and each round, the opponent gets better and better. Rest and pacing (and even injury recovery) is a much more delicate balance becuase we’re only dealing with a tightly contained best of 7 series. (The defending champion Golden state Warriors got burned very badly by this issue and were dethroned.) Opponent can and do adapt because now, each team plays the ssame opponent until one team wins 4 games, which can take anywhere between 4-7 games.

Another big big big big difference that Moray completely missed is the climate of officiating, the tendency of refs to call fouls. Moray’s players, especially his stars, have exploited this so well, that hey even developed physical techniques, such as the land forward jump where they’ll go upp and instead of coming straight down, as normally expected, propel their feet forward under the feet of the opponent who lands on top of them and gets called for the foul. Getting the opponent called for a foul, putting their players in danger of fouling out (6 fouls and you’re out of the game) and their own players to the free throw line a lot is religion in Houston. It’s analogous to what Brad Pitt, playing Billy Beane in the movie “Moneyball” always said as the greatest attribute of a batter: “He gets on base.” Moray’s equivalent: “He gets to the free throw line.”

Moray completely miised the reality that in theplayoffs, all of which are nationally televised, the NBA does not want to destroy the visual pacing of the game by calling a lot of fouls and having too much of the game played from the free throw line.So the refs are much slower to call fouls and they often turn a blind eye toward contact that would have been whistled in the regular season. This, plus all the other differences, prove disastrous for Houston in the playoffs and this season was the worst, when the team was roundly ridiculed for whining about how officials weren’t calling fouls as often as they expected.

Is Moray a fool? Are analytics bogus? No on both counts. Analytics work and Moray’s analytics worked brilliantly , , , when he was correctly defining the problem. Nothing failed. It’s just that in the post season, he did not have an algorithm suitable for the task at hand. He missed out when it came to defining the problem. He’s a smart guy and I expect once he owns up to the nature of his challenge, he’ll address it and solve it . . . unless the teams’ new aggressive owner loses patience and fires him before he fully defines and solves his new task.

Now, back to our regularly scheduled p123.

My distaste for the way I see backtesting and sim used on p123 is precisely illustrated by the Darrel Moray conundrum. I see these as effective and often brilliant solutions – to the wrong problem. Investors can no more generalize from sample periods to live periods than Moray could from the 82-game drawn out foul-heavy regular season to the concentrated best-of-7 swallow-the-whistle playoff environment. This is exactly what I try to get at when I talk about how we can’t assume the past predicts the future.

Analytics in sports and analytics in finance are the same. Both are at their best when they are seen as taking advantage of modern technology and analytics techniques to help do human tasks more effectively and efficiently than human judgment alone could accomplish them. My DDM-based presentation is not a silver bullet or a formula. It’s a logical roadmap to help one define the problem to be solved on p123 or wherever via analytics.

If the problem is defined as showing strong equity curves in a sample period, then do not expect it to succeed in a live period unless one has the good luck to experience a live period that just so happens to resemble the sample period. And as it turns out, many who work this way on p123 have, indeed, benefitted from exactly that, just as the Houston Rockets would have benefitted had the ref been willing to call fouls in the playoffs the same way they did during the regular season, something that alone could have offset all their other playoff analytic deficiencies. And as it also turns out, the unusual persistence of historic trends has not favored my less momentum oriented approach, meaning my models have often not showed the live performance I’d have liked. But unlike many I’ve seen on p123, I at least know WHY my results have been what they were and, having a lot of experience in this business and recognizing the monstrous inflection point, or perhaps plateau, on which we’re sitting (the interest rate situation), I won’t alter models that are designed to be stable.

I suppose a reason why I’m not on the forum so often any more is to reduce the probability of a tantrum that is unsuitable to any forum posting. But you can be sure I see all and am at least mentally flaming the living daylights out of anyone who illustrates a point by showing sim results – or at least without accompanying such a presentation with a verbal explanation of why the poster believes the numbers/trend were what they were.

Backtesting is a great tool but it must be used properly. It’s not there for people to show fabulous equity curves. It is there to show how the kinds of stocks picked by your model perform under various conditions – to give your judgment an assist that was not technologically feasible 20 years ago.

I’ll close this with a link to a Seeking Alpha article I published nearly a year ago, in which I actually presented and discussed bad backtest results but explained why I was going ahead with the model anyway.

This is how I use testing, to help me see if I’m getting what I think I’m getting, not to allows me to show how great I am at producing vertical or near-vertical equity curves. (In fact, the more beautiful the illustrated results, the less likely I am to read the text of the post.)

It is good that you consider mainstream machine learning.

You should fight—with all of y0ur effort—fringe ideas.

If you do not know the difference you should hire someone who does before implementing any of them.

My point is this: I trust your present knowledge better than most. At least you have done some of your Econometrics homework and have calculated a p-value.

And you have a degree to show for it.

-Jim

And the reason I still have the degree is because nobody has tested me on whether I remember how to do it.

:slight_smile:

Your ideas on misspecification have been spot on. And you show your abilities elsewhere too. It is good to be humble, however.

Something else that I miss when you are not more active.

-Jim

Jim/Marc, I was just thinking about what you both wrote.

The context of using common statistics practices for backtesting.

When we build models, it is with a hope that it will work under ALL market conditions/regimes - no matter what. This implies that the in-sample backtesting we are doing are samples that are representative of the WHOLE population, going back to the first stock trading in the 1600s in Antwerp.

My understanding of one of the fundamental requirements of the use of statistics is to have a completely random sample and not one that is pre-biased in some way (say by ultra low interest rates).
I don’t think that the samples we are using now for backtesting come anywhere meeting that criteria. (I am not blaming P123 for this. It is the reality of what we have).
And therefore standard statistical techniques really do not apply - unless it is clearly specified what the underlying assumptions are that could affect the biases of the samples.

Otherwise, the use of p-values, etc is all nonsense. And ditto with the use of backtests. Misleading.

So if our goal is to end up with a model that will work somewhat well in ALL market conditions, we absolutely need to be clear to ourselves (or others, in the context of Designer Models) how the samples are situated in the context of the backtesting period and their representation of the WHOLE population for ALL scenarios.

What are the macro trends that are determining what our backtests are indicating?
And how are these macro trends related to other environments or our current environment?
How is this described in a conceptual, rigorous academic framework? Marc’s talking about the effect of interest rates on P/E ratios is an example, I think.

Frankly, other than Marc, I don’t see posts about that very much on the Forum. It would be good to see. Otherwise, I think we are just alchemists trying to create gold from iron (backtests).

David,

I struggled with this several years ago.

Bottom line is what we do is not “stationary.” Which just means all of your concerns (and ones Marc has mentioned in the past) are valid concerns.

What AQR and Patrick O’Shaughnessy do is called cross-validation: Using a cross-validation metric.

Ideally, using a standard cross-validation method where the test FOLLOWS the training period: something that P123 thinks is garbage.

We will never have a cross-validation metric at P123 that is in standard usage elsewhere. Nor will we ever have support for any standard cross-validation method.

Nor anything that uses a standard deviation for that matter.

-Jim

yes, I guess what I was really saying was that if there is no logical / fact-based narrative to describe what the designer thinks is happening during the period of the environment for the backtesting results and the effect on the model; that it is insufficient to just use statistics to describe the backtest results.

We all have 2 choices:

  1. trust an authority figure, But he/she uses a method whether they bother to tell you or not.

  2. use the best methods available recognizing the limitation anyone has for predicting the future.

There are some standard techniques that address any specific limitation regarding populations, samples, linearity, nonlinearity etc.

Worse–for me–than trusting an authority figure is being forced to use the authority figure’s method while they lecture me. Worse still is when they lecture me on things that they have no training in.

The situation now: lectures about statistic from someone who has never calculated a p-value and refuses to use standard deviation, ever. While they advocate the most fringe statistical methods possible.

Does that seem strange or is it just me?

-Jim

I’m not sure about this, David. Investors set prices through buying and selling stocks, and what motivated investors to buy and sell in 1969 may be quite different from what motivates them in 2019. When I build a model it is with the hope that it will work over the next three years. I have absolutely no hope that it’ll work in 2069.

I’m willing to be convinced that I’m doing things the wrong way. But that’s the way I approach backtesting: I want to have an edge over other investors today. I can’t possibly hope to maintain that edge forever.

I couldn’t even stick with a model that works under all conditions. The first time my model spit out $BYND and $TLRY as a buy suggestion, I would trash it, regardless of how in favor it was at the moment. Most of our models, at least ones that we actually use real money with, will be systematic manifestations of our already deeply held investing principles and beliefs.

That depends on who you are and what task you’ve chosen for yourself.

If you are an academic sort and are looking to address universal theories, then I suppose that is something at which you might aim. But if you do, be prepared for the results you get. You may, indeed, wind up with a model that does test well on that basis. But even those who work this way openly recognize, as they must, that there can be prolonged periods during which the norm doesn’t hold. If you’re in your 60s and facing retirement, would you be satisfied with a strategy that worked on the whole from 1600-2019 but which went cold in 2018 and may stay cold until, say, 2050, at which time it works again for 500 years?

It’s easy to say we want a universal strategy our desire for this sort of thing can quickly turn to revulsion when real dollars and cents are on the line. And this even assumes one can ever find such a strategy in the real world. The only thing constant is change, so if I were shown a strategy that did work all the way back from 1600, my first reaction would be to wonder where the errors were. That’s just not the way the word works. So it would take a lot to convince me that the study is legit.

I’m on treacherous ground here since I’m not a professional statistician, but I believe this is not a correct answer. There are actually many different kinds of samples and I believe that the random sample is not suitable to what we do. Instead, stratified sampling may be better (How Stratified Random Sampling Works, with Examples). There are many other sampling techniques and my memory is a bit sketchy so there may be things even better than stratified sampling Purposive sampling (Purposive Sampling Definition and Types) perhaps?

Marc,

I am going to admit to not knowing much about stratified samples.

Haven’t you recommended testing over periods in the past that are similar to what you expect going forward?

I am going to admit to being too good at knowing what to expect in the near future either. But you have been right in the past with regards to direction of interest rates etc.

Are there any particular periods that you recommend we do not backtest over? Ones that you think may be most representative? This seems like a good method for getting a sample (especially if one knows more about Finance than I do).

Thanks.

-Jim

Jim,

Double Damn.

Frist, I was hoping you were going to be the one to explain stratified and purposive sampling to e, and perhaps add some useful ones I missed. :-(( FWIW, I be;leive what you summarized from what I said in the past is in line with purposive sampling.

Second, I was really hoping nobody would ask what periods we should test because that question is so darned hard to answer. I’d say thje ‘70s. but we can’t do that because we don’t have the data and even if we did, I’m not sure it would be useful considering all the structural changes to the economy and financial markets since then. We (quant as a whole) are in a very precarious place right now as we switch to a regime for which we lack data. So now, whatever period I test, I won’t go forward if the last year’s rest result is something I can’t explain. I’m OK if it stinks, as long as I understand why it stinks.)

I think we’re starting to develop a new testable regime from early to mid 2017, but we’ll see. Now, more than ever, it’s important to understand results. I’d rather see a test that matches an expectation. of bad performance than good results that can’t be explained.

Yeesh, that answer is clear as mud. Maybe I’ll come up with something more coherent tomorrow.

Notice no one is claiming that good statistical methods are a source of excess returns.

I’ve never met a statistical test that told me something I already didn’t suspect to be true or false. In this sense, they are simply plot devices we can manipulate to intrigue ourselves with truisms and/or shape reality to fit our own narratives. Exploratory statistical methods, being complexity reducing tools, are different altogether.

Anyway, my point is that the mainstream never gets it right, but is usually slightly less wrong than the fringe. But sometimes the fringe surprises us by flipping convention on its head, even if it often lacks the language of ecclesiastical authenticity. Remember Soren Kierkegaard: “the crowd is untruth.”

David,

I am claiming it now.

Under the category of analytical in the original post (so not a contradiction).

Even if you are just looking at the returns of the top bucket—and nothing else—that is the mean-return (annualized) for that bucket. A statistic. And I have clearly made excess returns by looking at that.

No question.

The buckets are like a histogram: one of the first things you learn about in a statistics course. No one said they do not teach the most important things first.

Adding variance to make a Sharpe Ratio, arguably for some, helps to determine whether excess returns, out-of-sample, were due to luck or the use of good factors in the port.

There might be other ways of getting the same information but I have made money looking at that too. No question.

David, I thought you liked Alpha. Is that a statistic? You just like to look at it or do you think it is helpful in some way?

You have to mean that methods more complex than the ones you use do not add much. Is that what you mean?

While not as good as what P123 does I could show you some evidence that a linear regression of some data works.

Certainly, I can show you that a regression of daily volume (or a function of volume) relates to slippage which is an important part of whether we make excess returns or not. Probably anything else I might show you could be debated and I might want an NDA anyway.

But getting a regression on slippage is a pretty clear example where statistics can work (that is not a big secret). And it is asking a lot to expect someone to just eyeball thousands of trades and predict the slippage on the next trade.

There may be other ways to look at slippage but this too has helped me earn excess returns. No (rational) question.

I really think P123 is a bit of a cult—and I can show you some statistics to prove that;-)

It is good that Marco can do most of the work for us. Getting enough high quality data and grouping it in a way that variance is less of an issue. And displaying the pretty buckets (where variance is less of a factor) so we THINK we are not using statistics.

As far as more complex statistical ideas: he is making the data stationary for us by “differencing” it. As well as reducing the importance of variance.

Thank you Marco!

Thank you for making it possible for us to make money (using statistics and machines) while we self-identify as Luddites.

Yeah. It’s a cult.

Marc is looking like one of the few who is not beyond redemption or who was never in the cult.

-Jim

Can you repeat that in a sworn-to and notarized affidavit so I can show it to my family, all of who have long considered me beyond redemption notwithstanding such things as my outrage over last night’s episode of the Bachelorette.