Somebody misses me? Aw shucks . . .
I’m surprised too. When did I ever become OK with that? I am open to machine learning and am personally still in the “learning” stage when it comes to this sort of thing. I’m always open to learning about something new, although I’m very comfortable ultimately turning thumbs down if what I learn is that it can’t really do what it’s supposed to do (as is the case with optimization, when I eventually learned how much a prisoner it is of its specifically defined sample).
Perhaps a better way to explain myself might be to temporarily drop financial talks and switch to another field undergoing a quant revolution; sports, where the word used is “analytics.” It’s become so prevalent, that baseball managers, once all powerful iconic figures, have seen their stature reduced to the point where they are little more than glorified functionaries stationed in the dugout to implement directives from the GM and the analytics department that feeds him information. But today, its no longer confined to baseball. It’s penetrated the NBA in a big way, and the NFL as well.
The NBA’s Houston Rockets are an interesting case study. Darrel Moray, their analytics-obsessed GM, has managed to build a team that wins a lot, a heck of a lot, and is a championship contender year in and year out. But lately, they’ve developed an alarming tendency to crap out very badly in the playoffs under scenarios when they entered as favorites, to if not win outright, at least fight to the bitter end.
One sports media personality who I like a lot (who often uses stock market language to express opinions including a weekly segment during football season called “Buy, Hold Sell”) Colin Cowherd, presented an interesting, and I think very persuasive take on the situation and it’s somewhat analogous to what I keep saying here about past and future.
Moray does, is, indeed, very successful in using analytics to solve the problem as he defines it – winning games. But he has not succeed in recognizing the difference between winning games and winning a championship. Moray’s modeling presumes that success in one will confer success in the other. But that is not so.
The 82 game regular season is a prolonged stretch in which the Rockets play everybody, good teams, middle of the road teams and bad teams. No single game is do or die (except maybe here and there towards the end as teams jockey for final playoff positioning). But players can pace themselves (and rest where needed within games and even by sitting some out). And no team really gets an opportunity to adapt in the moment to any other team because they don;t play the same team in consecutive games.
The playoffs are a completely different animal and all of Moray’s algorithms, that succeed so well over the 82, need to be trashed; either he needs a completely different model or would be better off using no model. In playoffs, you only play good teams, and each round, the opponent gets better and better. Rest and pacing (and even injury recovery) is a much more delicate balance becuase we’re only dealing with a tightly contained best of 7 series. (The defending champion Golden state Warriors got burned very badly by this issue and were dethroned.) Opponent can and do adapt because now, each team plays the ssame opponent until one team wins 4 games, which can take anywhere between 4-7 games.
Another big big big big difference that Moray completely missed is the climate of officiating, the tendency of refs to call fouls. Moray’s players, especially his stars, have exploited this so well, that hey even developed physical techniques, such as the land forward jump where they’ll go upp and instead of coming straight down, as normally expected, propel their feet forward under the feet of the opponent who lands on top of them and gets called for the foul. Getting the opponent called for a foul, putting their players in danger of fouling out (6 fouls and you’re out of the game) and their own players to the free throw line a lot is religion in Houston. It’s analogous to what Brad Pitt, playing Billy Beane in the movie “Moneyball” always said as the greatest attribute of a batter: “He gets on base.” Moray’s equivalent: “He gets to the free throw line.”
Moray completely miised the reality that in theplayoffs, all of which are nationally televised, the NBA does not want to destroy the visual pacing of the game by calling a lot of fouls and having too much of the game played from the free throw line.So the refs are much slower to call fouls and they often turn a blind eye toward contact that would have been whistled in the regular season. This, plus all the other differences, prove disastrous for Houston in the playoffs and this season was the worst, when the team was roundly ridiculed for whining about how officials weren’t calling fouls as often as they expected.
Is Moray a fool? Are analytics bogus? No on both counts. Analytics work and Moray’s analytics worked brilliantly , , , when he was correctly defining the problem. Nothing failed. It’s just that in the post season, he did not have an algorithm suitable for the task at hand. He missed out when it came to defining the problem. He’s a smart guy and I expect once he owns up to the nature of his challenge, he’ll address it and solve it . . . unless the teams’ new aggressive owner loses patience and fires him before he fully defines and solves his new task.
Now, back to our regularly scheduled p123.
My distaste for the way I see backtesting and sim used on p123 is precisely illustrated by the Darrel Moray conundrum. I see these as effective and often brilliant solutions – to the wrong problem. Investors can no more generalize from sample periods to live periods than Moray could from the 82-game drawn out foul-heavy regular season to the concentrated best-of-7 swallow-the-whistle playoff environment. This is exactly what I try to get at when I talk about how we can’t assume the past predicts the future.
Analytics in sports and analytics in finance are the same. Both are at their best when they are seen as taking advantage of modern technology and analytics techniques to help do human tasks more effectively and efficiently than human judgment alone could accomplish them. My DDM-based presentation is not a silver bullet or a formula. It’s a logical roadmap to help one define the problem to be solved on p123 or wherever via analytics.
If the problem is defined as showing strong equity curves in a sample period, then do not expect it to succeed in a live period unless one has the good luck to experience a live period that just so happens to resemble the sample period. And as it turns out, many who work this way on p123 have, indeed, benefitted from exactly that, just as the Houston Rockets would have benefitted had the ref been willing to call fouls in the playoffs the same way they did during the regular season, something that alone could have offset all their other playoff analytic deficiencies. And as it also turns out, the unusual persistence of historic trends has not favored my less momentum oriented approach, meaning my models have often not showed the live performance I’d have liked. But unlike many I’ve seen on p123, I at least know WHY my results have been what they were and, having a lot of experience in this business and recognizing the monstrous inflection point, or perhaps plateau, on which we’re sitting (the interest rate situation), I won’t alter models that are designed to be stable.
I suppose a reason why I’m not on the forum so often any more is to reduce the probability of a tantrum that is unsuitable to any forum posting. But you can be sure I see all and am at least mentally flaming the living daylights out of anyone who illustrates a point by showing sim results – or at least without accompanying such a presentation with a verbal explanation of why the poster believes the numbers/trend were what they were.
Backtesting is a great tool but it must be used properly. It’s not there for people to show fabulous equity curves. It is there to show how the kinds of stocks picked by your model perform under various conditions – to give your judgment an assist that was not technologically feasible 20 years ago.
I’ll close this with a link to a Seeking Alpha article I published nearly a year ago, in which I actually presented and discussed bad backtest results but explained why I was going ahead with the model anyway.
This is how I use testing, to help me see if I’m getting what I think I’m getting, not to allows me to show how great I am at producing vertical or near-vertical equity curves. (In fact, the more beautiful the illustrated results, the less likely I am to read the text of the post.)