Index | Recent Threads | Who's Online | Search

Posts: 87    Pages: 9    Prev 1 2 3 4 5 6 7 8 9 Next
Last Post
New Thread
This topic has been viewed 3831 times and has 86 replies
Jrinne
Re: To Quant Or Not To Quant, That Is The Question

I know I cannot prove this, but whenever somebody came up
with a complicated model to manage a complex world, I usually saw it fail or underperform (not only in trading, but in business too).

Andreas

Andres,

I could not agree more. I highlighted what I agree with most.

Notice one really cool thing the AI specialist Marco hired did: He ruled out a poor method before putting any money in it. Without having to paper trade for X number of years.

Now that was worth something. I think that if he were really good he could have found a good model. Indeed, probably should have known to start with a better model. But he did accomplish something pretty important!

I guess some at P123 believe in that cross-validation, hold-out sample statistical stuff more that I thought. They don't just believe in it, they paid for it. Very smart, IMHO.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 6, 2019 4:57:09 PM       
Edit 9 times, last edit by Jrinne at Dec 6, 2019 5:22:12 PM
Jrinne
Re: To Quant Or Not To Quant, That Is The Question

Marco,

I want to thank you again for sharing this information from the AI specialist.

One thing immediately stands out about this and I have been trying to think about the best way to say this.

I wonder, if he had studied it, whether he would have found that sims perform better than random?

I am not debating whether sims can work but rather I wonder what he would have found. And how he would have studied this. I do think having this comparison would be key to getting a perspective on the value of the study and for knowing the best course of action.

Maybe he did this—I don’t want to assume he didn’t. There has been recent talk in the Forum of the value of equal weight of factors. I wonder why he wouldn’t have, at least, run a sim with equal weight of the 50 factors. Considering he had to split up the samples this might not have met his criterion for statistical significance.

Yea, he should have done this, or something like it. I am not going to assume he didn’t of course.

Thank you again for your consideration and for sharing this. I just find this very informative and interesting. I actually think it is probably wise not to proceed with this now.

As always I like and appreciate what P123 is doing already

***Edit*** BTW, Did he use rmse or mae (or something else) as the metric for his Random Forest? I would really be interest (and important).

I do get that this is confidential and that the AI specialist probably did sign a NDA. But I would be interested in as much information as you feel comfortable sharing.

Thank you again.
-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 7, 2019 12:38:25 AM       
Edit 8 times, last edit by Jrinne at Dec 7, 2019 6:33:50 AM
Jrinne
Re: To Quant Or Not To Quant, That Is The Question

Jim -

While I agree with 99% of what Marc wrote here, I would certainly be open to some very PRACTICAL suggestions of concrete quant tools that could be integrated into our current functionality. These suggestions should include some kind of specification as to how the tool would be applied. "Integrate Python" doesn't tell me what's involved or how it would be used or how one could accomplish it. These suggestions should be comprehensible to wannabe quants like me. There are dozens of things I'm working on to improve Portfolio123 and its capabilities because I really care about its users. But they're all in the nature of "add a universe parameter to the Rating command" or "enable buy-driven simulations" or "add buy and watchlist buttons to the screener" or "enable hedging in ETF simulations." We're willing to entertain larger ideas too, like "in addition to designer models, let's have stock-picker models" or "allow users to import their own factors with an API." So if you want to make a concrete suggestion as to how P123 could implement random forests, I'd be happy to listen. But you would need to walk me through the process so that I understood exactly how it could be done. And only you could do that, since you're the only one here who has actually used random forests to optimize ranking systems.


Yuval,

Sorry I could not explain this to you.

If you are still interested you might consider asking Marco at this point. I believe he understands and had an internal study done. Perhaps he could share it with you.

Again, my apologies about my poor ability to explain.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 7, 2019 6:50:53 AM       
Edit 2 times, last edit by Jrinne at Dec 7, 2019 6:55:27 AM
Jrinne
Re: To Quant Or Not To Quant, That Is The Question

I agree 100%, at least in the short-run, that moving to Python is probably not best. Undeniably moving to FactSet is more important and should be focused upon. Anyway, this is a well considered decision on the part of the P123 staff and not my decision to make anyway.

I do find the study interesting and I hope it is a reasonable topic for discussion.

What the study Marco discusses showed is that one Machine Learning Method was not statistically significant. This finding is not too meaningful, IMHO.

I can also show this for a sim or two. In fact, the AI expert would—with absolute certainty—conclude that none of the Designer Models can be shown to be better than random, statistically.

You may disagree, but I can assure you that this would be the conclusion of an AI expert. This is because of the multiple comparison problem (there are a lot of models).

In fact--as a whole using all models--he would conclude that Designer Models are inferior to their benchmark over the last 2 years. Statistics like this need to be looked at in perspective, I think. I am sure I can find someone who will quickly agree with me on this point.

Any future evaluation of this (if there is a future evaluation) should try to compare sims to machine learning. Possibly using the same factors for each comparison. Preferably for more than one method.

That experiment will be repeated multiple times by multiple different people should P123 decide to integrate Python at some time in the future. Again, I am not necessary recommending this, and I am not all that sure I want everyone to have this ability anyway.

In fact, maybe I don’t. I may not need to advocate for another equivalent of Renaissance Technologies' RIEF Fund (that uses fundamentals) being made possible through P123.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 8, 2019 11:42:27 AM       
Edit 23 times, last edit by Jrinne at Dec 8, 2019 2:32:26 PM
Barn
Re: To Quant Or Not To Quant, That Is The Question

Jim I think you are going too far down the rabbit hole here and losing sight of the bigger picture. Having an edge in the market trades is likely not going to come from some form of random anything. Lets not lose sight of what the market is. It's a place to trade perceived value on pieces of companies that are doing business everyday. Business is something dynamic and therefore the price of the company is dynamic as well. P123 is no more than a tool at comparing pieces of information about these companies/shares and making a decision weather one share is a better deal vs another share. I don't expect it to be right every time, that would be naive. If I was lucky enough to find a correlation through some random variable of a billion permutations then what does that actually give me? If I break it down then I can think of this problem as finite, be it a HUGE number but non the less the problem has a finite solution. There IS some optimal combination of factors and weights that can be combined together to give the best backtest. However is that magical algo optimized .... you bet it is and the likely hood of it to perform as it did in the past is much less likely due to peak optimization at the present. It reminds me of "An infinite number of monkeys with an infinite number of type writers will have one monkey that produces the best book ever written". However once that monkey finished the book and you asked that same monkey to start typing again ... what are the odds that it will write another book just as good or better?

IMO a good trading system needs to encompass all the components properly to work in the future.

1) A universe of stocks that deserve to be compared against each other
- If I didn't have any tools then I wouldn't try to compare something like a mining company to a internet company. Unless I knew otherwise I wouldn't want them in the same universe.

2) A ranking system that takes advantage of comparing factors and formulas that may be relevant to the universe. I agree with Yuval that it's worth examining how well your ranking pulls down the bad stocks as well as how well it pumps up the top stocks. We should arm ourselves with filtering bad in every layer of the trading system that we can. Even at the expense of backtested alpha. Capital preservation matters or you can't play the game.

3) A trading system with buys that filter the statistical outliers missed in the earlier stages. And sells that put reward / risk ratio into a proper perspective with slippage and commissions in mind.

4) A weighting system because a system's picks will hold a historical bias through some factor or formula and it makes sense that if we trust a ranking system to make a bias then we should trust a weighting system to hold a bias as well.

The really hard part goes back to what this WHOLE process is really all about. We are trying to use history to give us insight into the future of a dynamic object. An object that can be influenced by currency changes around the world, Political laws, tariffs and taxes around the world, companies decisions, competitor inventions, lawsuits, natural disasters and the list goes on and on ....

The point I'm trying to make is that I feel you can't get lost in the statistics of the past when we live in a world where stock is priced into the future. Concentrate on building a system that makes sense in good times and in bad times and I think there would be a lot more success on P123.

Just my $0.02 ... Actually I need that back for my port.
Barn

Dec 10, 2019 9:27:29 AM       
Jrinne
Re: To Quant Or Not To Quant, That Is The Question


The point I'm trying to make is that I feel you can't get lost in the statistics......

Barn,

You should do what you feel comfortable doing or what you like to do.

Optimizing sims and ranks is a quantitive method whether one likes to admit it or not. Something you seem to admit to in your post. I actually like the method too.

If you think sims and rank performance are the best for every single situation then...Well, l I wonder if one can say that about anything. This person is the best person for every job. This hammer will be the best tool for everything I do. I do not even need a toolbox.

Defies common sense. I will not go through the math proving that other methods are better for nonlinear situations. Something de Prado stresses. It is also mentioned in the book about Renaissance Technologies that Cary originally posted about (The Man Who Solved the Market).

But this should be common sense too. And didn't Renaissance Technologies resolve any doubt on this topic with both its Medallion and RIEF funds?

Good luck. And do what makes you feel good.

As Marc says, it should be fun and whether you are losing money or not is not always the most important question:

and that it is not necessarily tied to “performance.” Back when I was at Value Line, I used to say that we were not in the financial services business but rather in the entertainment business. Stock picking is FUN and people love doing it,....

-Marc

The Designer Models are entertaining--at least--even if they seriously underperform their benchmarks. So no worries. But you would think--for the price I am paying --there would be some show girls like when I played blackjack;-)

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Dec 10, 2019 9:51:55 AM       
Edit 33 times, last edit by Jrinne at Dec 10, 2019 11:16:44 AM
SnortingElk
Re: To Quant Or Not To Quant, That Is The Question

I spent some time with QuantConnect which seems to be a competitor to Quantopian. As best I could tell, Quantopian took a hit when they stopped proving trade integration, whereas QuantConnect provides this. One of my motivations was the ability to integrate additional data, and QuatConnect has support for Quandl and for any other custom data.

My perception is that it would take a lot of time for P123 to compete in this space. QuantConnect basically has an open-source “engine” called Lean that can be installed locally, or that is run online with a web UI like P123’s online engine.

The Lean engine can consume either Microsoft C# Code or Pearl Script. Because you are writing code, especially if you run locally, you can pretty much write anything you want and integrate other libraries. So very extensible. QuantConnect has template classes with standard event handlers and a framework that includes prebuilt universes, a portfolio manager, etc.

I found the online service frustratingly slow for builds, reports, etc. so I downloaded the engine into Microsoft Visual Studio which is a much nicer editor and very fast compiler, especially if writing in C#. The problem with the local engine though is that unlike the online portal, you must have access to your own data. I connected up Tiingo locally, which was fine, but I think there are similar issues if you want to trade using the local version; possible but additional work since its “open source” and QuantConnect makes money only through the online version.

I like P123s approach that is “pseudo quant” without the need to write class files. I did think that QuantConnect had excellent interactive tutorials and a ton of samples; it appears they even have a library of re-implemented strategies they took from Quantopian.

I like Yuval’s new ranking content. Personally, I think P123 would be better off providing better custom data support, longer custom functions, perhaps dropbox or other integration for automating screens etc., and more sample content, updated documentation, an easier path from a ranking system to a sim with buy/sell rules, etc. It seems like there are a lot of easy wins, and that competing in the QuantConnect space through robust programming support would take significant work.

Dec 16, 2019 3:28:03 AM       
Edit 1 times, last edit by SnortingElk at Dec 16, 2019 3:29:44 AM
judgetrade
Re: To Quant Or Not To Quant, That Is The Question

@Jim --> thought about it over the weekend: R2G --> I took my flagship trading system and put it to a R2G. The question is, will I publish it? Probably not!
Reasonas: I can only charge 1000, that is to low for me since I do not know if the the sub will suck up all the liquidity and I would loose this liquidity in my own trading. On about 850k I can make north of 25% a year (yes with DDs, but I do not care about DDs!) with this trading system, why should I publish it? Look at QuantConnect, they charge 25k for a system a month! Then this would be interesting for me.
I doubt that there P123 users publish their actual best trading system, why should they?


Best Regards
Andras

Dec 16, 2019 3:57:31 AM       
judgetrade
Re: To Quant Or Not To Quant, That Is The Question

@SnortingElk wow! Your Posts are super!!! What is your backround?

QuantConnect:

I had a brief look at the platform, what I did not find was PIT Price and fundamental Data. --> Correction: Found it, will dig deeper, question is: how good is it?

"The majority of the fundamental data update occurs once per month." --> Here is a problem! P123 provides their updates
weekly!

So this stands:
The gold of P123 is the PIT Data (including fundamentals!) I will have a deeper look on it, though. A combination of P123 Data
and QuantConnect would be interesting. But this project would need funding north of a ton of cash (> 1 Million Dollars?).

Best Regards
Andreas

Dec 16, 2019 3:58:47 AM       
Edit 2 times, last edit by judgetrade at Dec 16, 2019 4:13:53 AM
judgetrade
Re: To Quant Or Not To Quant, That Is The Question

@Barn
Could not agree more on the monkey example!!!

Question: I have no idea what you mean with the following point:

"4) A weighting system because a system's picks will hold a historical bias through some factor or formula and it makes sense that if we trust a ranking system to make a bias then we should trust a weighting system to hold a bias as well."

Could you give an example, I do not use any weighting I think.

Point 1) is also very interesting!!!

"1) A universe of stocks that deserve to be compared against each other
- If I didn't have any tools then I wouldn't try to compare something like a mining company to a internet company. Unless I knew otherwise I wouldn't want them in the same universe."

Really tossing my brain on this. I do not do this so far, I only exclude financials and filter for small caps but this is it.
How do you do this, do you work with different sectors to shape a universe?

Best Regards

Andreas

Dec 16, 2019 4:02:20 AM       
Posts: 87    Pages: 9    Prev 1 2 3 4 5 6 7 8 9 Next
 Last Post