Is the data used by P123 reasonably accurate?

Over the past two weeks I’ve been running manual backtests to try to verify some OTC strats that posed 3,000% annualized returns according to backtests on this website - I had issues verifying much of the information used in the backtest data of my tests,many companies simply wouldn’t have data publicly available on their sites from a while ago. Has anyone actually figured out if the information provided by P123’s database is accurate?

Another example happened today, where I ran into a Simulated stock strategy that turned my net worth of 5,000$ into $569,780,349.11 over the past 5 years which is 950% annualized. Id love if this were accurate but I suspect it’s highly unlikely and am having a rough time verifying the information used and figuring out what my actual return would be if I used the strat.

TL:DR I suspect there’s a significant amount of misinformation used by P123, do you know if that’s the case?

Another fun message that I’m getting on the OTC market
“Total market value of simulation around 05/21/2019 exceeds $1 Trillion.” The 15 positions in my portfolio probably have a combined market cap of 20,000,000$

Do you have liquidity and stock price minimums on your backtests? If you are including something that trades an average of 5 shares for .001 or some variation then the backtest will be meaningless. You have to use common sense.

Yep i do on the backtests - I’m only buying positions that trade at about 500$ volume per day. Is it reasonable to expect a 30% monthly return on positions that are that small though? Backtests suggest you can do 30%~ monthly up to about 100,000$, of which you can still easily do 50% per year up to about 1,000,000$ (about 100 available positions on average at 5,000$ traded per day and a laxing of the backtests requirements) according to p123’s data.

Many of the graphs are characterized by a massive single day increase of portfolio value which is primarily what makes me unsure.

Here’s an example of what I’m talking about - see the large increase? I dont know enough about backtests to actually verify whether or not that’s legitimate but the minimum average volume is set to 500$ per day over the past 200 days - AvgDailyTot(200)>(500)

https://imgur.com/a/ZF0QcQB

Even with the massive 1000% results removed from the tests, you get 50% compounded monthly on 2 of the past 5 years and 400% montly for another
https://imgur.com/a/orvxmtK

the 1.5k% return is setting minimum price to 0.01

Depending on the size of trades you do, I would highly recommend increasing $500 per volume to at least 10x that, if not more. Think about the math behind it. I’m a pretty small trader, average position size ~$50k and I generally leg into trades. I wouldn’t set the daily trade total for less than $100k at the very minimum. Thinking you can buy 10% of the daily trade total and not considerably altering the price isn’t feasible.

So here are the results at 100k$ volume per day- possible 40,000$ portfolio size given about 8k per position?
https://imgur.com/a/ASTj9xd
still insane returns though, but thanks for your advice

I personally don’t bother with anything less than $3 price averaged over the last year, and minimum $1,000,000 Average Daily Volume (averaged over 60 days). You need liquid stocks to get good results. Anything else and you can get great simulations but is probably not practical in real life.

Jjames, Don’t know liquidity or slippage of the strat, but if I do my math right, at 447% annual returns 40k turns into over 1mil in 2 yrs, and 195mil in 5yrs. The strat is also trading in and out of 2 to 6 stocks for most of the backtest. I guess it doesn’t seem realistic to move that much money very quickly unless megacap - but just challenge the assumptions to see what might be possible. I don’t work illiquid companies much except in very small $ amounts because they’re difficult for me to trade without moving price - but there are people here who do trade them.

I’ve written about this elsewhere. You will get inaccurate results unless you put in minimums for mktcap, price, AND daily volume.

A simple solution: use the PRussell 3000 universe, which is designed to exclude untradeable stocks.

So is this the way to go at P123? Chaim and Steve reflect what I have thought in the past. But I am also willing to adapt and try to use whatever works. Are they right?

Chaim’s median Sharpe Ratio for his designer models is 0.63 which is excellent. So maybe.

Steve’s median Sharpe Ratio is 0.66. I am pretty sure there is some survivorship bias but I am not aware of Steve removing any models recently. I do not keep close track of Steve’s models however. I have not kept track of Chaim’s models at all by the way. Again maybe.

But there is a very savvy designer that has an EXCELLENT understanding of both the factors available at P123 and platform itself. He uses some pretty advanced quantitative techniques too, in my estimation. I mean this sincerely. Maybe I do not have enough experience to comment on factor usage in which case I will just say his knowledge of factors (and the use of the platform) is far superior to mine.

His median Sharpe Ratio is 0.25. He has only removed one of his models so there is very little survivorship bias.

But his only successful models are micro-caps and small-caps with relatively high turnover (relative to crazy returns 25 stocks ultra-low turnover high liquidity, say).

His successful models (relative to the benchmark) have a Liquidity (lowest 20%) of $159,451 for his micro-cap model and $320,951 for his small-cap model and they may include OTC stocks.

This argues that perhaps Steve and Chaim are exceptions to the rule that it is hard to do well with larger-cap models. The designer models in general would support this argument.

The designer models do not suggest micro-caps are generally an easy solution either, BTW. Designer models focused on micro-caps do not do better as a group.

Maybe Steve and Chaim are exceptions to the rule because they are exceptional. That would make sense. Yuval obviously has some exceptional skills too: some of which I have delineated above. But one has to wonder what us regular folks should do.

Just observations—all completely anecdotal other than the aggregated experience with Designer Models. I guess I have 3 thoughts.

  1. this is NOT a settled issue, and

  2. I hope P123 continues to be a platform that seeks to provide solutions for people who want to remain in the higher-liquidity stocks.

  3. The API provides a solution where we do not have to debate features based on investing styles in the forum as much. People with a wide diversity of trading styles (including liquidity preferences) can develop their own features with the API and a little manipulation of the data—whether in a spreadsheet, a programming language or a third party application. Or they can use the DDM and discretion without have to ask for a lot of new features I would think.

I honestly do not have an opinion as to what is best at this point. I have considered moving to (or including more) micro-caps if that is best. Using bootstrapping as has been recommend (perhaps in a Python program) or numerous iterations of spreadsheet if necessary. Or perhaps–working with Steve some on this so far—expanding my quantitative techniques. Or become more informed about the field of finance—like Chaim (probably Yuval and Steve are pretty well informed in this area too). I DO NOT INTEND TO BE CRITICAL OF ANY LEGAL METHOD THAT MAKES MONEY. If you think I have been I apologize (or you may be mistaken).

In summary, this is not a settled issue and I think it is crazy to ignore anything that has been shown to work. Maybe we do not have to debate this so much with the new API. Especially since there is no clearly-superior method for the average retail investor at P123 so far.

Jim - I don’t know where you came up with “median sharpe ratio of 0.66”. In any case, Designer Models make the assumption that a model, once designed, is good for life. The markets change constantly and a model that is good for one type of market may suck for another.

As for trading micro-caps and nano-caps, you can have some success but in most cases liquidity will impact your ability to trade the system as is and you won’t likely achieve the paper-traded results. Another factor little mentioned is that on this platform, stocks that disappear are assigned the last known price, not $0, which is what you will get in many cases. So if you see a micro-cap system holding a lot of stocks that trades based on an extreme pullback, you may get great paper results but may be completely unrealistic in real life.

Take care

Steve,

I think I agree with everything you have said.

Let me just focus a little on my agreement with the above quote. There are 2 ways to deal with this. One is to diversify in an intelligent way. The other is to use some market timing while still remaining adequately diversified.

But each strategy has to have positive expected value to be rationally included in a portfolio. Advanced hedging strategies (or remarkably good and not overfit timing strategies) excluded.

This is my focus now. That does not prevent me from diversifying with P123 models. And It does not stop me from looking for models that can be classified as “all-weather strategies” within P123.

I think a lot of P123 strategies could (should) be candidate’s for a diversified strategy, btw. So I apologize if I have emphasized the success (or failures) of any particular type of strategy in the recent market regime. If I did, there is a good chance I will be wrong over the next 3 years.

Jim

There is actually a third approach that I’m sure you would agree with. And that is an adaptive model, or at least a model based on paramaters that adapt over time, as you might get with an AI-based approach.

"But each strategy has to have positive expected value to be rationally included in a portfolio. "

Thank you for giving me the opportunity to shamelessly promote my Designer Model Inspector Sector’s Cloud Computing. https://www.portfolio123.com/app/r2g/summary?id=1439560

It doesn’t matter whether people subscribe to my model or seek their own strategy, not specifically targeting this area is unwise, or at least has been for the last year. In five years from now, this model may be retired but it has done its job. I had another model, Aerospace and Defense, which was doing great until Boeing’s Max737 issue followed by the pandemic (flights cancelled). Does that mean it was a poor model and should not have been offered? Life happens and market change.

Another thing is that I am often told that there is too much volatility / risk in the Cloud Computing model, why don’t you offer a 25 stock version? The answer is simple. 5 stocks or 25 stocks doesn’t change the risk that much. The fact that it is 5 stocks means that you can diversify into other markets and other model designers. But people don’t get it…


Here’s the crux of it: if you’re including stocks that are very difficult to trade and have extremely low prices, the results will often be quite crazy.

I include a lot of stocks in my backtests because I regularly trade stocks with low volume. I own a number of stocks with volume of less than $10,000 per day. But when you see huge jumps in your charts, zero in on them and figure out what you’re buying and at what price, and see if that’s realistic.

As for practical advice: price should be at least $0.50, and create sensible minimums for volume and bid-ask spread.

Yuval, what have you found about using the bid/ask spread?

Whenever I get extraordinary results, I always add something like this liquidity check:

LoopMin(“Vol(CTR) * Close(CTR)”, 50) > 2000000 // Liquidity check

…which means it had to trade no less that $2M of stock in any of the last 50 days.

I have it set up as a formula – $LIQUID50 – so it’s easy to add.

Try 100*loopmedian(“spread(ctr)/close(ctr)”,35). It’s a terrific indicator of how hard the stock is to trade–i.e. how high your transaction costs will be. In my own experience, that number correlates a lot better with transaction costs than does median daily dollar volume.

But be warned: there are a lot of stocks for which we don’t have spread data. So if a stock has a spread of 0, that doesn’t mean it’s easy to trade! To protect myself against this, I actually use this formula: 100*eval(loopmedian(“spread(ctr)”,35)=0,na,loopmedian(“spread(ctr)/close(ctr)”,35))

I want to disagree here concerning a special siutuation.

Important: exclude OTC stocks. But the universe without OTC stocks gives a lot of opportunities in unliquid stocks and the data is o.k… The smaller
the stock, the higher the inefficiencies you can exploit.

If, and only if, your port is really small, it is possible to beat transaction costs calculated by P123.
I ran modells on 0% transaction costs on very small stocks (non OTC) and I was able to replicate this into my trading for years (not any more bc port is too big know for my main strat).

Even now I always try to beat transaction costs calculated by p123. If you do not beat them try to improve your trading (e.g. sacale in, scale out).

Also you might trade a strat that accumulates stocks over severy days or weeks and you get out in an event which then provides
a lot of liquidity to get out (in the new book of Schwager unknown market wizzards there is a trader who does exactly that).

So it might be fine to create sims with no or very low transaction costs, then test if you are able to replicate this in your trading.

Best Regards
Andreas

[quote]

Yuval, are you using this in a rank or in a screen? If the later, what range are you setting it to?