New screener backtest issues

Research
1

We have launched a new algorithm a few weeks ago for the screener so that daily results are displayed in the graph even when non-daily rebalance frequency is selected. Thanks to martinfierro we identified a flaw and we hope it has been corrected. This flaw was not present in the previous version.

The new algorithm will allow us for future enhancements to screen backtesting since it has some inherent problems vs portfolio simulations (from the Simulation tab).

The main drawback to screen backtesting is that there are no sell rules. A sell is simply done if a stock fails the rules. This is far from ideal… in the real-world sell decisions are normally different from what got you in the stock in the first place.

However that’s how screen backtesting works. There are no plans to add separate sell rules since that’s what the simulation tool is for. However there are several enhancements that are possible within the screener backtest. The new algorithm will allow us to implement some of these.

Here’s the text from another thread to explain the major issue with screen backtesting. Please refer to the previous thread started by martinfierro to read more about the flaw that was fixed.

This re-write has revealed a major problem with screen backtests that was always present. The way it works is that at every rebalance period the weights of each position are set to equal. This is not real-world. For example if you buy the stocks of a screen you will most likely not set the weights back to equal the following week. You will incur in too many transaction costs to do that. The cost of bringing the positions to equal weight that were present from the previous rebalance, is assumed to be zero.

The problem is even greater with daily rebalances, since all weights are made equal each day.

It is however standard practice with screen backtests, and should be a good enough approximation. Even if the cost of bringing the positions to equal weight is assumed to be 0, the performance most likely reports an under-performance with real-life since it is a good rule of thumb to let your winners run. It is also important to note that rebalancing all positions to equal weight is actually possible nowadays with brokers like foliofn with very little cost.

Having said this, we want to be better than that. What is needed is a way for you to specify what to do with stocks that greatly over/under perform between rebalances: do you want to maintain the weight? Should some profit-taking take place? etc.

Maybe simply adding a flag to not rebalance to equal weights and adding profittaking rules might be enough.

The new algorithm will allow us this future capability once we’ve defined exactly what the options are.

We look forward to your suggestions.

2

bump

3

Here’s a thought, the screen backtest should have these two options:

  • screen freq (daily, weekly, etc)
  • rebal freq (weekly, etc)

At every screen-freq interval the screen is run and stocks that do not pass are sold, new ones are bought. The ones that remain are not re-sized to be equal weighed to the new positions

At every rebalance-freqy all positions are brought to equal weight. There could be a threshold to leave a position alone if it’s, for ex, within 10% of the equal weight.

Seems intuitive, but there are caveats. For example what if during a screen, new positions are found, but none are sold? Where does the money come from to buy the new positions?

Or what if only one is sold and there are 5 new ones? How much to buy of the new ones?

These problems are attenuated if you select the screen backtest to run with only, for ex, the top 10 stocks. BUt there can certainly be situations where the screen doen not even return 10 stocks.

Other problems include:

  • what if the screen returns at times 20 stocks at times 1. Should all the money be used to buy 1 stock?

All these problems can be solved with adding many options/parameters to tell the screen backtester what to do in each situation. What Would be better is a simple approach with as little options as possible

4

I think the solutions is to think about it in real-world situation.

If you have 10 positions, and you are told there are 5 more that fit your criteria, but all 10 existing positions still fit as well, what would you do? Probably nothing, leave everything as is

If you have 10 positions and you are told that 5 more fit your criteria, and only one of the existing 10 does not, what would you do? Most likely sell the one that dropped off, and pick the best one out of the 5 that you are told to buy.

If you have 10 positions, and are told that 5 do not fit your criteria anymore, and there are no new ones, what would you do? Probably sell the 5 and leave the cash on the sidelines

I think this algorithm combined with a screen-freq and a rebal-freq should satisfy most people.

Naturally one more option is needed which is what max percentage of the total are you willing to invest in one stock? This will take care or the situation where not enough stocks pass the screen.

The complexity of the screener increases considerably, but it becomes usable in the real world. An option to download all transactions should be avail to see exactly what the backtester is doing.

What do you think?

5

I’m curious what people do with the screener. There seems to be an effort to turn the screener into the simulator and I’m not entirely sure why.

The screener conveniently allows me to see and compare things the simulator doesn’t, allowing one a quick feel and added detail in some cases but when it comes to a thorough backtest, that’s what the simulator is for.

Considering my use of the screener, screen frequency and rebalance frequency options are sufficient. Max percentage weightings don’t strike me as necessary for my own use. What I would like to see however are statistics like maximum adverse excursion (MAE) and maximum favorable excursion (MFE) per stock and portfolio for the rebalance period not just the the Min. and Max. end % as well as winning % for the rebalance period.

6

Sterling, over 80% of the 7M+ requests to the sim servers we’ve had since the start of p123 were screens and screen backtests. It’s what most people are used to.

I’ll create a feature request, including your suggestion, to gauge the level of interest for the list on enhancements I mentioned.

We’re in the middle of a big project at the moment (more details soon) and once completed we’ll re-prioritize again all the outstanding requests. Screening does seem to come up a lot in the threads.

7

Marco:

I like all the options you mention. I would especially like to see the maximum% for a stock.

Another reason why people use the screener is to test out shorting ideas which the simulator does not do.

Regards,
Brian

8

All,

 I think far too much effort is being spent on the screener.  What I use the screener for is to screen for stocks that I might buy/sell during the next market session.  For backtesting I use the simulator.  Any backtest performable on the screener should be performable in the simulator. So the screener backtest function is superfluous.

 What I really think the P123 staff should be spending their time on is implementing stop & limit orders, money management, and equity curve capabilities in the simulator.  These would be far more useful.

 --Bill
9

Bill

I disagree. I am a very advanced trader and for equities I only use the screener, and the backtest within the screener (my other systems are for options, futures and currencies). For equities my fundamental/technical screen in P123 has performed better than hundreds of pure technical scans I have programmed in the past.