Successive runs of a sim give different results

Report a problem or error
1

It took me quite a while to make this reproducible-
The sim is: https://www.portfolio123.com/port_summary.jsp?portid=1620646
It uses a custom series (Hi/Lo Beta Ratio-3) passed through a custom formula ($BETAHI3).

First Step: Modify the custom series (in specific here, a parameter is changed from .075 to .75), and generate the chart.
Second Step: Run the sim.
Third Step: Run the sim again. (no changes to anything)

First run:

Quick Stats as of 8/14/2020
Total Return 366.28%
Benchmark Return 331.95%
Active Return 34.33%
Annualized Return 7.38%
Annual Turnover 1,002.25%
Max Drawdown -27.60%
Benchmark Max Drawdown -60.83%
Overall Winners (132/227) 58.15%
Sharpe Ratio 0.49
Correlation with S&P 500 Equal Weight ($SP500EQ) -0.15

Second Run:

Quick Stats as of 8/14/2020
Total Return 361.41%
Benchmark Return 331.95%
Active Return 29.45%
Annualized Return 7.33%
Annual Turnover 825.17%
Max Drawdown -27.84%
Benchmark Max Drawdown -60.83%
Overall Winners (109/188) 57.98%
Sharpe Ratio 0.46
Correlation with S&P 500 Equal Weight ($SP500EQ) -0.18

I left the sim as-is after a First Run, you should notice the same change above if you rerun it.

2

I’m looking into this issue.

3

It looks like there’s an issue with daily rebalance. I haven’t found the root cause of the varying output, but I’ll work on it more on Monday.

4

Thanks- If it’s any help, I’ve seen cases where the result varied as: A-B-A-B-A-B… or A-B-B-B… or A-B-C-C-C…
And often times the sim backtest run seems to glitch at a random percent completion, and restart counting up from 0%.

5

The issue seemed to be triggered by a combination of daily rebalance and custom formulas. I have made a change to try to correct the behavior; let me know if you see further problems.
Regarding the restart you’ve observed: Are you making changes to a referenced aggregate series prior to running your simulation? If so, you’d see an apparent restart, but it would say “Processing Series” for the first part.

6

I’m getting slightly different results from the sim backtest, but successive runs give the same result now (tried 3 in a row), so success there.

As to your question about the restart, yes it is when I make a change to a referenced aggregate series before the sim run, but during the sim run I don’t see a message “processing series,” (That only happens on the series page when creating a graph.) I only see the sim run “back up” at some random point and then proceed to 100% finished.

7

sglinsky (steve?)

did you add a buy rule:
Close(0,GetSeries(“your series name”)) or true

it might help solve your remaining problem

jerome

8

Yes Jerome, I am doing that to trigger a series update-
Thanks-

Steve