Index | Recent Threads | Who's Online | Search

Posts: 116    Pages: 12    Prev 3 4 5 6 7 8 9 10 11 12 Next
Last Post
This topic has been viewed 4832 times and has 115 replies
wwasilev
Re: Factor and data series importing

1) setting a ticker to 1 adds it to the stock factor on the specified date
2) setting a ticker to 0 removes it from the stock factor on the specified date
3) a ticker setting remain in force until over-ridden by a new value/date
4) when there's an update to a list, set new entries to 1, set removed entries to 0, leave unchanged entries alone
5) if you run OSX/Linux you can use comm and awk to make the process somewhat easier
6) plan on spending much, much more time on dealing with delisted/renamed stocks
7) hopefully, p123 will automatically handle delisted/renamed stocks in currently active stock factors

Jan 26, 2020 1:22:14 PM       
Edit 2 times, last edit by wwasilev at Jan 26, 2020 1:49:03 PM
nisser
Re: Factor and data series importing

Wouldn't it make sense(and be less onerous) that with any new update, previous tickers are automatically zero unless updated with a new "1"?

Jan 26, 2020 6:00:40 PM       
yuvaltaylor
Re: Factor and data series importing

Wouldn't it make sense(and be less onerous) that with any new update, previous tickers are automatically zero unless updated with a new "1"?


The idea behind importing factors was not to create lists of tickers but to assign values to tickers. So let's say you are able to get a source for executive compensation and you want to use that to buy and sell stocks. Companies normally report this annually. So your CSV file will look something like this: 01-05-2019,XYZ,3200000 / 02-04-2019,ABC,425000 / 02-17-2019,LMNOP,1200000 etc.

So if a company comes out with a new report listing their executive compensation, you want to quickly be able to add that to your custom factor, without having to repeat all the old ones. You definitely don't want a fallback that assigns all the old data zeros.

Now some users want to use factor importing in order to assign ones and zeros to companies depending on whether they are in a list of companies that someone else has created, which is great. But we already have custom lists for that. If you wanted to create 20 lists based on the holdings of a certain ETF at different rebalancing intervals, it might be easier than using custom factors. Let's say you're trying to replicate the holdings of an imaginary ETF named WOWY. You create 40 custom lists of its holdings on 40 different dates. You then call up those lists in your backtest by using the DaysSince command. If DaysSince(date) is greater than X, use this list; if it's greater than Y, use this list instead. You can nest Eval commands in a custom formula to do this. I think this would probably be easier than using the factor import, and you could have done this for years before now. The power of custom factor importing is much greater than just 1s and 0s.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Jan 26, 2020 9:33:29 PM       
Edit 1 times, last edit by yuvaltaylor at Jan 26, 2020 9:34:52 PM
geov
Re: Factor and data series importing

Now some users want to use factor importing in order to assign ones and zeros to companies depending on whether they are in a list of companies that someone else has created, which is great. But we already have custom lists for that. If you wanted to create 20 lists based on the holdings of a certain ETF at different rebalancing intervals, it might be easier than using custom factors. Let's say you're trying to replicate the holdings of an imaginary ETF named WOWY. You create 40 custom lists of its holdings on 40 different dates. You then call up those lists in your backtest by using the DaysSince command. If DaysSince(date) is greater than X, use this list; if it's greater than Y, use this list instead. You can nest Eval commands in a custom formula to do this. I think this would probably be easier than using the factor import, and you could have done this for years before now. The power of custom factor importing is much greater than just 1s and 0s.

Yuval, what you are missing is that the ranking system is applied to the universe, for example "All Fundamentals". That is a lot different than applying it to only the stocks in your point in time lists.

Jan 26, 2020 9:56:33 PM       
wwasilev
Re: Factor and data series importing

Well you definitely want to do this in a universe. DaysSince can work there. I'm guessing that you would eventually run into a rule count limit.

Walter

Jan 26, 2020 11:00:53 PM       
nisser
Re: Factor and data series importing

Now some users want to use factor importing in order to assign ones and zeros to companies depending on whether they are in a list of companies that someone else has created, which is great. But we already have custom lists for that. If you wanted to create 20 lists based on the holdings of a certain ETF at different rebalancing intervals, it might be easier than using custom factors. Let's say you're trying to replicate the holdings of an imaginary ETF named WOWY. You create 40 custom lists of its holdings on 40 different dates. You then call up those lists in your backtest by using the DaysSince command. If DaysSince(date) is greater than X, use this list; if it's greater than Y, use this list instead. You can nest Eval commands in a custom formula to do this. I think this would probably be easier than using the factor import, and you could have done this for years before now. The power of custom factor importing is much greater than just 1s and 0s.

Yuval, what you are missing is that the ranking system is applied to the universe, for example "All Fundamentals". That is a lot different than applying it to only the stocks in your point in time lists.


Not just that, but also that handling the stock factors in my manner is exponentially easier than nesting eval commands and dealing with dozens and dozens of different lists...

I didn't even think of this feature and I find it great, but the way its set up now seems less useful than what I immediately imagined it to work like.

Jan 26, 2020 11:04:44 PM       
wwasilev
Re: Factor and data series importing

I'm not sure nested evals are needed. This seems to work in a universe. It has its own data management issues, but hey.

Attachment Screen Shot 2020-01-27 at 12.41.56 AM.png (95578 bytes) (Download count: 49)


Jan 26, 2020 11:45:23 PM       
Jrinne
Re: Factor and data series importing

I'm not sure nested evals are needed. This seems to work in a universe. It has its own data management issues, but hey.

Walter,

That is an awful lot of lines of code for a sim rebalanced weekly.

About 1,300 lines for a 20 year sim rebalanced weekly? Is that in the ballpark?

I do not know what Marko (Nisser) wants to use this for but it seems like there might be situations where one might have to go the stock factor route.

I get that it can all get done the way you and Georg have developed. But I haven’t used it with the outdated ticker problem being the worst problem for me.

Anyway I am not sure where I would disagree with anything Marko has said about ease of use.

Some sort of automatic zeroing would not be a bad thing.

Looks like we might have a handful of power users who use it mainly on ETFs now.

I guess I count myself among those who could spend a few weeks and maybe get something with only a few errors from incorrect zero dates —like holidays--that may not affect things too much.

Not a feature request. I do not know if P123 wants to move to other things or not and I am probably indifferent.

But I do not see where Marko is wrong.

I do think that features seem to be getting harder and harder to use .Getting harder and longer to program into the platform and are useful to fewer and fewer people when the features arrive. That is not going to get better as we move past the low-hanging-fruit. Maybe we are beyond the low-hanging-fruit already.

Not that Python would be a piece of cake for everyone. But as easy as this and it opens a huge number of features all at once.

Not a feature request. Just an observation.

And honestly I wouldn’t even know what to recommend if I were making a feature suggestion. But Marko has a point, I think.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Jan 27, 2020 5:14:11 AM       
Edit 13 times, last edit by Jrinne at Jan 27, 2020 7:45:30 AM
geov
Re: Factor and data series importing

Not just that, but also that handling the stock factors in my manner is exponentially easier than nesting eval commands and dealing with dozens and dozens of different lists...

I have done this with lists already for USMV, and it is very cumbersome. This is for a sim starting on 1/1/2013 and ending on 6/30/2019. The starting date is important because Portbars is being used, it cannot be changed, otherwise you get nonsense.

First you need to set up your historic holdings so your sim knows which list to use and when, this is done with Portbars.

Step 1: you need a custom formula as a buy rule in the sim such as: $USMV_starts_01_01_2013
Eval(PortBars<4,InList("USMV1212"),Eval(PortBars<61,InList("USMV0313"),Eval(PortBars<125,InList("USMV0613"),Eval(PortBars<193,InList("USMV0913"),Eval(PortBars<255,InList("USMV1213"),Eval(PortBars<318,InList("USMV0314"),Eval(PortBars<380,InList("USMV0614"),$USMV_Jan13P1)))))))

Then you need another formula for $USMV_Jan13P1
Eval(PortBars<444,InList("USMV0614"),Eval(PortBars<506,InList("USMV0914"),Eval(PortBars<568,InList("USMV1214"),Eval(PortBars<631,InList("USMV0315"),Eval(PortBars<695,InList("USMV0615"),Eval(PortBars<757,InList("USMV0915"),$USMV_Jan13P2))))))

And another formula $USMV_Jan13P2
Eval(PortBars<819,InList("USMV1215"),Eval(PortBars<883,InList("USMV0316"),Eval(PortBars<946,InList("USMV0616"),Eval(PortBars<1009,InList("USMV0916"),Eval(PortBars<1071,InList("USMV1216"),Eval(PortBars<1134,InList("USMV0317"),Eval(PortBars<1197,InList("USMV0617"),$USMV_Jan13P3)))))))

You are not done yet. And another formula $USMV_Jan13P3
Eval(PortBars<1260,InList("USMV0917"),Eval(PortBars<1321,InList("USMV1217"),Eval(PortBars<1385,InList("USMV0318"),Eval(PortBars<1448,InList("USMV0618"),Eval(PortBars<1514,InList("USMV0918"),Eval(PortBars<1572,InList("USMV1218"),InList("USMV0319")))))))

So now you have the holdings data from 1/1/2013 to 3/31/2019 in your buy rules which is good until 6/30/2019.

Step 2: You have to set up the universe USMV-ALL-since-Jan-13, which is InList("USMV1212to0319") and are all the stocks that USMV had held over the period 1/1/2013 to 3/31/2019. This does not overcome the problem of ranking the stocks that you have in the current list at a specific time, but it is good enough, better than ranking All Fundamentals.

AAP AAPL .... ZBH ZTS

So if you think this is easier to do than uploading a stock factor $$USMV into the universe, and no buy rules in your sim, then please go ahead with the above.

Jan 27, 2020 7:58:17 AM       
Edit 5 times, last edit by geov at Jan 27, 2020 8:25:17 AM
wwasilev
Re: Factor and data series importing

That is an awful lot of lines of code for a sim rebalanced weekly.

Jim, Agreed. Especially problematic if it runs up against a rule count limit. I plan to stick to stock factors for this use case. However, as Yuval pointed out, stock factors are meant to capture continuous data. Say you had AAII's Sentiment Survey data, you wouldn't want to clear older entries when there's an update.

Walter

EDIT: I was going to suggest adding commands to stock factors, but that has issues, too. Maybe.

EDIT 2: I think my use case example is wrong. Ignore that bit.

Jan 27, 2020 7:59:14 AM       
Edit 2 times, last edit by wwasilev at Jan 27, 2020 8:37:23 AM
Posts: 116    Pages: 12    Prev 3 4 5 6 7 8 9 10 11 12 Next
 Last Post