Unless I am missing something it is just ridiculously difficult to get basic information about the positions bought and sold by a sim.
Even something as basic as the Sector of the positions when they were purchased. Is there a workaround, because the best I can do is use the screener for each of tickers on the date they were purchased? This is extremely time consuming obviously, unless you write a script to do it but then you run up against the requests per hour limit almost immediately.
It have a hard time believing that FactSet or Compustat are restricting our ability to know the Sector or MktCap of the transactions in our sims. This just seems like an extremely poor UI issue.
Obviously if there is some easy fix I will like an ass, but at the moment I an extremely frustrated.
Open up a screen. In the rules, put in Portfolio([the number of the simulation, the 7 digits at the right of the URL]) = 1. Put in any date in the “As of” box. Run the screen and choose any P123 report or create your own custom report. You’ll be able to see all the information you need about all the holdings of the simulation on that date.
Thanks. That is somewhat better… but still requires me to run the screen for each date in the simulation which is still extremely frustrating to get the data I need.
All I am trying to do is create a Sector exposure over time chart for the Sim in question. It shouldn’t be this hard.
A simple fix for this is something I have requested before (featureReqID=1474). Just let the Portfolio() function be used in the Custom Series Tool.
It would then be easy to do calculations like number of CONSDISC or number of Large Cap stocks over time. It should also enable the creation of a pseudo equity curve series for a particular strategy for use in hedging, etc.
You can add a rule like sector=Tech to the screen that Yuval suggested where the first rule is the Portfolio id. Then run a rolling backtest in the screen. The Pos field in the results will show you the number of stocks the portfolio had in that sector on each date. Export the results to Excel. Then do the same thing for next sector. You would still have to do this for each sector, but it wont take long since there are not many sectors.
I think people who would like to use large amounts of data have 2 separate problems. 1) Getting the large amount of data downloaded and 2) getting it into a usable format.
This thread seem to be more about getting the data into a usable format. Once that is done the download volume is not that large it seems.
I have been asking for more access to just the ranks and returns. It was just confirmed that this is not a provider issue (in another post). And the download volume is not huge for my request. Or, actually, it may be huge. I am not sure how to judge that.
Ultimately, I think the formatting issue is the biggest issue that Data Miner does not really solve. Plus, people are already getting the data with “scripts” or web scraping anyway (perhaps). Maybe I will just learn to do that if P123 has other important priorities.
I am probably just not getting it yet. But it seems that if P123 wanted to facilitate data downloads it could do so now.
P123 could be charging me for data on a per data download fee now.
This is just for ideas. Maybe I do not need this anymore and I am not much of a programmer so I have much to learn.
More importantly, I am getting better at using statistics to find weights and reducing overfitting in the usual ranking systems at P123. This is a compliment of the present ranking system and not meant in a negative way. Still, I might try a neural net on some of this if P123 made it a little easier. I do think neural nets offer some incremental improvement based on what I have seen.
I believe some form of DataMiner would be useful for my particular problem and a variety of other problems. With respect to this particular issue, it should be simple to allow for a single query of the database that specifies a table of dates and tickers and requests factors (Sector, MktCap, Revenue, etc) for those particular dates and tickers. The API that was released should allow for this, but it restricts a query to using a single date (as far as I can tell). This creates a number of difficulties and in some situations significantly increases the number of queries required to get a very small amount data. Given the current restrictions on the API it made it completely unusable for use cases like I have described in this thread.
Dropbox - samples - Simplify your life has nine different operations and anyone who knows Python could program additional ones. I think you can use the DataMiner to do what you need it to do, though I’m no expert since I don’t know Python myself. The idea behind it was to allow developers to create their own applications. The possibilities are enormous, and ideally the DataMiner will replace web-scraping. We’re in the process of changing the restrictions and pricing, but I’m eager to hear your thoughts.
I think I will leave most of the comments to people more informed than me—like Daniel above.
I can say none of the downloads from the samples you linked to were usable for me.
I am on my Mac now and it has been a while but one sample had the date in the column. It had tickers and ranks but no weekly returns (if I remember correctly). If I downloaded returns separately I would not have been able to transpose the dates into rows and match up the dates, tickers and the returns with the ranks.
But if I can get there (somehow) I can produce any model one could want in time it takes the computer to run (Random Forest, Neural net etc).
I am hopeful that I can get a sample that is close to what I need and modify it or even do some of the modifications in a spreadsheet if necessary.
Daniel’s comment make me wonder, however. I hope I am not taking what he said out of context. But even Daniel said that in its present form it is “unusable.” Even for him.
To be clear. I am happy with the port I am now using at P123. It would be a great bonus if the factors in this port could be put into a neural net (or a boosting model). A nice bonus but it would not be the end of the world to just stick with the port. Or find another way to build a neural net.
Just want to add that your only market should NOT be Python users, I think. Although marketing is not my area of expertise.
I suspect many Econ majors wanting to use Econometrics in P123 have used only SPSS to date. And SPSS does do some things that Python does not do as well. Or at least some things that are truly hard to do in Python are in a dropdown menu in SPSS.
My present Port was developed using a free SPSS clone (JASP).
If I am having trouble getting things into an array with column headings of date, returns, rank of factor 1, rank of factor 2, rank of factor 3…rank of factor n, then I suspect a few of the Econ majors will too.
People using SPSS will need their data to look like I describe above in a csv file they can upload into SPSS.
Ultimately I will figure it out. I could use a sample giving a download that looks like the above but you probably do not have tailor it to me. Again, I will figure it out. I do think there is a market for people who are even less sophisticated that I am (if that is possible).
Thanks for your response. Our current plan is as follows. We will change the restrictions and add pricing to the DataMiner. We will add a few capabilities to the API which aren’t there yet so that it’ll be able to do almost anything one can do on the website. If the particular application you want to use isn’t available we will create it for you for a fee. It’ll take a little time before this is all rolled out, but sometime in the next few months, if you want to use the DataMiner and the particular task you want to use it for hasn’t been developed yet, we will work with you to make it happen.
For example I take it that you would like as inputs some screening or universe rules, a ranking system, and a date range, and as outputs you would want the tickers, their ranks, and their prices for each Monday in the date range. I believe this is possible using the API end points that we have currently in place. We would just need to write the Python code for it.
I agree you can do what I am asking with the DataMiner, but the problem I ran into was with the restrictions. I ran out of my query quota before I was able to get the data needed to analyze one simulation. Looking at the Data function again just now and I think it might be possible to get around the restrictions, but I am not sure without testing it. Looking forward to seeing what the revisions are regarding the restrictions and pricing as I expect I will get a great deal of use out of the DataMiner once it is opened up a bit.
