Index | Recent Threads | Who's Online | Search

Posts: 82    Pages: 9    Prev 1 2 3 4 5 6 7 8 9 Next
Last Post
This topic has been viewed 4539 times and has 81 replies
dnevin123
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

marco, is there not a way to throttle based on a % of available computational bandwidth instead of a fixed number of requests? The fixed number of requests is somewhat arbitrary and not necessarily "fair" (i.e. some requests will require significantly more computational bandwidth than others).

For example, the task I was attempting to complete was to grab the rank for a small number of stocks (2 to 3 on average) for a number of different transaction dates. For a single simulation, this quickly runs into hundreds of requests, but I would imagine (I might be wrong) that this uses far fewer resources than a single request for a Rolling BackTest. If both the Rolling Backtest and calculating the Ranks of a couple 100 stocks on a single day are treated as "1 Request", then the usage metric being used seem a bit arbitrary. (I could be misunderstanding something here).

IMO, this is similar to the limit on the number of nodes in a Ranking System. Where I can't have more than 200 nodes (or whatever it is) in a single ranking system, but I can have 5 or so ranking systems in a simulation that each have 199 nodes.

Anyway just wanted to poke the bear a bit, and see if there might be an alternative way of managing things. I completely understand that there has to be some way of sharing resources between users. However, I would much rather have to wait a long time to get my request fulfilled, then run up against a hard limit and not be able to make my request at all until the next month.

Thanks,

Daniel

May 21, 2020 9:15:59 AM       
philjoe
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Was this ever changed? I just ran marketcap data for 10 years on a weekly bases and now I'm done for the month...

The pull only ran for about 45 seconds, and resulted in a 9MB csv file.

Jun 2, 2020 3:31:52 PM       
Edit 1 times, last edit by philjoe at Jun 2, 2020 3:36:37 PM
marco
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Sorry we did not get back to this. Several things kept us quite busy. We'll come up with some plans for API & Miner very soon.

Also I hope to finalize something with S&P tomorrow for those that want to continue on with an S&P data & license. Anyone interested in this , please let me know as it will give us some leverage. Thanks

Portfolio123 Staff.

Jun 2, 2020 7:59:13 PM       
Quantonomics
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Sorry we did not get back to this. Several things kept us quite busy. We'll come up with some plans for API & Miner very soon.

Also I hope to finalize something with S&P tomorrow for those that want to continue on with an S&P data & license. Anyone interested in this , please let me know as it will give us some leverage. Thanks


I would be interested in keeping S&P for another year of a few months. This will allow most issues to be ironed out and a smoother transition.

Jun 3, 2020 7:47:19 AM       
Edit 1 times, last edit by Quantonomics at Jun 3, 2020 8:51:14 AM
dnevin123
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Sorry we did not get back to this. Several things kept us quite busy. We'll come up with some plans for API & Miner very soon.

Also I hope to finalize something with S&P tomorrow for those that want to continue on with an S&P data & license. Anyone interested in this , please let me know as it will give us some leverage. Thanks


I concur, another few months with S&P will probably be necessary.

Jun 3, 2020 8:40:24 AM       
philjoe
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Can I ask about updates on this or is it too soon still?

Jul 8, 2020 3:16:09 PM       
philjoe
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Any updates on the restrictions on this feature?

Aug 5, 2020 7:31:05 AM       
yuvaltaylor
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

We'll be changing the restrictions and offering pricing quite soon. In the meantime, please share your experiences with the Data Miner. What do you like and what don't you like?

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Aug 5, 2020 9:52:47 AM       
philjoe
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

The restrictions make it useless. I pulled one factor (say market cap) for 10 years and then it said I was done for my monthly usage.

Other than that making it useless, everything else was great.

Aug 6, 2020 8:36:20 AM       
Jrinne
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

The restrictions make it useless. I pulled one factor (say market cap) for 10 years and then it said I was done for my monthly usage.

Other than that making it useless, everything else was great.


Yuval,

As it is, some of the web scrapers may begin to use the Data Miner App a little. It will remain a niche market BUT THERE IS A HUGE POTENTIAL HERE.

I am not as sophisticated as Philip as far as programming. But once the data has been manipulated (sliced, sorted, concatenated etc) using Python, I think I can hold my own as far as building a neural net, performing a ridge regression, boosting, using a random forest, etc.

So, I am trying to say that I defer to Philip and others as to whether the downloads are adequate and can be manipulated into a usable form.

But, unless Philip posts again and says that he loves the formats available, I think there is a lot of room for improvement.

Most important, a label is required. That would be returns over the rebalance period. If one is rebalancing weekly that would be the returns for the next week.

Ideally the output would have a usable index (e.g., a hierarchical index consisting of date and ticker).

But whatever index you use, I believe the best format for a download is an m x n matrix (or array) with column index, label (returns), and factors (P123 factors and functions).

Factors are just the factors (or functions) from P123 in a column with the ticker and date in a row (indexed).

This format would allow you to do bootstrapping, build a neural net, perform ridge regression, do a random forest, boosting etc, etc, etc,….WITH NO FURTHER MANIPULATION.

Anyway, as I said, if Philip or anyone else has a format that they prefer based on their programming skills then I think I can learn to manipulate their format as long as it has a label and a usable index.

But you cannot do anything without being able to manipulate the label into a usable format (generally an array, matrix or DataFrame). Even things considered unsupervised learning like principle component analysis and k-means clustering need the returns to construct the correlations (correlation matrix).

Thank you for your question.

Best,

Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Aug 6, 2020 9:05:43 AM       
Edit 11 times, last edit by Jrinne at Aug 6, 2020 12:46:55 PM
Posts: 82    Pages: 9    Prev 1 2 3 4 5 6 7 8 9 Next
 Last Post