Index | Recent Threads | Who's Online | Search

Posts: 82    Pages: 9    Prev 1 2 3 4 5 6 7 8 9 Next
Last Post
This topic has been viewed 4528 times and has 81 replies
Quantonomics
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

The restrictions make it useless. I pulled one factor (say market cap) for 10 years and then it said I was done for my monthly usage.

Other than that making it useless, everything else was great.


I think this comment sums up my experience as well. I won't be using it because the restrictions are too harsh ( on a magnitude of at least x10 to x100) compared to the website.

Aug 6, 2020 11:43:27 AM       
Edit 1 times, last edit by Quantonomics at Aug 6, 2020 12:23:16 PM
philjoe
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

The restrictions make it useless. I pulled one factor (say market cap) for 10 years and then it said I was done for my monthly usage.

Other than that making it useless, everything else was great.


Yuval,

As it is, some of the web scrapers may begin to use the Data Miner App a little. It will remain a niche market BUT THERE IS A HUGE POTENTIAL HERE.

I am not as sophisticated as Philip as far as programming. But once the data has been manipulated (sliced, sorted, concatenated etc) using Python, I think I can hold my own as far as building a neural net, performing a ridge regression, boosting, using a random forest, etc.

So, I am trying to say that I defer to Philip and others as to whether the downloads are adequate and can be manipulated into a usable form.

But, unless Philip posts again and says that he loves the formats available, I think there is a lot of room for improvement.

Most important, a label is required. That would be returns over the rebalance period. If one is rebalancing weekly that would be the returns for the next week.

Ideally the output would have a usable index (e.g., a hierarchical index consisting of date and ticker).

But whatever index you use, I believe the best format for a download is an m x n matrix (or array) with column index, label (returns), and factors (P123 factors and functions).

Factors are just the factors (or functions) from P123 in a column with the ticker and date in a row (indexed).

This format would allow you to do bootstrapping, build a neural net, perform ridge regression, do a random forest, boosting etc, etc, etc,….WITH NO FURTHER MANIPULATION.

Anyway, as I said, if Philip or anyone else has a format that they prefer based on their programming skills then I think I can learn to manipulate their format as long as it has a label and a usable index.

But you cannot do anything without being able to manipulate the label into a usable format (generally an array, matrix or DataFrame). Even things considered unsupervised learning like principle component analysis and k-means clustering need the returns to construct the correlations (correlation matrix).

Thank you for your question.

Best,

Jim


If I remember correctly it outputs a CSV file, which Python can handle well.

Aug 7, 2020 11:33:05 AM       
Jrinne
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)



If I remember correctly it outputs a CSV file, which Python can handle well.

Philip,

That’s right. I have no trouble uploading (or downloading) a csv file in Python. Or in R for that matter.

But I still need the matrix (or array) I described above--at some point in the process. Otherwise I have no practical use for the Data Miner App.

I am willing to do some manipulation of the data. But as far as I can tell, something like this is pretty standard for most Python libraries.

I am really just describing a pandas DataFrame as you know. I would not mind getting where I want to be with a few lines of code (including the slices of data) as I have done here:

df=pd.read_csv('/Users/JamesRinne/opt/ReadFile/poptindex.csv')
f1=df[2000:]
f1_slice=pd.DataFrame(f1[['QQQ','XLE','XLU','XLK','XLB','XLP','XLY','XLI','XLV','XLF','GLD','LQD','TLT','SLYV','IWO','SPY','IWM’]]

Thanks.

Best,

Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Aug 7, 2020 1:57:52 PM       
Edit 43 times, last edit by Jrinne at Aug 7, 2020 4:34:37 PM
mm123
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Can someone explain to the non-programmers among us what this means in terms of sharing our intellectual property? And will it facilitate an easier way to share (beyond P123) our Strategies & books, or results from our Screens?

Aug 9, 2020 11:20:51 AM       
philjoe
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

I don't think it changes anything in terms of protecting your intellectual property (as in, you currently have no protection and will continue to have none). But it gives you a much easier way to export data from the website (to use in other programs like Excel or Python).

Aug 25, 2020 9:10:51 AM       
ivillalongabarreiro
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Are there news regarding the limits of the Data Miner?

thanks in advance.

Sep 7, 2020 7:21:00 AM       
yuvaltaylor
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Are there news regarding the limits of the Data Miner?

thanks in advance.


Changes are in the works. I should have more information for you later this month.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Sep 7, 2020 9:55:01 AM       
ivillalongabarreiro
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Hello,

Thanks @Yuval. I am looking forward to see how it advances.

Meanwhile, I would like to learn how to get the best of it. And I am facing a a roadblock.

It seems quite easy to change the "ranking system" or the rules within a rank. At the end, there are not too many combinations.

However, if I would like to see the evolution through different dates, the only solution is writting one by one all the nodes?

Let's put an example. I would like to see the result of 20 different rolling backtets every 4 weeks. Like the result that I can download with the Rolling Backtest in the screener, if I select Holding period & rebalance period of 4 weeks.

For that I have the feeling that I need to create a node for each different period of 4 weeks and rank, which will result in an enormous amount of work. While using the website I only need 20 backtests.

Am I correct? or Am I missing a point?

thank you very much in advance!

BR!

Sep 15, 2020 4:14:49 AM       
Jrinne
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

I may not be understanding Ignacio correctly.

Even so, for those familiar with Python, wouldn’t it be nice to have all of the data with a date column and be able to slice the data by date? Maybe be able to use the date column to sort by date as well as rank and returns to see how the ranking system is doing at finding the best stocks?

One download and all the rest being done with slicing (sorting, etc) within Python. A small script doing the rolling backtest or cross-validation or bootstrapping…or whatever. Much of this code already written in Scikit-Learn.

I may be missing something (probably am actually). More than likely, I am missing something and 2.0 will address any inefficiencies others are noticing—which would be good. But Ignacio’s post is not the first post like this.

More succinctly: I think P123 would benefit from attracting and marketing to the Kaggle crowd. https://www.kaggle.com

You know, the Kaggle crowd who competted to win $2,000,000 to better be able to predict the prices of homes for Zillow.

Of course, as we all know, it is much easier to predict the price of a home--using the limit data available on a home—than it is to predict a stock price using P123’s data. Of course, Kaggle has paid to predict future stock prices for hedge funds too. And we all think those kinds of predictions can be made or we would not be here at P123.

Hmmm, P123 probably could not market to them. But they do use Scikit-Learn, universally, if you did decide that there might be a market there.

-Jim

From time to time you will encounter Luddites, who are beyond redemption.
--de Prado, Marcos López on the topic of machine learning for financial applications

Sep 15, 2020 5:40:03 AM       
Edit 11 times, last edit by Jrinne at Sep 15, 2020 11:34:57 AM
yuvaltaylor
Re: NEW: Data Miner App & P123 API -- v1.0 (beta)

Hello,

Thanks @Yuval. I am looking forward to see how it advances.

Meanwhile, I would like to learn how to get the best of it. And I am facing a a roadblock.

It seems quite easy to change the "ranking system" or the rules within a rank. At the end, there are not too many combinations.

However, if I would like to see the evolution through different dates, the only solution is writting one by one all the nodes?

Let's put an example. I would like to see the result of 20 different rolling backtets every 4 weeks. Like the result that I can download with the Rolling Backtest in the screener, if I select Holding period & rebalance period of 4 weeks.

For that I have the feeling that I need to create a node for each different period of 4 weeks and rank, which will result in an enormous amount of work. While using the website I only need 20 backtests.

Am I correct? or Am I missing a point?

thank you very much in advance!

BR!

I think I understand your question. By "nodes" you mean "iterations," right? If so, then the answer is yes, you are right.

One thing that we need to do is to enable the DataMiner to give, as an output, a batch of .csv files that would be the same as what you can download after running a backtest on the screen. That would solve your problem, if I read you right. Personally, this would be of great use to me too.

But right now we are working on lifting some of the restrictions, pricing use cases with fewer restrictions, and creating a service that would help you use the DataMiner for practically anything you can think of. I'm sure that batches of .csv outputs will be an option sooner or later.

Yuval Taylor
Product Manager, Portfolio123
invest(igations)
Any opinions or recommendations in this message are not opinions or recommendations of Portfolio123 Securities LLC.

Sep 15, 2020 4:55:10 PM       
Edit 1 times, last edit by yuvaltaylor at Sep 15, 2020 4:56:04 PM
Posts: 82    Pages: 9    Prev 1 2 3 4 5 6 7 8 9 Next
 Last Post