We had to create a new constraint for ranking system regarding the number of nodes. Due to the massime memory requirements for large ranking systems they re now limited to a maximum of 250 node (including composite nodes). If you need large ranking systems to data-mine factors you will need to break up the system into smaller ones.
We apologize for this inconvenience. We’ll look at other possibilities to allow for large systems, but the math does’t lie:
A ranking system with 500 nodes requires over 70MB of memory for one week alone. It can also tie up an execution thread on a machine for minutes for ranking.
So, recently there has been some emphasis-upon and expansion-of the API because of interest in AI/Machine learning. Specifically, Boosting has been looked at and gained some acceptance because ranks alone can serve as inputs (or factors for the Silicon valley pot-heads as per Steve). Good.
But I think there is a lot of data mining going on that may not fit everyone’s definition of AI/Machine learning. This last post would support that I think. Double-good.
I think the API is not a machine learning/non-machine learning issue. Never has been. Extra-good that P123 has moved beyond that debate I think.
I don’t necessarily want to have more than 250 real nodes, although I might. I am building a library using the RS and I use composite node titles as notes and for organization (not the kind of grouping organization everyone else uses them for).
To your main point, as soon as I get completely up to speed in all things P123, I plan to jump head first into the API and get busy so full steam ahead on that front.
Cool!!! And I am sure you have a good use for the API and the data you are gathering now. Whether your method will be my focus for investing over the next few months (or not) is of no concern. I certainly hope it makes you a lot of money!!!
My point—if I have one—is that I am glad we are beyond the days that we had to pass every idea through the various and forever-changing preferences of all the P123 staff and members.
As a specific example, I am glad that Steve does not have to tell people whether he has plans to use XGBoost or TensorFlow and how he plans to validate any time-series or cross-sectional data. Or whether he just wants to do a rolling backtest for that matter. Not my concern unless he is interested in my opinion, I believe.
And I am glad that P123 clearly wants to help Steve and others with regard to some of the data in the API: Like with Z-Scores for inputs (or even discussing pseudo-standardizing the ranking data with the cumulative distribution function with Steve). That impressed me very much, frankly. P123 is becoming “big tent” in the sense that it is welcoming diverse methods and members inside.
Anyway, wishing you the best with your data gathering and investing,
“That impressed me very much, frankly. P123 is becoming “big tent” in the sense that it is welcoming diverse methods and members inside.”
I am thinking up new applications right now that may not even be ML/AI. For example, an “RS Writer”, something that takes the manual labor out of creating ranking systems. It could be XGBoost-related, but the idea would be to suck the data out of P123, do some processing by certain s/w algorithms and auto-generate an RS text file that one can input back into P123. It would make all the hours of painful RS development disappear like magic.