IMHO this is good. More could be written on the different subject of avoiding overfitting. But it would be just that: a different subject.
In fact, I think this is REALLY good. I appreciate that Yuval has shared it.
There are algorithms that could reduce the computer time, I think. Algorithms akin to “gradient descent.” But P123 would have to help a little.
So, just for example, I might start with ten factors.
Ten factors each with weight of 0.1.
Starting with the first factor I would increase it to 0.15 and normalize the other factors as P123 already has the ability to do. I would then see if the rank performance improves by whatever metric I chose.
If it improves I would increase the weight to 0.2 and again normalize the other weights. I would continue to increase the weights until the performance of the rank performance test declined and move back to the best weight.
Then I would move on to the next factor.
If the very first increase from 0.1 to 0.15 caused a decline in the performance of the rank performance I would move in the other direction from 0.1 to 0.075, say… Continuing to decrease the weight until the performance stopped improving.
I would also try a weight of 0 and take 0 as the weight if there was only a small difference between the maximum performance and the performance with 0 weight. This is called regularization. At what difference in performance one would move to 0 weight would be set by the user.
When I went through all of the factors I would be done. Or perhaps I would start with the weights of the first iteration and move through each factor again—increasing and decreasing each weight in a similar manner. Stopping when the computer no longer makes any changes.
This saves computer time because the program stops at a maximum “gradient” without trying every weight for each factor. Somewhat more accurately the program actually looks for the point where the gradient is flat at a maximum or minimum. And for whatever reason they like to look for minimums—reversing the sign if necessary. So the DESCENT in “gradient decent” is essentially descending until the minimum is found. But you can change the signs on what I have said and it will be a very basic “gradient descent.” We can debate whether this is concave and can differentiated another day (it might not be the case). In the end I am happy to “borrow” from this technique to find a maximum without duplicating it.
Ultimately, the exact details for an algorithm used in this post are not important. Programers are used to creating loops that decreases computer time. P123 could also borrow from Markov-Chain Monte Carlo and STAN methods, for example. Or use the “metropolis” algorithm. The exact details would be determined by people who—unlike me—can actually program.
This type of thing is being done now at P123. It is actually automated but with the use of crude spreadsheets.
I understand that P123 may save computer time by NOT automating this and letting only the few struggle in inefficient ways with their spreadsheets: as is the situation now. Albeit with inefficient methods that eat up computer cycles.
Of course, any savings in computer time by requiring human manipulation of spreadsheets (to slow things down and reduce the computer inputs) does increase the human workload: something that Yuval notes in the above quote. P123 will have to decide what is best with the state (speed and cost) of computers today.
I will not call this machine learning but we may be able to borrow from some of their techniques–even if it is only in our spreadsheets for now. I do not care what we call it as long as we learn from any/all good technique(s).
And just a question. If the ranks could be re-optimized on a regular basis—to take into account recent changes in the market—would our models perform better? If automated, it could even be done with designer models. Ultimately, P123 (or whatever comes after) will do this.
Computer time really is an issue I suspect. And I could understand that when P123 starts doing this they might start with a few designer models. Maybe use it as a source of revenue by charging a fair price for the computer time to those who think it might be useful.
-Jim