| Index | Recent Threads | Who's Online | User List | Search |
|
|
 |
Portfolio123 » List all forums » Forum: Simulations and Portfolios » Thread: Out-of-sample question |
|
Total posts in this thread: 11
|
[Request a Feature]
[Post new Thread] [Add To My Favorites] [Watch this Thread] |
| Author |
|
|
garryp
Advanced Member UNITED STATES Joined: Feb 15, 2008 Posts: 104 Status: Offline |
I tend to develop and test a sim/port over the complete range of my available 5 year range of data. But I am slowly coming to realize that I may be curve-fitting by doing this, and I have no remaining out-of-sample data to help uncover if I am. Is it better to develop using up/dn market sequence(s) in the past, then verify by test over more recent out-of-sample data periods? This alone might justify the cost of gold membership to get the additional 2 years of data. --Garry P. ---------------------------------------- -- Garry P. |
||
|
|
DennyHalwes
Advanced Member UNITED STATES Joined: Apr 28, 2004 Posts: 1532 Status: Offline |
Gary, See my reply to your other question here. One of the advantages of the other 2 years of data is that it was a bear market. The last 5 years has been a good Bull market with a few corrections. Using all the data enables you to test over a bear market and a bull market. Denny  ---------------------------------------- "The significant problems we face cannot be solved at the same level of thinking that we were at when we created them". Albert Einstein |
||
|
|
dwpeters
Advanced Member
|
Hi, This is an important question and one I strongly suggest you answer before putting real money behind a port. There are a number of discussions along these lines. Here is one. I particularly suggest you checkout Olikea's post in that thread where he shows how some very strong sims fell apart out of sample. Although there is certainly value in testing systems on data from 7 years ago, my opinion is there is more value in out of sample testing on recent data. My perspective is as noted by Mark Twain: "History, rhymes; it does not repeat." The markets changed significantly over the last year. Small Cap Value outperformed from 2001 into 2007 and is now clearly underperforming, at least vs. Large Cap Growth. So what worked best in 2001-2002 (small cap value) is very unlikely to work best now. I have found that good ranking systems tend to be fairly robust and that sims with very many trading rules are not - I did a number of tests comparing minimal buy/sell rules to my favorite buy/sell rules - minimal rules clearly did better out of sample. I have a combination of ports based on some public ranking systems and a couple private ones, all with minimal buy and sell rules. I am tracking the performance through 20 stock ports, trading much smaller versions, and allocating more to the methods that are outperforming in the current environment and less to those that are not. I won't be setting any records for performance on this site (way too much good competition) but I've found the hard way that higher performance often comes with higher volatility - even if this was not seen in backtesting. Don |
||
|
|
danparquette
Advanced Member
|
I was trying to figure out a buy rule which could be used to create 2 universes out of the existing universe of stocks with each universe having approximately the number of stocks. Then you could optimize the sims with the buy rule in place which would limit the universe to 50% of the total available universe. Once you have optimized it, you could switch the buy rule so that the sim uses the other 50% of the total universe - that would give you your out of sample test. But the catch is that the factor used to split the universe has to be something that should have no effect on returns. For example, you could not use a rule like Price > 30 to define one universe and Price < 30 for the other since that would affect returns. I was going to do something simple like CompanyName < "m", but there is no factor for CompanyName. I also thought that maybe a buy rule could be if price was even then its in set 1 and those where price was odd would be the out of sample group - but there is no remainder function in the math functions. Anybody have an idea how to do this? ---------------------------------------- Dan |
||
|
|
Stittsville123
Advanced Member
|
Wasn't there a thread on using the Random function for this sort of thing? Steve |
||
|
|
marco
|
Dan, I like this idea. We need two new functions: StockId Trunc(number) StockId returns our internal integer id for a company Trunc() returns the integer portion of a number. You could then write something like Trunc(StockId/2) = StockId/2 This will return all stocks with even id Trunc(StockId/2) != StockId/2 This will return all stocks with odd id Should be pretty close to 50/50 ---------------------------------------- Portfolio123 Staff |
||
|
|
danparquette
Advanced Member
|
Marco, That sounds like a good solution and it would give us 2 universes. How about a function like Right(text, Num_chars) to retrieve the right most digit? The database probably has this function available already. This would make it more flexible. For example: Buy rule for Universe 1: Right(StockId,1)=0 OR Right(StockId,1)=2 OR Right(StockId,1)=4 OR Right(StockId,1)=6 OR Right(StockId,1)=8 Universe 2: Right(StockId,1)=1 OR Right(StockId,1)=3 OR Right(StockId,1)=5 OR Right(StockId,1)=7 OR Right(StockId,1)=9 Then we could vary the contents of the 2 universes. Thanks. ---------------------------------------- Dan |
||
|
|
marco
|
We'll add this as well Mod(val,modulo): modulus operation that returns the remainder of val/modulo (parameters are converted to integers) To define three random universes you could do this: Between( Mod (StockID,20), 0,6) Between( Mod (StockID,20),7 ,13) Between( Mod (StockID,20),14 ,19) ---------------------------------------- Portfolio123 Staff |
||
|
|
marco
|
That example was overly complex. This way you'll be able to define three random universes: Mod (StockID,3) = 0 Mod (StockID,3) = 1 Mod (StockID,3) = 2 ---------------------------------------- Portfolio123 Staff |
||
|
|
crakes
Advanced Member
|
Wow. Great idea Dan, and very elegant solution Marco. I can't wait to hear how this new mode of in-sample testing proves out! |
||
|
|
[Show Thread Printable Version] [Post new Thread] |
Re: Out-of-sample question
