| Index | Recent Threads | Who's Online | User List | Search |
|
|
 |
Portfolio123 » List all forums » Forum: Announcements » Thread: IMPORTANT: users running scripts are filling up the database |
|
Total posts in this thread: 17
|
[Request a Feature] [Add To My Favorites] [Watch this Thread] |
| Author |
|
|
marco
|
We've encountered major problems overnight. The database server ran out of disk space. The culprit was simply too many simulations are being ran and saved by automatic data-mining scripts. In a matter of a couple of weeks rogue scripts managed to fill up the database with data that is probably never looked at again. My guess is the scripts run simulations, download the performance and/or statistics and simply leave the simulation behind. In one single user account I found over 50,000 simulations each containing thousands of transactions and related data. We can no longer allow this unrestricted access to unlimited disk space. This also affect normal operation since database indexes grow as well slowing everything down for everyone. We are doing the following to correct this problem and avoid future problems: - Simulation transaction details will be deleted for any simulation older than 6 months. Detailed transaction data is most likely never looked at after the initial run. It's simply too much data. This does not in any way compromise your system. To regain the transaction details you will simply have to re-run the sim. If you must keep transaction details for ever you can download them into spreadsheets. - No more than 1,000 stored sims per account. Only about 20 users are currently exceeding this limit. Anybody with over 1,000 simulations is most certainly running scripts so we please ask you to do it judiciously. If your script runs simulations to download statistics it should re-use the same simulation over and over. The 1K is limit is now in effect. Let us know if you need help cleaning up your account as the interface might not be able to handle displaying thousands of sims so you can delete them. We are still going to have to do a lot of work to recover from this fill-up. It's much easier to grow a database than shrink it. New hard drives are also being ordered and we're going to do db maintenance throughout the week to try to bring disk usage levels back in check. ---------------------------------------- Portfolio123 Staff |
||
|
|
marcc
Member
|
Thank you Marco this service is too good to let indiscriminate users clog it up...I am sure i speak for all P123 users when I say thank you for continuing to upgrade the functionality. Marc Chaikin |
||
|
|
jerrodmason
Advanced Member UNITED STATES Joined: Jan 14, 2005 Posts: 599 Status: Offline |
FWIW, Marco, you may recall that I have for years been trumpeting the idea that you should be charging by the demand on your resources, not access to data. In other words, someone who runs a few sims a month, not saving the details, is limited to 5 years of data, but is by far a better customer than someone who pays more but runs thousands of sims, whether they get saved or not. With these modifications to TOD, you are only addressing part of the problem. This broken-record-dead-horse-beating member once again urges you to rethink the notion of limiting data access and in some sense charging for level of usage. Ultimately it is your call, of course. Just my two cents. ---------------------------------------- The smart money was on Goliath. "He's not the Messiah. He's a very naughty boy." |
||
|
|
strader1
Advanced Member
|
Marco: I take strong exception to your post. According to your "total" statistic I have 1723 Sims. I have NEVER run a script. I have, however, worked diligently for years to develop Port's that do well going forward. Moreover, I need these Sims for further development. Finally, to the best of my knowledge, I have complied fully with the terms of usage: so, I resent the implication that I am doing something "rogue." Respectfully, Bill |
||
|
|
Sterling
Advanced Member
|
I would like to raise a concern with this statement: "Simulation transaction details will be deleted for any simulation older than 6 months." Is that necessary and is this policy going to be implemented immediately? I have ended up keeping older simulations from a couple years back instead of replacing them with newer ones. I keep them precisely because I cannot run them again without losing information. They serve as a base comparison. Please reconsider this policy. Better is to set a safe maximum and then subject anything over that to deletion after a set timeframe. |
||
|
|
danparquette
Advanced Member
|
I have about 1400 simulations. And I have made an effort in the past to clean up stuff I dont need - otherwise I would have many thousands more. The 1000 sim cap is too small. I used to run a lot of automated scripts, but I always reused the same sim - the name was hardcoded in my script. Everytime a sim ran the script would store the results in Excel and then rerun it. The 1400 sims I have were created manually. I thought there was something in place that limited the number of sims a user could run per day (or hour?). Can you put something in place that limits the number of sims a user can save each day? As far as the history goes, deleting the history of the old sims is okay with me. The history is really the issue, right? I assume that is what is eating up the disk space. How about a limit on the # of sims that have the history? Automatically delete the history on sims once a sim is 6 months old (or a year old?). I'm fine with only being allowed to have 1000 sims with full history (or even 500) and unlimited sims without the transaction history. ---------------------------------------- Dan |
||
|
|
Stittsville123
Advanced Member
|
Marco - is there any way that the system could be designed such that, beyond a certain (reasonable) limit, the user's PC is used for storage of sim results? Steve |
||
|
|
o806
Advanced Member
|
Marco: It sounds like most users would not be affected by your proposed changes, however several users would find these limiting. These users appear to be doing their work with the detail and care that one would expect in a business setting. Is that not what the "quant" membership is for? What about applying your proposed new limits to everyone below the "quant" membership level? As for the "quant" level, it could have significantly higher limits, perhaps double or triple the number of total sims. As for maintaining histories for sim, I expect it is true that most users likely never look at the trade details after the sim is a or two month old. However, there are a few who clearly want to be able to go back and have a detailed look at the trades in a 2 or 3 year old sim. Why not give users the option to keep sim trade histories for more than 6 months for a small extra fee -- say perhaps $1 per sim per month ($12/year). The monetary incentive would motivate users to only save the histories for important sims. That $12/year should easily cover the extra hard disk space needed. Of course, the user could avoid the $12/year cost by downloading the trade history for a sim. Portfolio123 is a great product. I applaud your desire to keep it operating in a speedy manner. Regards, Brian |
||
|
|
marco
|
We will carefully study all options. I can't comment on the right solution now since we've been working around the clock trying to free up space in the database. It's a slow process since we can't shut the machine down, and the process is harder the less free space there is. Things are much better now. At the moment we had to immediately impose the 1000 limit to avoid further problems and begin clearing up some space by deleting the transaction details. We're trying to focus on the outliers and script sims but we can't guarantee some of the details of the "human" sims won't get deleted. That should generate enough space for things to go back to normal. The 1000 limit will remain until we upgrade the machine with double the space and we figure out what to do about policy. Needless to say a limit will exist, we just don't know what it will be just yet and how it will affect each membership level. The 1000 limit is necessary because it's the only thing we can do to stop scripts. We obviously underestimated the size effect of sim data and of scripts. We'll have checks in place to monitor size more closely to get a fair warning. As I said all of this happened in a month. Thank you for your comments. ---------------------------------------- Portfolio123 Staff |
||
|
|
Tomyani
Advanced Member
|
Marco, I would like to delete my entire 'unclassified' simulation folder (2300 plus simulations). Can I do this in one step? Thanks. Tom |
||
|
|
[Show Thread Printable Version] |
