Select Language

Uploads offline to prepare for Cata

Rollie

 
Uploads have been disabled while the update process completes existing uploads and we prepare the site for the switch to Cataclysm!

We hope to resume uploads within a day or so.
EpikPhailure says:
December 7,2010 at 1:2
Sweet! I was hoping Deathwing didn't break the site!
Rollie says:
December 8,2010 at 8:48
I am going to keep uploads offline for a couple of additional days as I work on the database. It is going to require some changes otherwise the deluge of incoming data will cause the db to fall farther and farther behind.
Rollie says:
December 8,2010 at 8:53
Matter of fact, I'm halting updates right now and working on some changes that will take some time to complete and will restart updates to see what the effect is afterwards.
EpikPhailure says:
December 8,2010 at 17:23
So Deathwing did break the site?
Rollie says:
December 8,2010 at 20:19
I think so!
mtgirl1979 says:
December 9,2010 at 8:57
Keep up the good work Rollie, I look forward to seeing the boom of players back for the expansion. 8)
Rollie says:
December 9,2010 at 9:5
Database cleanup continues to run. I won't be able to run it on the whole database as at the current rate it'd take about 15 days to process the whole thing. Letting the current batch of cleanup finish and going to test a file to see if it has helped *crosses fingers*
Korwynd says:
December 10,2010 at 13:15
Besides the database, I notice you didn't mention that the addon would have to be updated as well to include the new races. It'll be interesting to see the population growth of new characters and how many of those will belong to new races.

Rollie says:
December 10,2010 at 16:22
Updates have resumed, and will limp along through the weekend as I will be unavailable to continue tweaking the database until Monday. Perhaps by then it will have at least caught up to what all has been uploaded.

Uploads will remain offline through the weekend, sorry folks!
FuxieDK says:
December 10,2010 at 18:44
When enabling uploads, could you please (temporarily atleast) disable the 15 minutes waiting time between uploads?
Some of us have quite a few updates to upload and waiting 15+ minutes is pretty long...
bringoutyourdead says:
December 10,2010 at 21:7
I would suggest just the reverse. Cutting down the delay time between file upload will just flood the server with a lot more files. File open, close and delete operations are time/resource consuming.

I don't know why limit couldn't be set to 1 hour or more.. after all with the delay from the callback linkage in wholib.. most of my census runs are now over the 30 minute mark. It seems that every time I check Census+ is running again. (just kidding)[or not]
FuxieDK says:
December 10,2010 at 21:59
[quote:bcd5f69e1d="bringoutyourdead"]I don't know why limit couldn't be set to 1 hour or more..[/quote:bcd5f69e1d]Currently, I have 15 files I need to upload.. More will come over the weekend..
With 1 hour delay, it will take the better part of a day.
With 15 minutes delay, it can be done in 4 hours (if I do't produce more files)
With no delay, they can be uploaded in a few minute and we can "get on with our lives"...
FuxieDK says:
December 10,2010 at 22:0
[quote:3bb58a1714="bringoutyourdead"]I would suggest just the reverse. Cutting down the delay time between file upload will just flood the server with a lot more files.[/quote:3bb58a1714]1 second delay or 10 hours, will make the same amount of flooding.. We have files to upload, and upload we will..

Better get it over with ASAP...
Asagi says:
December 12,2010 at 9:24
:( Is there anything we can do to help?
shadowslip says:
December 12,2010 at 12:50
Hi.

I'm new to this. (not even made my first upload yet)

Just wondering why some people would have more than one file to upload.
I wouldn't want to miss uploading something.

thanks
:)
Pennet says:
December 12,2010 at 15:44
[quote:4b41531d31="shadowslip"]Just wondering why some people would have more than one file to upload.[/quote:4b41531d31]

Some people choose to upload manually, and for those people, there is a 10 MB file size limit. If you look at the Top Contributors page (Community menu), those people are scanning multiple factions across multiple realms, and can reach that file size quite easily. So, they have to subdivide their uploads to stay below the limit.
shadowslip says:
December 12,2010 at 18:9
aah. okay thanks
:)
1974ER says:
December 12,2010 at 18:34
Welcome from me as well!

Additionally, Rollie has temporarily disabled uploading, so those of running a lot of censuses are really piling up files. I am approaching 20 subdivisions already and since it's already Monday for me, I can safely say that that figure will break today.
EpikPhailure says:
December 13,2010 at 12:39
It has been a week since the uploads went offline. My suggestion is to delete all the old updates and start over.

Just my 2 cents.

Asagi says:
December 13,2010 at 15:31
Ah but they are only old after like 30 days. If I remember correctly.
1974ER says:
December 13,2010 at 17:44
I posted the reasons against deletions in the thread in General discussion, so not repeating myself here.
FuxieDK says:
December 13,2010 at 22:43
[quote:2003c7f394="EpikPhailure"]It has been a week since the uploads went offline. My suggestion is to delete all the old updates and start over.


Just my 2 cents.[/quote:2003c7f394]That would be the worst idea ever.. It would ruing updates at the most critial time in this sites history; At expansion release..

Without level-updates, we're left with nothing...

Rollie says:
December 14,2010 at 8:39
I am planning on working on it more throughout the day as work permits. I'm as concerned as anyone else with the general performance of things lately.
Jonathan91 says:
December 14,2010 at 14:43
We understand your time is limited Rollie. You take your time, do what needs to get done. We will be here for you when you?re ready. Day by day line by line. :)
Asagi says:
December 14,2010 at 20:10
Oh I totally understand. Hence if there's anything we can do to help just ask!

Mean while I keep running the Census.

Rollie says:
December 15,2010 at 8:55
Unfortunately I'm getting slammed with work... everyone trying to get stuff done before the holidays. I've been working 16 hour days for the past 5 days =/
PrimeKnight says:
December 15,2010 at 10:36
Did you need help in fixing your database performance? Please let us know, or perhaps just explain what the problem is. I'm sure that we have a lot of people who work in software development or database administration. I would be very happy to help you out if you needed it.
Asagi says:
December 15,2010 at 13:21
LOL, that's why I keep offering but I understand being slammed. :D
Rollie says:
December 16,2010 at 12:57
@Prime - I would be happy to have some folks offer suggestions to improve db performance.

I know of some things that can be done, but none of them are simple solutions that could be done on the same hardware without a lot of downtime.

MySQL 5.5 was just released yesterday which supposedly has much better support for partitioning and that would be very helpful. The problem is actually making that move.
gmmmpresser says:
December 16,2010 at 15:9
What I don't understand is why this has suddenly gone bad now?
We have had other upgrades & patches applied to the system without any major problems before, but since 4.0.3 (?) it appears as though the wheels have just fallen off.
1974ER says:
December 17,2010 at 2:43
Rollie can you give you much better answer than I can... My belief is that the database has simply grown to a size, where it is pressing the limits of the programs Rollie uses to manage it.

We have, according to the database stats base... just passed the 95 M characters marker... 95,014,599 character entries, says the page... before I went to bed it was still hovering around 94,981,xxx or so...
Balgair says:
December 17,2010 at 10:10
What ER said - this isn't the first time the site's had major database issues, due to the sheer size of it, but in the past, Rollie's usually managed to find a bandaid that's got it running again for a while. This time, it seems the scale is just too large for the current hardware limitations, hence it being rather harder to solve.
EpikPhailure says:
December 17,2010 at 11:16
[quote:5615e40dc9="1974ER"]...We have, according to the database stats base... just passed the 95 M characters marker... 95,014,599 character entries, says the page... before I went to bed it was still hovering around 94,981,xxx or so...[/quote:5615e40dc9]

Again, not to upset anyone, but really, there can not be 95 MIL toons on this game anymore, I feel that a good purge and restart would really help out this site, this addon, and more.
Anything over 30 days should be deleted.
In the interest of getting this addon back up and running, this database and site working again, sometimes sacrifices need to be made.

To those who feel that their hard work will be lost, I think you are loosing the big picture here.
This site and addon are to help people with current data, while there are some people I am sure who like old legacy data. Their numbers are much fewer than those who want the current stats.
I have heard this site discussed on my old realm, of Mal'Ganis, because of the free transfers in place now.
A lot of people want to know data before they take the plunge on a new site.
However at the current state, most people who don't know all the details except that the data on this site now is getting close to be over two weeks old, are loosing faith in it.

This again is all my two cents. Please don't attack me for my opinions.
Rollie says:
December 17,2010 at 11:39
There is a purge of sorts taking place to cull out old, unplayed toons.

And yes, this has been a problem before. There is a critical mass of characters that once reached cause the database to really chug to make the necessary updates.

With proper partitioning and the appropriate application logic to handle it, it can be improved considerably, that just requires time, which is something I don't usually have a lot of, but lately I've had zero of.
IroncladzOTE says:
December 18,2010 at 14:50
Still waiting!

So when is this update up?

Fett says:
December 18,2010 at 19:1
Well, let's see, on December 9 Rollie said that at the current rate it would take about 15 days. He hasn't had time to make the improvements he was hoping for, so my math says it should be complete about December 24. My guess is that updates will go back up after that. The moral of the story? Be patient.
EpikPhailure says:
December 19,2010 at 0:9
[quote:d72d0f10d3="Fett"]Well, let's see, on December 9 Rollie said that at the current rate it would take about 15 days. He hasn't had time to make the improvements he was hoping for, so my math says it should be complete about December 24. My guess is that updates will go back up after that. The moral of the story? Be patient.[/quote:d72d0f10d3]

Don't mess with the Fett!
Rollie says:
December 19,2010 at 0:25
Hehe, well, I plan on bringing them back online before that.
1974ER says:
December 19,2010 at 7:51
I look forward to that, as I am already "sitting" on almost 40 .luas... which means that it will most likely take at least a couple of days before I have even gotten rid of most current backlog I have waiting... :D
EpikPhailure says:
December 19,2010 at 9:47
[quote:fcef065f20="1974ER"]I look forward to that, as I am already "sitting" on almost 40 .luas... which means that it will most likely take at least a couple of days before I have even gotten rid of most current backlog I have waiting... :D[/quote:fcef065f20]

LOL, You are going to be backlogging your self, worse than the server was!
FuxieDK says:
December 19,2010 at 9:56
[quote:a6aed9db07="EpikPhailure"][quote:a6aed9db07="1974ER"]I look forward to that, as I am already "sitting" on almost 40 .luas... which means that it will most likely take at least a couple of days before I have even gotten rid of most current backlog I have waiting... :D[/quote:a6aed9db07]

LOL, You are going to be backlogging your self, worse than the server was![/quote:a6aed9db07]I have 26.. :D
EpikPhailure says:
December 19,2010 at 10:27
LOL, my realm's (Gul'dan) pop is so low, that I have been doing scans since Rollie came out with this new version, and my CensusPlus.lua is only 982 KB!
shadowslip says:
December 19,2010 at 17:55
i'm on sylvanas, a really busy realm and mine's only 123kb
1974ER says:
December 20,2010 at 2:33
EpikPhailure & Shadowslip!

If you are really running censuses on just one faction, please post the character amount visible on your censusplus screen... because the activity descriptions and file sizes seem mismatched, which makes me curious...

Thank you!
EpikPhailure says:
December 20,2010 at 8:10
[quote:f85b0f6e04="1974ER"]EpikPhailure & Shadowslip!

If you are really running censuses on just one faction, please post the character amount visible on your censusplus screen... because the activity descriptions and file sizes seem mismatched, which makes me curious...

Thank you![/quote:f85b0f6e04]

I am running on both factions.

However, I am not sure what you mean by "post the character amount visible on your censusplus screen"

Do you want me to take sceen shots?
cconover says:
December 20,2010 at 12:30
Only slightly off-topic, the new races are a great addition, but it's still only including up to level 80.
bringoutyourdead says:
December 20,2010 at 14:26
Go to General Discussion
Topic Test Cata Version
page 3
for Rollie's link to the latest beta from Akhtar which includes the wholib compatibility. Latest version includes all races tracking level 6-85 and stops a 5 minute timeout on zero return queries that previous beta had.
shadowslip says:
December 20,2010 at 16:27
I'm not sure what you need either but here's a screenshot

http://img263.imageshack.us/img263/2708/census.jpg
1974ER says:
December 20,2010 at 16:43
I was just looking after the value of "Total Characters:" seen on the screen... Thanks for the screen, Shadowslip. Also, could you give the currently filesize, as I think the screenshot is much newer than the filesize you posted earlier

EpikPhailure, screenshots are not necessary, just give me the values of "Total Characters*" on both sides and your current .lua file size, thank you. :)
shadowslip says:
December 20,2010 at 18:27
File size 123 KB still.
and to make sure I'm reading the right file, it lives in:
C:\World of Warcraft\Interface\AddOns\CensusPlus

and is called:
CensusPlus.lua
EpikPhailure says:
December 20,2010 at 18:28
H: 8419
A: 1308

CensusPlus.lua = 1,247 KB
EpikPhailure says:
December 20,2010 at 18:30
[quote:577d7230fe="shadowslip"]File size 123 KB still.
and to make sure I'm reading the right file, it lives in:
C:\World of Warcraft\Interface\AddOns\CensusPlus

and is called:
CensusPlus.lua[/quote:577d7230fe]

no... should be:

\World of Warcraft\WTF\Account\~ACCOUNTNAME~\SavedVariables\CensusPlus.lua
1974ER says:
December 20,2010 at 18:34
Shadowslip... that't the WRONG file.. the correct file is in your C:\World of Warcraft\WTF\Account\%account%\SavedVariables\CensusPlus.lua

, where %account% is your account name.

Also, EpikPhailure, those figures seem ok. :)
shadowslip says:
December 20,2010 at 18:43
aaah ok

:D

871KB
1974ER says:
December 21,2010 at 1:37
Shadowslip, ok, that seems like a reasonable size. :) I assume you have added a bit of data after that screenshot. :)

Thank you both. :)
Beastmistres says:
December 21,2010 at 4:48
I know im a lurker, but i've been running census since the cata release and have been waiting for uploads to come online. I know my server is utterly unbalanced at the moment and I just cannot wait to find out how bad >.<

I know patients is a virtue, but I cant waiiiit e.e
Beastmistres says:
December 21,2010 at 4:49
And sorry for the double post. I accidentally sent that, and guess we cant edit our posts, but is it possible for you to give us a guess for a timeline on when they will be back up? If not the shortest, but the longest delay until they are up and running again? D:
1974ER says:
December 21,2010 at 6:33
I know this isn't a big help.. but... if you have been running a fairly similar amount of censuses on both sides at same times, you may be able to draw some conclusions from your local data. But if you haven't that won't really help.

As for timeline, on page 2 of this thread, Rollie was implying that uploads could be resumed before the 24th... which I would appreciate as well, as I already have 40+ folders of unsent .luas. :D

Only Rollie can give you a more precise answers, sorry. :/
Beastmistres says:
December 21,2010 at 7:6
[quote:ed0f568e62="1974ER"]I know this isn't a big help.. but... if you have been running a fairly similar amount of censuses on both sides at same times, you may be able to draw some conclusions from your local data. But if you haven't that won't really help.

As for timeline, on page 2 of this thread, Rollie was implying that uploads could be resumed before the 24th... which I would appreciate as well, as I already have 40+ folders of unsent .luas. :D

Only Rollie can give you a more precise answers, sorry. :/[/quote:ed0f568e62]

No need to apologize, and i have been running census frequently on both sides. I do pretty much have a decent understanding with data i've gathered so far, but i know there are others who also run census on my server which is even better for overall data. ^^
1974ER says:
December 21,2010 at 8:8
More data is indeed good... which is exactly why I have dozens of .luas piling up on my hard drive. :D We are going to need a lot of it too... the most wanted page is epicly full of zeroes. :( Only the German Alliance and both sides of some Spanish realms still fight against the time and even they are succumbing... :/

On server submission page, the situation isn't (yet) quite as sad, but the flashing red figures are becoming more and more prevalent there... :/
Yingolfine says:
December 21,2010 at 9:49
Have just installed CensusPlus and wonders when you will be ready for Cata and open up for uploading of the data again?
FuxieDK says:
December 23,2010 at 1:38
[quote:9cdee35c77="Rollie"]Hehe, well, I plan on bringing them back online before that.[/quote:9cdee35c77]It's pretty much last chance, to deliver on that.. :P

We're hoarding LUAs like crazy people and are almost running out of disk space (exaggeration improves understanding)..

Tomorrow is christmas.. We need this present.. We need our upload-fix... ;-)
1974ER says:
December 23,2010 at 5:15
Fuxie, don't forget that Rollie is about 8 hours behind us (as I write these words, he most likely (and hopefully) sleeping soundly in a nice warm bed... + most Americans count towards 25th, not towards the 24th as many of us Europeans do. :D

Even so, being able to upload would be appreciated soon... I am approaching the 50 .luas mark and even if I assume an average file size of only 3 MB, that amouns to about 150 MB of data... and at approximately 3 files per hour, it will already literally take me days before the backlog is cleared on my end after the uploads have been re-enabled. :D
FuxieDK says:
December 23,2010 at 5:30
True, true...

But on the other hand, CET is the most populated timezone in the world (400+ million Europeans + a large chunk of Africa), and thus a defacto standard..

I'm well aware that Rollie probably sleeps at 7CET, but that doesn't mean he yen't wake up, before midnight.. ;-)
Asagi says:
December 23,2010 at 19:58
LOL, I'm kind of curious 1974'er why you have several LUA's is it because of size or something else?
FuxieDK says:
December 24,2010 at 4:20
[quote:8bd174376a="Asagi"]LOL, I'm kind of curious 1974'er why you have several LUA's is it because of size or something else?[/quote:8bd174376a]Not just him.. I have 31...

Because, if you e.g. level from 80 to 81 on day one (and CP sees it) and then you level to 82 on day 2 and CP sees it again, you loose the 81-update, as CP only stores the last update..

By saving the LUAs (until upload is enabled again) we can have all those updates, that we otherwise will miss out on..
1974ER says:
December 24,2010 at 9:14
Also, with the amount of censusing I do, the file quickly expands to a size, which the manual uploading can't handle... so I can't go over 10 MB.

Oh... and Merry Christmas everyone! :D
Fett says:
December 26,2010 at 18:16
Okay, I concede that my estimate of December 24 was inaccurate.

I still think our best strategy is to be patient!

FuxieDK says:
December 27,2010 at 2:42
[quote:67d8789507="FuxieDK"][quote:67d8789507="Rollie"]Hehe, well, I plan on bringing them back online before that.[/quote:67d8789507]It's pretty much last chance, to deliver on that.. :P

We're hoarding LUAs like crazy people and are almost running out of disk space (exaggeration improves understanding)..

Tomorrow is christmas.. We need this present.. We need our upload-fix... ;-)[/quote:67d8789507]Rollie, Christmas have come and gone.. We're now into the New Year season...

Please gief info!
1974ER says:
December 27,2010 at 5:48
Some sort of update on the situation would indeed be appreciated... I am now crawling towards .lua number 60... haven't reached that quite yet... but the SavedVariables folder still has already blown to a size exceeding 210 MB.
azmak says:
December 27,2010 at 9:22
I know you guys have known Rollie for a long while now but relax; it's the holidays! Let Rollie enjoy them. What? Did you expect him to be working on this on Christmas Eve and Christmas Day? He'll get to it when he gets around to it.

You sound like a bunch of kids going "Are we there yet? Are we there yet?"

:roll:
1974ER says:
December 28,2010 at 5:25
Haha, no, we don't expect Rollie to work during Christmas. As to work, there are people who do work during the holiday season... a very close relative of mine got off from work around 9 on the 24th, by 21 on the 25th, she was already back at her job.

Also, a friendly request for a status update hardly qualifies as "Waah!!! Are we there yet???? Waah!!!!". I am way too old for that, anyway... :D :D :D

On a further note, in several jobs I have held over the past 20 or so years, it has been necessary to send messages, which basically amounted to "Apologies. Delivery will be delayed. Will send further info ASAP." A short note in a similar style would hurt noone and would make the less patient people less fidgety, I should think. :)
FuxieDK says:
December 29,2010 at 1:31
[quote:ecc96809f5="azmak"]I know you guys have known Rollie for a long while now but relax; it's the holidays! Let Rollie enjoy them. What? Did you expect him to be working on this on Christmas Eve and Christmas Day? He'll get to it when he gets around to it.

You sound like a bunch of kids going "Are we there yet? Are we there yet?"

:roll:[/quote:ecc96809f5]Actually, it's normal curtisy to give info in due time.
When you set a time-frame, you never let it slide, WITHOUT INFORMING..
That means, if I tell you that I will have more info for you on friday, then I bloody hell have to inform you SOMETHING no later than friday..
That "something", on the other hand, can be a "this takes more time that expected, new info on tuesday" message, which would buy me 4 extra days (until a new status was required).

Uploads have been disabled for over 3 weeks now to empty the queue.. However, the queue was emptied over a week ago, without any info except "I hope to have it ready before xmas"-message.
Asking for info, isn't too much.. It's only fair..

Also, if Rollie shouldn't work on CP during the holidays, please tell me when he should... By his own words, he's had too much work the last year to do anything with CP..
Balgair says:
December 29,2010 at 6:50
By what he's been saying he's also had family issues though, which, whatever they are, may well take up a lot of time at a typically family-orientated holiday time like this.

Hassling him and acting all impatient isn't going to make things happen any faster frankly - if he's not able to get on the forum to give us an update, he won't even see these posts until he [i:d0783276cf]can[/i:d0783276cf] give us an update, and the amount of impatient posts by then isn't likely to make him feel too good about things. Just give him a break, it'll happen soon enough :)
Pennet says:
December 29,2010 at 8:46
I think people are losing sight of the fact that this is an addon, produced without profit, for a [i:51470a7f15]game.[/i:51470a7f15] Is it useful? Sure. Is it necessary? No. If Rollie said the hell with it, I'm not doing it anymore, would anyone's game experience change in any measurable way? No. Some of you might actually have some free time on your hands to read a book, play with your kids, get some sense of proportion.

The sense of entitlement that some of you are displaying is kind of obnoxious, and those entitled comments would not be much impetus to get back to work on it, if I were in Rollie's shoes.
akhtar says:
December 29,2010 at 10:2
I think that it is more fixing the website now than the addon, since the addon has been updated.
mtgirl1979 says:
December 29,2010 at 10:7
[quote:0d574866f0="Pennet"][b:0d574866f0]I think people are losing sight of the fact that this is an addon, produced without profit, for a [i:0d574866f0]game.[/i:0d574866f0]....

The sense of entitlement that some of you are displaying is kind of obnoxious, and those entitled comments would not be much impetus to get back to work on it, if I were in Rollie's shoes[/b:0d574866f0].[/quote:0d574866f0]

I agree with the basic message in this.
Rollie says:
December 29,2010 at 18:9
I actually did do a lot of work over Christmas catching up with RL work so I now finally have time to work on the site!

And I did that pretty much all day today. I have working version ready to go and am in the process of uploading to the site now. I will enable uploads but the processing of said uploads probably won't be able to start until tomorrow. I am having to a last minute database change to accommodate the updates.
Rollie says:
December 29,2010 at 18:38
Uploads have been re-enabled, and only have a 30 second time delay requirement. I'll re-enable the standard 15 minutes once folks have had a chance to catch up any backlog they wish to upload!
FuxieDK says:
December 30,2010 at 1:16
[quote:09f2bbf6f0="Rollie"]Uploads have been re-enabled, and only have a 30 second time delay requirement. I'll re-enable the standard 15 minutes once folks have had a chance to catch up any backlog they wish to upload![/quote:09f2bbf6f0]Oh noes.. 30 sec? Then I have to spend 21 minutes on uploading.. :-D
Jokes aside, nice to see, we can upload all our files fast.. ;-)
1974ER says:
December 30,2010 at 3:46
This is great news. :) I probably hold the biggest backlog (unless Balgair has been really busy :D ). However, due to the MASSIVE size of said backlog (over 250 MB), I am going to use a much more conservative upload interval. I uploaded the first of my files a few minutes ago without any problems, so things are looking good. :)

Also, as far as I am concerned, you probably should resume the standard delay only after the holiday season (including new year's eve) is over, as some people might not be able to upload their backlogs today.

On a further note, I see that the DB only shrunk by about 400k characters, from 95M to about 94,6M characters. How are things looking on the database front? Should we be prepared for another overload shutdown? Or are things looking good? :)
Rollie says:
December 30,2010 at 16:29
It's running better, but still slower than I'd like. I am still planning on some additional modifications, but they'll have to wait for now if possible.

Leave a Reply

You must be Logged In in order to post comments.


 
Advertise Here!