Files not processing properly

asmodai · Posted: Mon Oct 27, 2008 6:34 am Post subject:

Are we back on track or playing catchup?

Right now I see files from the 23rd are still not processed on my page update page.

Also, is the graph at the bottom of your WR updates supposed to update? Looks like it stopped at the end of July.

Rollie · Posted: Mon Oct 27, 2008 11:10 am Post subject:

I'm looking into it to see what's up.

As for the graph, that's prob bugged and I'll look into it as well.

asmodai · Posted: Tue Oct 28, 2008 8:54 am Post subject:

Thanks Rollie. Wish I could help (being a programmer myself), but I can at least point out what I am seeing. Smile

bringoutyourdead

I know that around 10MB the web upload process starts to fail accepting.
It now appears that around 15MB uniuploaded files fail to get the data extracted.

The Gzipped file uploads fine (at much less then 15MB!) and then the file gets marked Y processed but with no actual updates.

Pruning my data back and resending will get updates to actually show up.

I'm now having to prune back to under a week.

The joy of being a big census taker and swapping out well counted servers for those that are blinky red under counted.

I know I could just delete/restart my data file when I do server swaps... but I want to retain my counts for the servers I actually play on.

Rollie · Posted: Sun Dec 21, 2008 7:54 pm Post subject:

I'll have to look at the specific file in question, but my guess is that the really big files are timing out during parsing, something I added not too long ago to prevent singular files from hanging the entire process.

bringoutyourdead · Posted: Mon Dec 22, 2008 12:30 pm Post subject:

Yikes!! I was afraid of that.
My 08/16:14:08 file was marked Y processed with zero data.
The interesting thing was that the DB stats page was showing 16:10:xx as the last processed file. (Mixed signals)

I would suggest that you place in faq or sticky that top contributors limit the data file to 10-12 MB, as that seems to be handled ok.

For myself.. I'm going to play the rename game..
I'll keep two sets of CensusPlus data files..
The 1st set will be for the servers I actually play on.. so I can keep longer data on my client.
The 2nd set will be for the servers I just hit for census.. and I'll prune those down to last known good processed date (+1 day, I think, just in case)

That way I don't need to keep resending data you already have, and you don't have to take time to handle my edge condition... the last thing you or I want for your Holiday times.