WarcraftRealms.com

Re: Who name selection criteria

1974ER — Wed Nov 28, 2012 4:43 am

Author: 1974ER

Posted: Wed Nov 28, 2012 4:43 am

We may be nuts... but in a good way. Also, I would tend to agree with Bringoutyourdead's decision. Making alterations or more precisely addition(s) to the character list would increase the length of census without having a large improving effect on accuracy. One overall tricky thing is that there are such huge variations between factions, even within a single server and on large servers, also within the day. Some large realms double, triple, or in rarer cases quadruple their activity levels when one compares early mornings to the peak hours between roughly 19.30 and 21 (at least on EU servers, I haven't really looked into how things run in the US, with the higher timezone spread, etc).

Re: Who name selection criteria

bringoutyourdead — Tue Nov 27, 2012 11:59 am

Author: bringoutyourdead

Posted: Tue Nov 27, 2012 11:59 am

I have placed an alternate character list in the program, but to be honest the differences in processing time vs. completeness of data capture is so little that I don't feel there is any reason to change the list already in use.

Re: Who name selection criteria

Rollie — Mon Nov 19, 2012 11:01 pm

Author: Rollie

Posted: Mon Nov 19, 2012 11:01 pm

You guys are nuts =)

So for the Cliff Notes version of this, does it make sense to adjust the filtering?

Re: Who name selection criteria

bringoutyourdead — Fri Nov 18, 2011 4:59 pm

Author: bringoutyourdead

Posted: Fri Nov 18, 2011 4:59 pm

Just for giggles.. I ran the 15 least common english letters as selectors and with all of them only found 90% of the names.

Re: Who name selection criteria

bringoutyourdead — Fri Nov 18, 2011 2:07 pm

Author: bringoutyourdead

Posted: Fri Nov 18, 2011 2:07 pm

1974ER wrote:

Whether by desing or accident, the database no longer has a cutoff point.

Ah! I wasn't aware that Rollie had opened his storage back up to all levels again. I stopped playing right after the first of the year, when he was having problems. Well that makes the switch option I mentioned meaningless.

1974ER wrote:

Your first macro was tested several times... if there are more than 49 then the result always looks like this.
num = 50,Count = 49

Good.. hmm, unfortunate. as www.wowwiki.com shows

Quote:

numWhos, totalCount = GetNumWhoResults();
totalCount
Number - number of users matching the query
numWhos
Number - number of entries actually returned

Which implied the possibility that totalCount could actually be 'Total Count"
instead it really is nothing more then a indicator of incomplete return of all available information. sigh. that tosses aside another idea I had.
I had more then half expected this result, as it is how libwho-2.0 library presents the data back to the calling program.

1974ER wrote:

If the number is less than 49:
num = x,Count = x, where x is the number of characters found by the /who.

Which is what it should. {at least it shows I have learned something of the code. } And it verifies that the current CensusPlus code is vulnerable to data loss/corruption by another mod doing who requests directly or via the libwho calls.
The solution to the problem is to just finish the integration of the libwho library with CensusPlus.

1974ER wrote:

Also tried the second one multiple times, it produced another result of nothing.

That is the fun of working with Blizzards API stuff. They make it available but do not support. Much of the understanding that people had figured out about the API got lost when Blizzard changed the forum software. The code fragments are still available on the web, but the discussion about how to use them disappeared.
As I was giving you the DEFAULT_CHAT_FRAME: I was saying to myself, well yes all these examples in the wiki show this.. but where is default_chat_frame defined?

Re: Who name selection criteria

1974ER — Thu Nov 17, 2011 6:22 pm

Author: 1974ER

Posted: Thu Nov 17, 2011 6:22 pm

More comments:

Sample is too small in terms of SERVERS, not names themselves.

There is a chance that he didn't do one at all, but went with data from Wikipedia or other similar source.

As to your "Hmm". I did think it over, several times in fact, before I posted and I stand by my desire to have the high hit selectors first. And I stand by it exactly because the count isn't instant, not even close. (Semi-)instant counts would render the order irrelevant.

I did not actually look at or think of the graphs before your comment... but to be honest, the graphs support my way of doing things. As the character count goes up, the likelyhood of for example As to "blow 50" increases, rendering their exact amount irrelevant to a degree. However, the rare triggers will generally catch more characters the later they are in the sequence. And due to the population patterns, this will be true roughly 21 to 22 hours per day, the only exception being the point where population drops quickly and decisevily withing a short time period, generally between 22 and 23.30, server time, at least on all fairly high population EU servers I took a quick, closer look at (about a dozen). And even during that period, the high hit triggers first principle will guarantee that as many characters as possible are seen before they log off for the day...

Granted, things would be very different if the population was almost flat, but high around the clock. However, I have not seen a single server that would have that. Flat, low, sure, but flat, high, no. On a further note, low population servers aren't even very relevant, because they generally fail to even trigger letter level checks. Some fail to trigger class level checking... and some, very small ones... fail even to trigger a racial level check even on character level 85.

Secondly, http://www.warcraftrealms.com/charsheet/131852629 is one of my censusalts. Whether by desing or accident, the database no longer has a cutoff point. It appears more likely that the system now purges level 1s (and maybe up to level 9 or so) who aren't seen regularly enough. So or so, there are now very low level characters in the database and high level characters created after the last couple of times Rollie tweaked things, have histories starting from level 1 or so, if the faction has been censused a lot, example here, one of my newer alts:

http://www.warcraftrealms.com/charsheet/138693416

This differs from a lot of older characters, none of whom have histories below level 10... unless we go to really old characters who existed before cutoffs were ever implemented, in other words, as far as I can tell, mostly Vanilla and very early TBC stuff.

Your first macro was tested several times... if there are more than 49 then the result always looks like this.

num = 50,Count = 49

If the number is less than 49:

num = x,Count = x, where x is the number of characters found by the /who.

Also tried the second one multiple times, it produced another result of nothing.

In closing as it is too late again...

Ideally, we would like a wholely unlimited /who, which would allow us at maximum to do /who 1-85 and get a quick result. As has been noted elsewhere, even a limit of, for example 99 instead of 49 might basically eliminate the "blowover problems".

Re: Who name selection criteria

bringoutyourdead — Thu Nov 17, 2011 2:49 pm

Author: bringoutyourdead

Posted: Thu Nov 17, 2011 2:49 pm

1974ER wrote:

The amount of duplicates is so low, because you actually have too small a sample.

Hey! 62 thousand is a large number... why it is .. let us see.. hmm 0.07% of all characters listed in the Warcraft Realms database!

1974ER wrote:

As for letter order, it doesn't matter for you, since you work on a ready set of data. .

Truth.. but I have no other option, so I must work with what I have. I would love to have Rollie comment here that he ran a frequency analysis on an early subset of his data.. something like 20-30 million characters.

1974ER wrote:

I was referring to the impact of letter order to actual results of a census run. Why is that important? Well, simply because the censuses take sufficiently long that people can log in or out during the process. The most accurate method would check letters such as A, E, R, I and N first, for the rarer ones later.

Hmm.. you might want to think that over again. By the activity graphs found on this site, approximately 15 hours a day the later the time the more players would be counted on instant counts. So that would imply you would want to the high hit selectors later in your census run {which is what Rollie does}

As far as "accuracy and efficiency" are concerned, I think we just need to respectfully agree that we disagree.

One of the advantages of improved efficiency would be the ability to increase the number of chances to catch characters online. A simple change would be to include an option switch to allow a user to stop census runs at the Minimum Cutoff that Rollie had to accept on his database. (Or to have the run go all the way to 1-1 for local use.)

1974ER wrote:

Quote: "In a real world situtation if the complete new keys were used: 1974ER who currently has an update count of 16,139,532 with the new key update count would be 18,721,857"
Interesting hypotesis, but wrong.

Yes I know it was fallacious, but I couldn't find the tongue-in-cheek emoticon.

1974ER wrote:

I also tested the macro as supplied and suggested by you on multiple factions. The results: Absolutely nothing happened, no result, no error message, nothing, just nothing. Sorry.

Hmm.. not what I expected.. at least I thought it would print 'num = ,Count ='
ok.. how about this instead
EDIT: stop the presses... error found not GenNum.. but GetNum.. sigh
after submitting the manual /who that returns max results(+)
/run local numResults, totalCount = GetNumWhoResults(); print("num = ",numResults," ,Count = ",totalCount);
and if that doesn't display then
/run local numResults, totalCount = GetNumWhoResults(); DEFAULT_CHAT_FRAME:AddMessage(format("num = %i Count = %i",numResults,totalCount);

Assuming you get numbers with this.. an interesting experiment would be to do a /who 1-85
and see what numResults returns.. it could be the total number of characters in current faction online at that time.

Re: Who name selection criteria

1974ER — Wed Nov 16, 2011 3:44 am

Author: 1974ER

Posted: Wed Nov 16, 2011 3:44 am

Whoa, so many posts...

I'll try to comment some:

The amount of duplicates is so low, because you actually have too small a sample. For example, as of this writing, there are 207 "Barbie"s and 131 "L�golas"es on EU realms alone. (Figures from EU Armory.)

As for letter order, it doesn't matter for you, since you work on a ready set of data. I was referring to the impact of letter order to actual results of a census run. Why is that important? Well, simply because the censuses take sufficiently long that people can log in or out during the process. The most accurate method would check letters such as A, E, R, I and N first, for the rarer ones later.

Quote: "My 2nd goal is to investigate if there is anyway to (1st improve the accuracy of the census, 2nd to make the process of gathering the data faster.)"

If you have paid close attention to what I have written so far, you should have noticed that accuracy and efficiency are somewhat counterproductive and what we have right now appears to be a fairly good compromise.

Quote: "In a real world situtation if the complete new keys were used: 1974ER who currently has an update count of 16,139,532 with the new key update count would be 18,721,857"

Interesting hypotesis, but wrong. You can't extrapolarate my results from your data, because it's partially based on realms I can't access at all and largely based on realms I visit only rarely, if ever. Even if the data was based entirely on my results, the triggers, the behaviour (and character names) of all other players and my own censusing habits would still impact the end results.

I also tested the macro as supplied and suggested by you on multiple factions. The results: Absolutely nothing happened, no result, no error message, nothing, just nothing. Sorry.

Re: Who name selection criteria

bringoutyourdead — Tue Nov 15, 2011 5:41 pm

Author: bringoutyourdead

Posted: Tue Nov 15, 2011 5:41 pm

Stepping away from rows and rows of names... and looking at code again.

Would you run the following macro for me.

First do a /who query that will fail with too many results (censusPlus must not be running.. nor anything else that would call /who)

/Who 85-85 will probably work

after that is done, enter and execute the following macro from the chat line

/run numResults, totalCount = GenNumWhoResults(); out="num = "..numResults..",Count = "..totalCount;print(out);

this should get the counts from your who request, format them for easy reading and print the output.

Count should be the 49 limit that can be displayed.
num will be??? I believe it will be the actual number of level 85 characters online when the server process the /who request.

Re: Who name selection criteria

bringoutyourdead — Tue Nov 15, 2011 4:54 pm

Author: bringoutyourdead

Posted: Tue Nov 15, 2011 4:54 pm

So Rollie did select a reasonable collection of keys.

In a real world situtation if the complete new keys were used: 1974ER who currently has an update count of 16,139,532 with the new key update count would be 18,721,857

Re: Who name selection criteria

bringoutyourdead — Tue Nov 15, 2011 4:46 pm

Author: bringoutyourdead

Posted: Tue Nov 15, 2011 4:46 pm

So what if we change selectors and use the top 15 most frequent letters as found in my names list?
This will have to be the next posting.
Most common name... there are 16 names that are used 3 times and 653 names used 2 times.

Out of 62804 names in list, 62687 are found and 117 are missed
Duplicate finding of names 253030 times.

Letters with the _ after are the new substitute selectors.
-- count of new names found -- Percent with letter -- running total percentage - duplicate names found again -- percentage duplicates -- dup vs. find

A 1 = 37077 -- 59.4%
E 2 = 14112 -- 22.47% -- 81.51% -- 16521 -- 26.31% -- 1.17:1
R 3 = 4449 -- 7.08% -- 88.59% -- 25697 -- 40.92% -- 5.78:1
I 4 = 3476 -- 5.53% -- 94.12% -- 23745 -- 37.81% -- 6.83:1
N _5 = 1075 -- 1.71% -- 95.84% -- 24539 -- 39.07% -- 22.83:1
O 6 = 1240 -- 1.97% -- 97.81% -- 22523 -- 35.86% -- 18.16:1
L _7 = 467 -- 0.74% -- 98.55% -- 22019 -- 35.06% -- 47.15:1
S 8 = 304 -- 0.48% -- 99.04% -- 21175 -- 33.72% -- 69.65:1
T 9 = 113 -- 0.18% -- 99.22% -- 19378 -- 30.85% -- 171.49:1
H _10 = 96 -- 0.15% -- 99.37% -- 14761 -- 23.50% -- 153.76:1
D 11 = 75 -- 0.12% -- 99.49% -- 13603 -- 21.66% -- 181.37:1
U 12 = 93 -- 0.15% -- 99.64% -- 13450 -- 21.42% -- 144.62:1 -- at this point we have found ~ the same character percentage as the current selector keys find.
M _13 = 60 -- 0.10% -- 99.73% -- 13262 -- 21.12% -- 221.03:1
K _14 = 39 -- 0.06% -- 99.80% -- 12092 -- 19.25% -- 310.05:1
C 15 = 11 -- 0.02% -- 99.81% -- 10265 -- 16.34% -- 933.18:1

Old key duplicates found 204876, new key duplicates found 253030:
Old key duplicates Vs. Found = 3.27:1
New key duplicates Vs. Found = 4.04:1

So we could swap selectors and get 0.16% (117) more names found from this universe of 62804 characters.
Or we could cut the number of letter selector keys by 3 and process slightly faster. Dropping to 217411 duplicates and 62583 found or a ratio of 3.47:1.

Personally I would rather find the extra 100 characters in the universe. It is just a matter of asking the server the right questions.

Re: Who name selection criteria

bringoutyourdead — Tue Nov 15, 2011 12:47 pm

Author: bringoutyourdead

Posted: Tue Nov 15, 2011 12:47 pm

And the most common name (out of 62804 listed) is....
not worth telling..
there are 16 names listed exactly the same 3 times.
and 653 names that are exact duplicates (ie 2x)

And Legolas wasn't listed.. however Legol�s and Legolaz were.

Re: Only looking n-letter selections

bringoutyourdead — Mon Nov 14, 2011 9:27 pm

Author: bringoutyourdead

Posted: Mon Nov 14, 2011 9:27 pm

bringoutyourdead wrote:

Student: (Oh wait.. that is me!)
If the same selector letters are used in a different order will that impact total found vs. not found: answer no!
If the same selector letters are used in a different order will that impact duplicate counts and percentages (show numbers in your answer):
NO!, wait YES!, wait Maybe? That will require time and another posting.

Reverse order (correct order)
Y = 9343 -- 14.88%
U = 11894 -- 18.94% -- 33.81% -- 1649 -- 2.63% -- 0.13:1
T = 12886 -- 20.52% -- 54.33% -- 6605 -- 10.52% -- 0.51:1
S = 8484 -- 13.51% -- 67.84% -- 12995 -- 20.69% -- 1.53:1
R = 9851 -- 15.69% -- 83.53% -- 20295 -- 32.31% -- 2.06:1
P = 1191 -- 1.90% -- 85.42% -- 6122 -- 9.75% -- 5.14:1
O = 3278 -- 5.22% -- 90.64% -- 20485 -- 32.62% -- 6.25:1
I = 2993 -- 4.77% -- 95.41% -- 24228 -- 38.58% -- 8.09:1
G = 439 -- 0.70% -- 96.11% -- 9229 -- 14.69% -- 21.02:1
F = 208 -- 0.33% -- 96.44% -- 5430 -- 8.65% -- 26.11:1
E = 1146 -- 1.82% -- 98.26% -- 29487 -- 46.95% -- 25.73:1
D = 215 -- 0.34% -- 98.61% -- 13462 -- 21.43% -- 62.32:1
C = 152 -- -- 0.24% -- 98.85% -- 10124 -- 16.12% -- 66.32:1
B = 133 -- -- 0.21% -- 99.06% -- 8060 -- 12.83% -- 60.60:1
A = 372 - - - 0.59% -- 99.65% -- 36705 -- 58.44% -- 98.67:1

And we find as predicted.. The number of names found by the selector doesn't change.. since the selector done change .. order is irrelevant
And we find as (hmmm.) .. The total number of duplicate counts doesn't change.. therefore again order is irrelevant.

Further data mining nuggets:
Length of names that were missed:
7 letters = 4 names
6 letters = 9 names
5 letters = 33 names
4 letters = 70 names
3 letters = 73 names
2 letters = 29 names

Most common letters in the missed names: (A..Z only)
L = 76 names
N = 66 names
K = 47 names
X = 45 names
M = 43 names
Z = 35 names
J = 29 names
H = 27 names
W = 18 names
V = 17 names
Q = 10 names

10 Names have no common letters (A..Z)

So what if we change selectors and use the top 15 most frequent letters as found in my names list?
This will have to be the next posting.
Along with the most common name (single unique spelling) found.

Re: Who name selection criteria

bringoutyourdead — Mon Nov 14, 2011 9:07 pm

Author: bringoutyourdead

Posted: Mon Nov 14, 2011 9:07 pm

1974ER wrote:

the individual check by letter checks start from the end and finish off with As.

Ah you are correct I forgot the queues are created as LIFO (last in first out) .. but since I planned to play with order changes just too see what happens.. it isn't a big loss.

As far as the impact of duplicate names... it is surprisingly small... of the 62804 names in my lists, only 669 duplicate names are found or 1.1%. This is such a small amount.. I decided to ignore the small error that would be added to my percentages. (I may rethink this, but not until after I have looked through the various letter combos and orders)

Concerning your comment in regards to letter efficiency vs realm/language. Well yes that is given.. but meaningless.. since we only have one version of CensusPlus to be used on any/all realm. (Nor would I even want to suggest trying to maintain different versions!)

1974ER wrote:

but we don't want lots of triggers.

Not necessarily. As CensusPlus is now, Yes - the more triggers the slower it runs. But if the logic is changed and built in delays eliminated or at least decreased we may find that we could have more triggers and faster processing.

1974ER wrote:

Granted, I am a firm proponent of accuracy and some of your ideas suggest that you would be willing to sacrifice accuracy to speed things up...

No that isn't the case.

My 1st goal is to learn Lua programming, XML and event driven object programming.
My 2nd goal is to investigate if there is anyway to (1st improve the accuracy of the census, 2nd to make the process of gathering the data faster.)

As I stated at the beginning of this exercise, I'm going at it from the "feeling" that what CensusPlus uses as triggers might not be the best.
I may find that after all is said and done, that Rollie did pick the best triggers.

Re: Who name selection criteria

1974ER — Thu Nov 10, 2011 7:04 pm

Author: 1974ER

Posted: Thu Nov 10, 2011 7:04 pm

Stop! Halt! Seis! And so on!!!

You can't apply frequency math to the names like that!!! Why? Because the same names have a chance of repeating on multiple realms. In fact, you would be hard pressed to find a realm that does NOT have a Legolas, etc. Trying to find multiple characters called... say, Undowar (one of mine) would be much less likely. Both are seven characters long, have 3 vowels, etc, but they still share only 2 triggers.

Also, the individual check by letter checks start from the end and finish off with As.

Not to mention the fact that even the same letter has a variable efficiency dependant on the realm... Any check for � would be far more effective on a French faction than an English one, same would apply for � on German factions. Even the famous letter A isn't equally frequent on all realms!

And even further, provided that there are sufficiently many characters of specific type about... let's say we have 104 human paladins of level 85 online. 85 of the paladins have a name with A(s). 65 of them also have E(s), so there is overlap. In addition there are Z��, Zoyx, Wii and Quu.

Now, Census makes the first check and blows 50 on race, class and level. This first check may detect 0-4 of the "special" characters. This is where the trigger selection becomes important.

If we only use trigger "A", we will find 50 results for a grand total of 50 to 89. If we use triggers "A" and "E", we'll get 50 the "E"s too, with a theoretical maximum of 104 hits total. If we only use trigger "Z", we are guaranteed two hits and up to 50 randoms. But if we look for "A, E, I, O, U and Y" we will have a chance to find everyone except Z�� and even she might be found accidentally by the first check.

The thing is... the less triggers there are, that much more important it's to include "A" and "E" among them to maximise accuracy. If we intended to use a huge number of triggers, we might get away with not using "A" and / or "E"... but we don't want lots of triggers.

Granted, I am a firm proponent of accuracy and some of your ideas suggest that you would be willing to sacrifice accuracy to speed things up...

Anyway, I have been awake too long again, so I am stopping for now. :/