Usage Stats: Stop Discarding Early Forfeits

For usage stats, should we...


  • Total voters
    46

Antar

is a Battle Server Administratoris a Programmeris a Super Moderatoris a Community Contributor
Official Data Miner
#1
Currently policy is to discard from the usage stats any singles battle in a 6v6 format that lasts for fewer than six turns. This was my first attempt at weeding out players who were attempting to manipulate the usage stats. But since them, we've implemented a full-on weighting scheme, and it's not clear to me that the early-forfeit discarding is actually needed.

And if it is, then shouldn't I be doing something similar for 6v6 doubles and triples formats (throwing out battles that last for fewer than 3 or 2 turns, respectively)?

From a practical standpoint, here are the NU stats (standard weighting) for April 2016, with early forfeits discarded:

Code:
Total battles: 99399
Avg. weight / team: 0.175525
+ ---- + ------------------------- + --------- +
| Rank | Species                   | Usage %   |
+ ---- + ------------------------- + --------- +
|    1 | Mesprit                   |  20.8646% |
|    2 | Lanturn                   |  18.6276% |
|    3 | Hitmonchan                |  17.7743% |
|    4 | Hariyama                  |  14.0140% |
|    5 | Steelix                   |  13.9985% |
|    6 | Rotom                     |  13.7376% |
|    7 | Magmortar                 |  13.0243% |
|    8 | Xatu                      |  12.7274% |
|    9 | Skuntank                  |  12.5584% |
|   10 | Audino-Mega               |  11.8811% |
|   11 | Weezing                   |  11.5753% |
|   12 | Liepard                   |  11.4919% |
|   13 | Scyther                   |  11.4240% |
|   14 | Garbodor                  |  11.3061% |
|   15 | Tauros                    |  10.9387% |
|   16 | Mismagius                 |  10.8708% |
|   17 | Swellow                   |  10.7648% |
|   18 | Pyroar                    |  10.2092% |
|   19 | Malamar                   |  10.1954% |
|   20 | Archeops                  |   9.8025% |
|   21 | Rhydon                    |   9.6564% |
|   22 | Charizard                 |   9.3838% |
|   23 | Vileplume                 |   9.3335% |
|   24 | Samurott                  |   9.2181% |
|   25 | Primeape                  |   9.0710% |
|   26 | Shiftry                   |   8.6677% |
|   27 | Kabutops                  |   8.4692% |
|   28 | Quagsire                  |   8.4078% |
|   29 | Claydol                   |   8.0804% |
|   30 | Omastar                   |   7.0723% |
|   31 | Ferroseed                 |   6.9959% |
|   32 | Abomasnow                 |   6.8065% |
|   33 | Ludicolo                  |   6.7713% |
|   34 | Kecleon                   |   6.6413% |
|   35 | Manectric                 |   6.5683% |
|   36 | Golurk                    |   6.4712% |
|   37 | Lilligant                 |   6.2792% |
|   38 | Electivire                |   6.2766% |
|   39 | Kangaskhan                |   6.2251% |
|   40 | Miltank                   |   5.9729% |
|   41 | Haunter                   |   5.8181% |
|   42 | Poliwrath                 |   5.6313% |
|   43 | Sandslash                 |   5.5999% |
|   44 | Aurorus                   |   5.5935% |
|   45 | Barbaracle                |   5.3339% |
|   46 | Mantine                   |   5.2433% |
|   47 | Regirock                  |   5.2174% |
|   48 | Klinklang                 |   5.1270% |
|   49 | Piloswine                 |   5.0846% |
|   50 | Vivillon                  |   4.9327% |
|   51 | Combusken                 |   4.8835% |
|   52 | Zangoose                  |   4.7228% |
|   53 | Cradily                   |   4.6665% |
|   54 | Jynx                      |   4.6179% |
|   55 | Pinsir                    |   4.1265% |
|   56 | Musharna                  |   3.9405% |
|   57 | Rotom-Fan                 |   3.7803% |
|   58 | Pelipper                  |   3.7585% |
|   59 | Muk                       |   3.7307% |
|   60 | Cacturne                  |   3.5989% |
|   61 | Torterra                  |   3.4206% |
|   62 | Floatzel                  |   3.2662% |
|   63 | Mawile                    |   3.0433% |
|   64 | Gourgeist-Super           |   2.9369% |
|   65 | Throh                     |   2.1252% |
|   66 | Crustle                   |   2.1145% |
|   67 | Carracosta                |   2.0543% |
|   68 | Grumpig                   |   1.8546% |
|   69 | Exeggutor                 |   1.7341% |
|   70 | Prinplup                  |   1.6727% |
|   71 | Cryogonal                 |   1.5799% |
|   72 | Butterfree                |   1.5339% |
|   73 | Golduck                   |   1.3970% |
|   74 | Sliggoo                   |   1.3251% |
|   75 | Linoone                   |   1.2709% |
|   76 | Electrode                 |   1.2433% |
|   77 | Torkoal                   |   1.2093% |
|   78 | Kadabra                   |   1.1977% |
|   79 | Raichu                    |   1.1805% |
|   80 | Articuno                  |   1.1435% |
|   81 | Tangela                   |   1.1426% |
|   82 | Ursaring                  |   1.1028% |
|   83 | Flareon                   |   1.0741% |
|   84 | Camerupt                  |   1.0727% |
|   85 | Ninetales                 |   1.0399% |
|   86 | Roselia                   |   1.0132% |
|   87 | Probopass                 |   1.0122% |
|   88 | Victreebel                |   1.0062% |
|   89 | Chatot                    |   1.0044% |
|   90 | Clefairy                  |   0.9978% |
|   91 | Rampardos                 |   0.9647% |
|   92 | Lapras                    |   0.9486% |
|   93 | Golem                     |   0.8686% |
|   94 | Avalugg                   |   0.8537% |
|   95 | Ampharos                  |   0.8532% |
|   96 | Dusclops                  |   0.8384% |
|   97 | Altaria                   |   0.8263% |
|   98 | Leafeon                   |   0.8110% |
|   99 | Armaldo                   |   0.7928% |
|  100 | Leavanny                  |   0.7672% |
|  101 | Meowstic                  |   0.7666% |
|  102 | Arbok                     |   0.7647% |
|  103 | Ditto                     |   0.7592% |
|  104 | Gourgeist-Small           |   0.7496% |
|  105 | Dusknoir                  |   0.7053% |
|  106 | Jumpluff                  |   0.6897% |
|  107 | Rapidash                  |   0.6795% |
|  108 | Frogadier                 |   0.6675% |
|  109 | Volbeat                   |   0.6544% |
|  110 | Mr. Mime                  |   0.6067% |
|  111 | Regice                    |   0.6047% |
|  112 | Beheeyem                  |   0.5898% |
...
Note that this table differs slightly from what's up here. These stats were calculated using Onix, my rewritten stats engine which is being actively developed. Part of the differences I expect due to tech debt, but the battle counts should be the same, so I'm not sure if there's a bug in the old code or a bug in the new code.


And now here are the same stats leaving in those early forfeits:

Code:
Total battles: 118321
Avg. weight / team: 0.162388
+ ---- + ------------------------- + --------- +
| Rank | Species                   | Usage %   |
+ ---- + ------------------------- + --------- +
|    1 | Mesprit                   |  20.6261% |
|    2 | Lanturn                   |  18.5598% |
|    3 | Hitmonchan                |  17.7762% |
|    4 | Steelix                   |  13.8677% |
|    5 | Hariyama                  |  13.8620% |
|    6 | Rotom                     |  13.5643% |
|    7 | Magmortar                 |  13.1020% |
|    8 | Xatu                      |  12.8665% |
|    9 | Skuntank                  |  12.4685% |
|   10 | Audino-Mega               |  11.8592% |
|   11 | Weezing                   |  11.6744% |
|   12 | Scyther                   |  11.5123% |
|   13 | Liepard                   |  11.5042% |
|   14 | Garbodor                  |  11.1232% |
|   15 | Mismagius                 |  10.9286% |
|   16 | Tauros                    |  10.8244% |
|   17 | Swellow                   |  10.8062% |
|   18 | Malamar                   |  10.3085% |
|   19 | Pyroar                    |  10.2878% |
|   20 | Archeops                  |   9.8251% |
|   21 | Rhydon                    |   9.6103% |
|   22 | Charizard                 |   9.4179% |
|   23 | Vileplume                 |   9.2409% |
|   24 | Samurott                  |   9.2191% |
|   25 | Primeape                  |   9.1041% |
|   26 | Shiftry                   |   8.5962% |
|   27 | Quagsire                  |   8.4316% |
|   28 | Kabutops                  |   8.4272% |
|   29 | Claydol                   |   8.1475% |
|   30 | Omastar                   |   7.0709% |
|   31 | Ferroseed                 |   6.9947% |
|   32 | Ludicolo                  |   6.8168% |
|   33 | Abomasnow                 |   6.7906% |
|   34 | Kecleon                   |   6.6149% |
|   35 | Golurk                    |   6.5317% |
|   36 | Manectric                 |   6.5290% |
|   37 | Electivire                |   6.3985% |
|   38 | Lilligant                 |   6.2678% |
|   39 | Kangaskhan                |   6.1504% |
|   40 | Miltank                   |   5.9777% |
|   41 | Haunter                   |   5.8844% |
|   42 | Poliwrath                 |   5.6078% |
|   43 | Sandslash                 |   5.5700% |
|   44 | Aurorus                   |   5.5693% |
|   45 | Barbaracle                |   5.3862% |
|   46 | Mantine                   |   5.2810% |
|   47 | Klinklang                 |   5.2338% |
|   48 | Regirock                  |   5.1689% |
|   49 | Piloswine                 |   5.1054% |
|   50 | Vivillon                  |   5.0269% |
|   51 | Combusken                 |   4.9075% |
|   52 | Zangoose                  |   4.7929% |
|   53 | Cradily                   |   4.7737% |
|   54 | Jynx                      |   4.6583% |
|   55 | Pinsir                    |   4.1524% |
|   56 | Musharna                  |   3.9026% |
|   57 | Rotom-Fan                 |   3.7835% |
|   58 | Muk                       |   3.7766% |
|   59 | Pelipper                  |   3.7431% |
|   60 | Cacturne                  |   3.6434% |
|   61 | Torterra                  |   3.4401% |
|   62 | Floatzel                  |   3.2396% |
|   63 | Mawile                    |   3.1127% |
|   64 | Gourgeist-Super           |   2.8804% |
|   65 | Crustle                   |   2.1246% |
|   66 | Throh                     |   2.1111% |
|   67 | Carracosta                |   2.0425% |
|   68 | Grumpig                   |   1.7916% |
|   69 | Exeggutor                 |   1.7510% |
|   70 | Prinplup                  |   1.6255% |
|   71 | Cryogonal                 |   1.5659% |
|   72 | Butterfree                |   1.5376% |
|   73 | Golduck                   |   1.4303% |
|   74 | Sliggoo                   |   1.3104% |
|   75 | Linoone                   |   1.2945% |
|   76 | Electrode                 |   1.2450% |
|   77 | Torkoal                   |   1.1939% |
|   78 | Raichu                    |   1.1908% |
|   79 | Kadabra                   |   1.1773% |
|   80 | Articuno                  |   1.1251% |
|   81 | Tangela                   |   1.1067% |
|   82 | Flareon                   |   1.0872% |
|   83 | Ursaring                  |   1.0725% |
|   84 | Camerupt                  |   1.0474% |
|   85 | Ninetales                 |   1.0394% |
|   86 | Chatot                    |   1.0294% |
|   87 | Victreebel                |   1.0169% |
|   88 | Probopass                 |   1.0026% |
|   89 | Roselia                   |   0.9951% |
|   90 | Rampardos                 |   0.9611% |
|   91 | Clefairy                  |   0.9610% |
|   92 | Lapras                    |   0.9547% |
|   93 | Golem                     |   0.8716% |
|   94 | Avalugg                   |   0.8561% |
|   95 | Ampharos                  |   0.8469% |
|   96 | Dusclops                  |   0.8445% |
|   97 | Leafeon                   |   0.8153% |
|   98 | Altaria                   |   0.8147% |
|   99 | Armaldo                   |   0.8131% |
|  100 | Leavanny                  |   0.7702% |
|  101 | Ditto                     |   0.7640% |
|  102 | Arbok                     |   0.7601% |
|  103 | Meowstic                  |   0.7568% |
|  104 | Gourgeist-Small           |   0.7499% |
|  105 | Dusknoir                  |   0.6970% |
|  106 | Frogadier                 |   0.6797% |
|  107 | Jumpluff                  |   0.6693% |
|  108 | Rapidash                  |   0.6680% |
|  109 | Volbeat                   |   0.6529% |
|  110 | Mr. Mime                  |   0.6139% |
|  111 | Beheeyem                  |   0.6068% |
|  112 | Regice                    |   0.6056% |
...
[/quote]

We only have to go four mons down to see a difference in the ranking. And since this early-forfeit policy is known, it's unlikely anyone was even attempting to game the stats in this way!

Still, the differences are small, and the philosophical rationale is important. Hence why I pose this to PR rather than making an executive decision on my own.
 

Josh

=P
is a Team Rater Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnus
Mafia Champion
#3
Don't discard early forfeits. We have weighting specifically to removr their influence (among other things). If someone's forfeiting turn 1 in the 1800s that's obviously not sustainable since their elo will plummet and they had to win to get up there, so the stats are still valid in my opinion. If someone is forfeiting turn 1 in the 1100s that's still fine because low ladder is already weighted extremely low, it's fairly insignificant. The mons still got used and lost. That said, below 1050 elo is the exception imo, because you can spam battles and quit right away and in most tiers get them almost immediately (6 in two mins is the max iirc which is no problem).
 

Antar

is a Battle Server Administratoris a Programmeris a Super Moderatoris a Community Contributor
Official Data Miner
#4
So I've given this a little thought, and I think I'm going to go for a fourth option: discard battles fewer than 1 turn in length across the board (~5% of all OU battles from April 2016, for reference).

This should remove the vast majority of battles with connection problems or where one user, say, went to the bathroom in between requesting a battle and getting matched. Beyond that I do think additional drops adds needlessly complex logic and is redundant with weighting.

If there are no objections, I'll put this policy into effect starting this month.