Proposal Ladder Bots and Usage-Based Tiering

[LBN]

This is mostly rambling, be warned.
I think usage based tiering, at least unconditional usage-based tiering has overstayed its welcome. I've always felt it was silly how generations would end with Pokemon residing in a tier far higher than what its viability would suggest.

I'm aware this isn't the primary scope of the thread, but the mere existence of "OU by technicality" kind of explains pretty much all my gripes with the formula., for usage-based tiering I understand those tiers are the ones with an on-going ladder, but I see no real reason why UU can't have those Pokemon, and that logic can apply for every tier as well. Basically what I'm saying is there's a lot of instances over time where outright bad mons stick around in a tier ladder for god knows what reason and the tier below gets starved of a valuable Pokemon, or post-gen bans ruin a Pokemon's viability and then its stuck in purgatory. SV RU Forretress, BW RU Shitmonchan and BW OU Venusaur come to mind here.

Regardless, the focus of this thread is the bot problem, and I wholeheartedly agree with the above suggestions for council-vetoes, disallowing the use of bots entirely, and so on. What I haven't seen mentioned as an option in this thread, and while it is somewhat of a half-measure, missangelic's proposal of raising the requirement to make a mon rise out of a tier does alleviate this problem to an extent. I actually like the proposal a lot, and I think her proposal of raising only the rise-threshold would satisfy a lot of people's grievances in the thread here.

 

[Karxrida]

and some bots are basically just climbing ladder by taking forever every turn until opponents quit.

As a TF2 player whose game has been plagued by botting for years I think this alone is a good enough reason to blanket ban bots. But if you add that with how their presence leads to easily manipulating usage data (intentionally or otherwise) and their ability to scrounge for replays, you can see that they enable way too many avenues for malicious intent to be safely permitted. Threaten people with permabans for bot usage, I don't care. Just make a rule about it if nothing else to dissuade most users and leave only the obvious ones to be whacked with the hammers.

 

[Greybaum]

If banning the bots isn't technically feasible, is it possible to look at an allowlist rather than a blocklist?
Fully acknowledge that this would require a significant amount of unpaid work (and in the meantime we'd have to suffer with the status quo) but if we could set up a linked-registration system and only tier off usage stats from people with Smogon accounts then it'd fix pretty much every problem here. Instead of reactively removing accounts from the system after the fact - which is an idea that has already been dismissed, it'd be an opt in system. If this isn't possible either for technical or for personnel reasons then fair enough!

quick edit: To be clear this is only about usage stats. Absolutely nothing to prevent offsite users from laddering or otherwise playing on the site,

 

[Finchinator]

Some people reached out to me to make sure this discussion did not die out and to get a formal proposal in writing, so to lay out what I personally feel is best as the tier leader of OU:

• Stop dead-end discussion of (blanket) bans on bots as this was already deemed impossible by PS side
• Continue using weighted usage statistics to dictate tiering
• Implement a veto system only accessible to tier leaders and tiering admins for proven outliers
• Monitor situation for remainder of generation, reassess before release of generation ten

First bullet is most obvious: we should not waste time or exhaust resources on what was already stated to be a non-starter. We are in the business of finding solutions, not talking in circles for pages. Cassiopeia and others have already taken time to clarify this issue from that side.

Second bullet is half a matter of the status quo being salvageable and half process of elimination. We have worked with usage based tiering for all of Smogon's (modern) history now; this practice never has been and never will be perfect, but bots do not prove to be an existential threat to it and later in this post I will explain the way to workaround the current issue. As for alternatives, I do not subscribe to any of them as viable alternatives. Viability based tiering is a non-starter for me for the reasons I mention here; we would end up running into more trouble than good if we take this path, in my opinion. Tournament usage is at least intriguing on the surface level, but, upon doing some digging, the sample size issue is a huge problem. You would be using less than 2% of the sample we currently use and it would be susceptible to intentional human manipulation in circuit tours or OST -- whatever publicly joinable tournaments exist -- that far outweighs the threat level of bots on an active ladder. I outline more on this topic here.

Third bullet is the important part and all I care you takeaway from this post: If we want to have a "veto" mechanism to circumvent tiering changes, we need to be very specific about how it is applied and very careful about who can petition for it. We cannot ever risk infringing a veto upon natural tiering changes through legitimate usage or compromising the legitimacy of our current system. We should lay out the specific and airtight framework behind a tiering veto as follows:

• A veto can only be triggered by the affected tier leader with the approval of a tiering admin.
  • A veto can be petitioned for by a council member or tier leader of the affected tier. The community can request the council or tier leader look into it, but we will not just have a field day that cherry picks every single rise. This will otherwise make the whole process more of a mess than it is worth.
• In order for a veto to be triggered, tiering officials must get in contact with PS staff with some type of credible lead or evidence that can be used as probable cause to investigate the matter. Moreover, there must be some discernable evidence that bots have been used to inflate usage of a Pokemon.
  • We should only be implementing a veto when we are positive that a Pokemon would not hit reach the usage threshold without the usage inflation of ladder bots during the usage period. Evidence such as, but not limited to, prior usage stats, specific alts of the bots, and battle logs of the relevant alts can assist in this process.
• A veto should only be allowed within a certain period immediately following tiering changes.
  • We should not be waiting weeks to investigate these as the following usage period will be impacted more as more time elapses.
  • Vetos should not be seen as reactionary investigations to people being upset with changes, but rather immediate procedural fixes.
• Any and all vetos must be logged publicly in order to maximize transparency and assure public trust in this tiering tool being used in good faith to maximize the sanctity of usage based tiering.

If we can apply these or similar rules that get us as close to airtight framework as possible for vetos, which can serve as the inverse of quickbans functionally, then I think we are able to maintain our current system while being strictly better in regards to combatting ladder bots.

Finally, the fourth bullet just gives us room to revoke this in case it is not needed or make changes in the event that we need to. I think any new practices should be reevaluated at the end of their debut generation in order to make sure they are interacting with tiering in the intended fashion after all.

 

[UltraSword]

Third bullet is the important part and all I care you takeaway from this post: If we want to have a "veto" mechanism to circumvent tiering changes, we need to be very specific about how it is applied and very careful about who can petition for it. We cannot ever risk infringing a veto upon natural tiering changes through legitimate usage or compromising the legitimacy of our current system. We should lay out the specific and airtight framework behind a tiering veto as follows:

• A veto can only be triggered by the affected tier leader with the approval of a tiering admin.
  • A veto can be petitioned for by a council member or tier leader of the affected tier. The community can request the council or tier leader look into it, but we will not just have a field day that cherry picks every single rise. This will otherwise make the whole process more of a mess than it is worth.
• In order for a veto to be triggered, tiering officials must get in contact with PS staff with some type of credible lead or evidence that can be used as probable cause to investigate the matter. Moreover, there must be some discernable evidence that bots have been used to inflate usage of a Pokemon.
  • We should only be implementing a veto when we are positive that a Pokemon would not hit reach the usage threshold without the usage inflation of ladder bots during the usage period. Evidence such as, but not limited to, prior usage stats, specific alts of the bots, and battle logs of the relevant alts can assist in this process.
• A veto should only be allowed within a certain period immediately following tiering changes.
  • We should not be waiting weeks to investigate these as the following usage period will be impacted more as more time elapses.
  • Vetos should not be seen as reactionary investigations to people being upset with changes, but rather immediate procedural fixes.
• Any and all vetos must be logged publicly in order to maximize transparency and assure public trust in this tiering tool being used in good faith to maximize the sanctity of usage based tiering.

If we can apply these or similar rules that get us as close to airtight framework as possible for vetos, which can serve as the inverse of quickbans functionally, then I think we are able to maintain our current system while being strictly better in regards to combatting ladder bots.

I agree that this or something to that effect should be implemented. The fact that a bot managed to inflate the usage of Excadrill to OU status is insane and should be taken seriously. If we were to allow Excadrill to rise, then that would invite more people to inflate the usage of other pokemon with bots and that would be bad for the metagames of lower tiers. I am not really sure what else to say that wasn't better said by Finchinator so I will leave this here for now.

 

[Berks]

I just can't see a world where the veto mechanism doesn't induce a whole bunch of extra headache for everyone involved. It feels too difficult to "assure public trust" even with what most reasonable users would call sufficient because it's so easy to read bad faith into tiering decisions people disagree with.

My question about the bots is, outside of malicious bots that try to timer stall to climb ladder, what do they do that a theoretical shared account doesn't do? If they use good Pokemon well, they climb; if they use good Pokemon poorly, they don't. If they use bad Pokemon well, they might climb, and if they use bad Pokemon poorly, they don't. This is why we weight the stats, no? If triple weather is winning a lot of high level ladder games, maybe we're the ones who should be learning from that!

For the malicious bots that try to timer stall to climb ladder, implement a PS! policy that says bots can't timer stall. Don't do any insane extra work trying to find these bots though; just put a button in the battle UI that says "PS! does not allow bots that timer stall. Click this button if you think this user is a bot that is trying to timer stall you.". Then gather those reports up for a few days, mark the accounts that got reported on X% of the games they booted up (to filter out obvious troll presses), and send what should then be a short list to one of the four people with verification data access once a [time interval]. If there's five checks a week and four are bots, this is light work and we remove the worst bots, yeehaw! If there's 5000 checks a week and four are bots, then we go more drastic; I'm just not convinced this is as much of a problem as it's being made out to be with the ~5 examples of tiering "issues" over however many years we've been doing this.

 

[dk]

the veto thing finch outlined makes plenty of sense. as the ladder finds new ways to make itself worse, it only makes sense that the nebulous mass of unpaid labor that we call smogon bureaucracy grows in power to keep it the madness in check. we already trust tier leaders with guiding bans; this is a very reasonable increase to their powers.

here's a novel idea: end the thread here. lock it. we're 57 posts in; that's enough. we all have things to do, right? i understand talking about playing pokemon is more fun than actually playing pokemon quite often (look at the SPL superstar or whatever thread that instantly accrued 50 billion replies because this site is absolutely fascinated with inventing new ways to circlejerk), but let's just run the veto thing by tiering admins or chaos or whoever has to stamp it instead of unproductively arguing every last minute detail down to the individual grain