Potential Flaws with the Current Suspect System

[M Dragon]

If a suspect is voted OU, it shouldnt be a suspect again unless there is a good reason to nominate it again.
In the same way there is a supermajority for Uber, we could do exactly the same for OU (if it has more than 2/3 of the votes OU it cant be nominated again unless there is a good reason or there is a change that can have a huge impact on the metagame, or something like that)

[panamaxis]

There would of course, need to be cases for extenuating circumstances. If Excadrill was blocked from being a suspect and then Drizzle was banned, that would certainly be grounds (pun not intended) for it to be allowed as a suspect again.

Subjective as to what "extenuating circumstances" entails

If a pokemon is voted OU several times, it shouldn't be a suspect again, unless a big change happens in the metagame, or if there is a really good reason to make that pokemon suspect again (new broken set discovered, like SD yache chomp in DP or something like that, etc).

There's (relatively) big changes in the metagame very often. Who gets to decide if the reason is good enough? Who gets to decide if the metagame change is big enough?

I think we all realise if this was to go ahead there needs to be exceptions (because obviously having a blanket rule is a stupid idea). Problem is it's gonna be completely subjective as to when these exceptions should apply and then we're creating another problem... Who should decide when/if these exceptions should apply? Because its subjective we'd probably have to have some sort of nomination or paragraph process to decide when these 'exceptions' should apply and then you've essentially got the suspect process.......(unless someone has an idea for this?)

Unless someone can address the 'exception' issue I don't see this working anytime soon.

Oh and just for the record I'm a big supporter of blind nominations and paragraph (not essay) requirements. Jabba's paragraph system in UU of a couple of sentences explaining the suspect is a complete breeze compared to the old Jump/Aeolus paragraph system and I think this system would be good in OU.

[Tobes]

@ Chou Toshio
These are good points, although I had already conceded on this matter :P

@Discussion in general

So so far the consensus seems to be make nominations blind and place a very small restriction (as in one round where they can't be nominated) on repeat suspects. Perhaps permanently banning nominations as a suspect isn't the right way to go, but I do feel we need to replace some restrictions on nominations. Does anybody have any complaints or disagreements with these ideas?

[reachzero]

Regarding those that believe that there can be no significant evidence of bias in the review of paragraphs, I've already noted that I'm in favor of publishing all paragraphs, both accepted and rejected, immediately subsequent to each vote. That would give pretty clear evidence to all concerned exactly why each vote that was rejected was rejected.

[Destiny Warrior]

I support the paragraphs/sentences system, conditional to them being revealed publicly at <whatever time fixed>. It increases the transparency, and I think it would be possible for one of the 5 people to compile them all together and post them(also makes sure people can show they know the metagame). Essays are obviously a no-no since you're starting to get a bit exclusive at that point. Paras/sentences like in UU seem to be an ideal middle ground to me.

[AlphaJolt]

I also support paragraphs/sentences because it shows that somebody actually understands the metagame and can explain why they vote the way they do. Publishing them is a good idea also because then people can bitch less when they are rejected. It will give people an idea of "what makes a godo paragraph." I think that the paragraphs will encourage critical thinking in the suspect test process.

[Eternal]

Originally Posted by Fat AlphaJolt

I also support paragraphs/sentences because it shows that somebody actually understands the metagame and can explain why they vote the way they do. Publishing them is a good idea also because then people can bitch less when they are rejected. It will give people an idea of "what makes a godo paragraph." I think that the paragraphs will encourage critical thinking in the suspect test process.

Not necessarily. Paragraphs do allow us to understand the reasons behind a players decision, but there are only so little a person can put in a paragraph or two. For example, many people can argue that Garchomp is simply Uber because of Sand Veil combined with its great stats. While others may confront this with Garchomp's "slow" speed in this metagame along with a x4 weakness to Ice. These are valid points when voting with these decisions. Are we going to neglect anyone's vote if they believe that Garchomp should be Uber just because they mentioned how great it's Atk and Speed stats are and that it can sweep a team with Swords Dance without mentioning other important factors like Sand Veil, bulk, typing, 6.25% damage from SR, actual damage calculations to back up claims, etc? Well you can, but even if you're denying this person's vote, then that's questioning the person's ability to write up a valid reason (while the person may have the right notion as to why the Pokemon should be banned from laddering experience). The point-based requirement on the ladder already allows the user to vote. Someone might state certain claims as to why a Pokemon like Garchomp is Uber in a paragraph or two, but may forget other reasons because you can't really put all your reasoning in a paragraph or two. While if you want a full explanation, then that can be as large as a typical essay (and that is something we want to avoid for obvious reasons). Most suspect decisions are controversial, if they weren't, then everyone's (those that meet the requirement) vote should portray the same result.

But we do want our suspect voters to have logical reasons behind their vote. I would like to propose something different other than paragraphs. Why not have a debate thread like those of the suspect discussion thread, but only with users that have met the point-based requirement. The debate thread will feature the Pokemon that has been nominated for ban. Everyone that has met the point-based requirement must have at least one post with substantial information on the debate thread explaining their standpoint of the suspected Pokemon. And they are allowed to debate with others and their opinions. The debate thread will allow all users to come to valid decision before deciding to vote; to gain more input on the speculations behind other players that have also met the requirement.

[B-Lulz]

I actually think the debate idea Eternal just made is brilliant, it would sort out people who know what they are talking about to those who don't and maybe show people in the suspect thread how to not fuck up a thread.

Also no, I am wholeheartedly against people who haven't made reqs making suspect nominations. why should they have a say when they are going to have nothing to do with it anyway, the only reason I can see it being there is to give a sense of inclusiveness, and look where that got the 'np:' threads.

[M Dragon]

Originally Posted by Fat panamaxis

There's (relatively) big changes in the metagame very often. Who gets to decide if the reason is good enough? Who gets to decide if the metagame change is big enough?

What I meant, is that if someone wants to nominate again a pokemon that was voted OU, he should give a good reason for it.
Also, I like Eternal idea.

[franky]

While Eternal's proposal sounds civilized and positive in theory, I'd imagine it to be time-consuming because the players would never draw a conclusion due to the amount of subjectivity that occurs in discussions like these. Since everyone must post at least once, this is essentially like Jabba's system in the UU thread except players are allowed to have a rebuttal for their arguments. It's a great idea but the idea itself might push the round further because I'd imagine this type of discussion to be chaotic considering the amount of players qualified to vote nowadays. Should there be a deadline to these discussion, a lot of these debates might be incomplete leaving the player's arguments to ban something unfinished. I'm up for something like this but without the ability to debate with others- sorta like Jabba's system.

[animenagai]

Is posting a paragraph or 2 really that difficult? I understand that you can be a great player without being the most expressive/eloquent user, but if you think something should be completely removed from the metagame, I think it's reasonable for you to give a certain amount of justification for it. Just explain the viable sets and explain how it affects the metagame. Seems quite simple to me.

[Stellar]

New users frequently ask for links to threads containing discussion about recently banned suspects. A thread containing past paragraph submissions would serve as a good resource for these users.

[FlareBlitz]

Regarding repeat suspects: I don't think it's a good idea to prevent a suspect from being nominated multiple times, especially not so on an arbitrary basis like "has the metagame changed enough". There are quite a few reasons why a suspect would be deemed OU one round and Uber/BL the next: the suspect might have been overshadowed by other suspects, the suspect might be highly controversial and have even subtle metagame shifts push it over, a new pool of voters might have an entirely different sense of perspective on the suspect, etc. I do agree that repeat suspects should require more thorough explanations as to why they deserve to be suspects though.

I also love the debate thread idea. Not only will it separate people who know what they're talking about from people who do not, it may also convince some people to change their position on the suspect entirely, like what happened over IRC during the final suspect test in 4th gen UU.

[Haunter**]

I don't think it's a good idea to prevent a suspect from being nominated multiple times, especially not so on an arbitrary basis like "has the metagame changed enough"

Sorry but I don't really get the logic behind this (I'm not referring to you in particular FlareBlitz). So deciding when the metagame changes is an arbitrary choice, while allowing people to vote only after their reasons have been evaluated by a restricted number of potentially biased judges isn't?

[panamaxis]

What I meant, is that if someone wants to nominate again a pokemon that was voted OU, he should give a good reason for it.

Isn't this what we already do? Unless you mean...

repeat suspects should require more thorough explanations as to why they deserve to be suspects though.

Which I agree with. Having nominations for previously OU suspects held to a higher standard could potentially be a good middle ground. Higher standards could be stuff like explaining why <pokemon> is more banworthy this round then last round and why this is (what metagame changes have led to this for example).

@Haunter: Reach's idea of publicising all the paragraphs with clear reasons as to why they were rejected significantly lessens the notion of paragraphs as a mechanism to arbitrarily deny people voting rights.

[Haunter**]

If you think you can solve the arbitrariness problems just by making public the decisional process adopted by reach and co. about voting paragraphs, then the same logic should also apply to decisions about "relevant metagame shifts" that should be the condicio sine qua non to allow things voted OU twice (or more) in a row, to be nominated again at the end of every round. We can publicly discuss and democratically adopt a decision about whether or not we believe that the metagame has changed enough between a suspect round and another and then act accordingly about the nominations.

[FlareBlitz]

Originally Posted by Fat Haunter

Sorry but I don't really get the logic behind this (I'm not referring to you in particular FlareBlitz). So deciding when the metagame changes is an arbitrary choice, while allowing people to vote only after their reasons have been evaluated by a restricted number of potentially biased judges isn't?

Much about our suspect process is arbitrary, including the rating threshold. My argument against the "determine when the metagame changes" rule isn't so much based on the fact that it's arbitrary with reference to some ideal suspect testing standard, it's based on the fact that it's arbitrary with reference to its measurement. That is, it's hard to put a number on the metagame itself.

Your comparison to the paragraph system is valid on principle, but not so much when we actually look at the degree of subjectivity involved. A "good" paragraph and a "bad" paragraph are usually pretty clear-cut, at least based on my experiences in the UU nomination thread. Good paragraphs offer reasons that a suspect is broken based on the characteristics of a desirable metagame, while bad paragraphs do not. Perhaps there are paragraphs that are kind of in-between, in which case some subjectivity is required. I'm not a tiering lead, so I do not know how common such a case is.
But regardless of how subjective even the most borderline paragraph is, analyzing a metagame change is far more so. Would an increase in usage of 0.4% be significant enough to warrant re-testing? What about 0.3%? Or 0.5%? Would the change of popularity of a set (say, SD versus choice) be sufficient? And it's not just about the raw data, it's also about the interpretation of that data. What if a counter's usage drops? Would that mean the metagame is less prepared for a suspect due to other threats that require attention or that the suspect is no longer threatening due to the reduced preparation?

In order to make such a system workable, we would either need a very convoluted and detailed system of standards that would require constant revision, or we would need to rely on an opaque system of simply deciding whether the metagame has changed or not every three months for every possible repeat suspect. Neither of those two options would make the suspect process better in my opinion.

But I want to briefly touch on what may be a larger issue here. Ultimately, the entire point of all our other suspect qualification systems is to ensure that voters know what they're doing. If voters decide to nominate a suspect, and then vote it out, that should be that. If we need to introduce further protections such that our voters cannot nominate certain Pokemon, perhaps we need to fundamentally revise our estimation of the competence of our voting pool. I have no idea how we would begin going about this, but if our real problem is that voters make stupid decisions, shouldn't we prevent them from even being voters? And if we do feel that most of our voters are competent, then we should just let their votes speak for whether the metagame has changed enough to nominate and subsequently ban a suspect.

I hope that made sense :)

Originally Posted by Fat panamaxis

Which I agree with. Having nominations for previously OU suspects held to a higher standard could potentially be a good middle ground. Higher standards could be stuff like explaining why <pokemon> is more banworthy this round then last round and why this is (what metagame changes have led to this for example).

Yes, that's what I meant. Obviously we don't want suspects re-nominated ad infinitum because it becomes somewhat of a "guilty until proven innocent" type deal. However, opinions can and do change, and incorporating this into the existing paragraph system seems like the best way to go.

[M Dragon]

Originally Posted by Fat FlareBlitz

Much about our suspect process is arbitrary, including the rating threshold. My argument against the "determine when the metagame changes" rule isn't so much based on the fact that it's arbitrary with reference to some ideal suspect testing standard, it's based on the fact that it's arbitrary with reference to its measurement. That is, it's hard to put a number on the metagame itself.

Nobody said that.
The fact that a pokemon previously voted OU pokemon could be nominated every round times until it's voted uber (and after that, it has very little chance of coming back to OU) is a big problem in our current suspect system.

What I was suggesting was that if a pokemon gets voted OU (or supermajority OU lol), it shouldnt be nominated again, unless the person who wants to nominate the suspect gives a good and strong reason to nominate it again. Why would we suspect a pokemon again if we already voted it OU? Thats what the person that wants to nominate the pokemon again should explain us.

Im not saying "readmit a pokemon as a suspect if the metagame changes" or something like that.

Also, a discussion thread with the people that has voting rights (as Eternal suggested) would be a great idea.

[FlareBlitz]

Originally Posted by Fat M Dragon

Nobody said that.

My understanding was that Haunter and others were arguing for that position. See:

Originally Posted by Fat Haunter

As for blind nomination, I'm fine with those but still, if a Pokemon\ability gets voted OU twice in a row people should not be able to nominate it again, unless in the previous round we ban something that can potentially have a huge impact on the status of that Pokemon\ability.

My apologies if I misconstrued their position.

Originally Posted by Fat M Dragon

The fact that a pokemon previously voted OU pokemon could be nominated every round times until it's voted uber (and after that, it has very little chance of coming back to OU) is a big problem in our current suspect system.

Why?
As I said above, if we're confident in the competence of our suspect voters, it shouldn't matter if a Pokemon is nominated every round until the end of 5th gen, as its tiering status should still be reflective of what the community thinks its performance is like in the metagame. In fact, preventing the community from nominating elements that are divisive is a quick way to alienate a large proportion of its members.
I do agree that nominations for repeat suspects should be held to a higher standard, simply in order to separate the bandwagoners from people who know what they're doing, but I don't see a need to institute more drastic measures.

[M Dragon]

Originally Posted by Fat FlareBlitz

Why?
As I said above, if we're confident in the competence of our suspect voters, it shouldn't matter if a Pokemon is nominated every round until the end of 5th gen, as its tiering status should still be reflective of what the community thinks its performance is like in the metagame. In fact, preventing the community from nominating elements that are divisive is a quick way to alienate a large proportion of its members.
I do agree that nominations for repeat suspects should be held to a higher standard, simply in order to separate the bandwagoners from people who know what they're doing, but I don't see a need to institute more drastic measures.

People who know what they're doing should have no problems at all to explain why that pokemon should be a suspect again.
And thats the easiest way of "separating the bandwagoners from people who know what they're doing".

[FlareBlitz]

Originally Posted by Fat M Dragon

People who know what they're doing should have no problems at all to explain why that pokemon should be a suspect again.
And thats the easiest way of "separating the bandwagoners from people who know what they're doing".

I agree entirely. I mentioned that I would expect the nominations for repeat suspects would be held to a higher standard. Thing is, that's more or less the system we have now (at least in UU). My understanding is that the proposed changes are about a ban on repeat suspects until "the metagame changes".

[jrrrrrrr]

I have one question for the people who are saying that we should go back to paragraphs for everything- even ignoring the fact that it would be biased to some degree and would make more work for the staff, how do you answer critics of the suspect system who say "we hardly ever have enough voters to make a legitimate poll"? This UU test will have less than 25 eligible voters when you take away the alts (down from 31), I don't imagine that would improve if voters were required to write paragraphs that could reject their month or so of rating boosting. The OU tests would probably share a similar fate if we did this.

[JabbaTheGriffin*]

"Quality over quantity"

no one complained about the effect paragraphs had on the size of the voter pool last gen since at least it produces the most qualified voter pool possible (they did complain about having to write paragraphs though!)

[jrrrrrrr]

^Although that was before we had voting pools of 9 and 19 users on suspect test bans that impact the metagame for everyone, which is where the complaints really started. If we're going to opt out of a stable metagame for constant suspect testing I think we should make an effort to get more people involved, not less. It also helps curtail the effects of bandwagon votes and nominations. A pool of 80 voters might be enough to stop a bandwagon vote but 40 might not be enough. I think it would help if we set some kind of expectations of how many voters we want and then work on making testing policy from that, instead of just going back to the paragraph system that has failed us miserably already. I just feel like we've already had this exact conversation in PR before and it didn't end well.

OU Tests have been consistently around 50 voters this gen with the exception of the most recent one which was around 70. UU has had 31 and <25. LC has had 12, 15 and 17 voters in their three rounds. Are these numbers "good enough" or should we be looking for something else?

[Dubulous]

Perhaps I could be some help to this thread, perhaps not.

Vader and I have been in charge of the Suspect testing for Little Cup this generation, and in our three rounds we've experimented a bit with our requirements to see what works best for us. Our nomination process hasn't changed; it is the same process that you see used in both OU and UU. In my opinion this process works fine, however, it has to be up to those running the process to decide which suspect nominations hold water and which do not. Jabba hinted at this with Excadrill early on in this thread. After being nominated three times in a row, I would be much tougher in accepting nominations.

In our first two rounds, we used paragraphs solely to decide our voting pool. Paragraphs are ideal because they force a user to explain their reasoning, which in itself is a good practice while also serving as a check against ladder cheating. Several users have spoken about potential bias by the Moderators in regard to evaluating paragraphs. I disagree, with the notion (though it may be naive) that the Moderators in question are in the position they are for a multitude of reasons, one of which is that we can trust them to evaluate circumstances objectively and without bias. It is their job not to be biased, and if they can't do that then they don't deserve the position, in my opinion. I do not believe poor english skills were a real problem: when reading paragraphs we tried to look at what they were trying to say rather than what they were actually saying. Our problems were with users that were high up on the ladder but for various reasons had paragraphs that were somewhat lacking. Additionally, we had an issue with users plagiarizing what had been discussed in the threads in their paragraphs. This, ultimately, led us away from using paragraphs at all.

We switched to rating requirements for the next round, using the top 15+15 method first seen in OU voting. It worked well, but for the current round we decided to switch to a hard and fast rating requirement. After the round, Vader and I will evaluate the results from both rounds and make a decision as to which method we will use in the future.

In terms of the number of voters we've had each round, I'd say we're right where we want to be. Unfortunately, Little Cup doesn't quite have the playerbase that OU and UU have, which makes the low number more appropriate. Personally, I don't think that Suspect leaders should have a ballpark number of voters in mind, but rather, an eye to have only those voters that deserve to have a vote.