Potential Flaws with the Current Suspect System

[Tobes]

Decided to jump the gun a bit and post this now.

My main concern is that I feel after a certain number of consecutive votes for OU status a suspect should not be elligible to be voted on again. Let's take a look at Excadrill. Nomiated 4 times, voted on 4 times, OU 4 times. It's clear that Excadrill is not likely to be banned in the near future, so why is it still being voted on? An unlimited number of votes is just asking for the voter pool to eventually shift to just the right number of voters in favor of banning Excadrill, when all previous votes show for the contrary. There would of course, need to be cases for extenuating circumstances. If Excadrill was blocked from being a suspect and then Drizzle was banned, that would certainly be grounds (pun not intended) for it to be allowed as a suspect again.

I would hope that this reasoning would also apply for weathers.

Another concern I have is the allowance of users who have not obtained voting reqs to nominate suspects. They haven't earned the right to vote, so why do they have the right to decide what is voted on? I know that the higher-ups are weeding out the poor nominations, but this still seems like something that should not be allowed to me.

Discuss away. If you feel the system is fine as is, explain why. If you have other options you wish to discuss, then go for it, but please keep this from being a "current system vs council" thread, because that is not what this is about.

[M Dragon]

I agree with you.
Right now, the current suspect system seems like "vote a pokemon suspect n number of times until it gets voted Uber".

If a pokemon is voted OU several times, it shouldn't be a suspect again, unless a big change happens in the metagame, or if there is a really good reason to make that pokemon suspect again (new broken set discovered, like SD yache chomp in DP or something like that, etc).

[Kevin Garrett]

As I said in the OU suspect thread, if a suspect is unable to achieve two simple majorities, it should not be elligible to be a suspect unless one of its major checks or counters is banned or the mechanics change from a new game. I think it would remove subjectivity from the process. As it stands now, people can keep nominating whatever they feel like and get some bandwagon votes. If the community repeatedly votes the same thing, that should be it unless something changes within the metagame that directly impacts the suspect.

[franky]

Let's take a look at Excadrill. Nomiated 4 times, voted on 4 times, OU 4 times. It's clear that Excadrill is not likely to be banned in the near future, so why is it still being voted on?

Metagame shifts.

[Haunter**]

Where are these huge metagame shifts between the last suspect test and the current one? Seriously, somebody please explain me how Garchomp's ban is supposed to affect the status of Exadrill or Drizzle. I completely agree with KG's proposal.

[Smith]

I agree with franky in that Excadrill absolutely has the right to be voted on every single round. Every month the metagame shifts in incredibly unpredictable ways, and we have no idea how banning certain things will impact his performance. For example, we might think that the banning of Thundurus (if it were to occur) would not impact Excadrill's performance- but if Quagsire or Scarf Politoed usage were to suddenly drop, then Excadrill might be a lot more anti-metagame. We can't pretend to understand a beast as complex as the OU metagame well enough to automatically foresee how banning certain things will affect our metagame, and our suspect voters should not have to decide every round if we can vote on Excadrill again. If the people upstairs tell the voters that they can't vote on something "just cause" then we have defeated the purpose of having a public suspect process completely.

Another thing we should do is post our nominations in a "blind voting" type scenario, or perhaps submit them by PM (although reach will probably nix that). If anybody remembers a while back to when reuniclus was damn near uber, I think the only person who was concerned with him before the nominations thread was Jabba and that everybody kind of hopped on the bandwagon (although this certainly has the potential to be inaccurate so if I simply have my facts wrong here, please tell me). It would also prevent people from just posting 'BAN THUNDURUS' and potentially influencing a voter who somehow hadn't seen a lot of thundurus in his ladder run. If you have laddered up to 1450 then you should have an opinion about every suspect, and if you don't then Abstain from that suspect instead of just listening to every other person that posts in the suspect nominations thread who feels entitled to an opinion. By association I do agree that non-voters should not be able to nominate suspects.

[Oglemi*]

Having blind nominations isn't that bad of an idea tbh

Though I mostly agree with KG's and Haunter's sentiments.

[Mario With Lasers]

I think non-voters should have the right to nominate, but either a) make those nominations blind or b) why the hell can people post "ban thundurus" and have it count as a nomination ?__? We could have something similar to UU paragraphs/sentences, in which people will have to elaborate as to why they're nominating a certain element of the metagame if they aren't going to vote in that Round.

[JabbaTheGriffin*]

Voting on a mon numerous times probably isn't as dangerous as it seems on paper (just look at the Excadrill votes since R1). But I definitely think nominations for these mons should be held to a higher standard and people should have to explain why, regardless of past votes for OU, that mon deserves to be voted on again. I definitely think it'd be bad to drop a blanket statement that something can't be voted on again if it was voted OU a few times, but at the same time I do agree that it shouldn't be voted on every round unless there's good reason to vote on it again (like franky said, "metagame shifts").

As for the second "flaw," I don't see how this is a problem. Generally, what's voted on would be voted on regardless of who does the nominating. Maybe if one round all the nominations from users who didn't qualify contained Pikachu, I'd see the problem. But even then, it wouldn't do any harm because the voters would undoubtedly not ban it (well maybe Pikachu is a bad example...so strong). Now in a banhappy community, I could see something like that being a problem. Where no matter what the general voter pool sees in front of them, they just put a ban next to it. However, this is clearly not the case in this community. Smith brings up when people bandwagoned on my Reuniclus nomination (which isn't really the case, but let's just go with it). That round, reuniclus received an absolutely pitiful vote (iirc, it received the least Uber votes). I'm personally a fan of allowing maximum community involvement in the suspect testing process. The best way to do that is to allow for nominations from everyone. As long as it does no tangible harm to the suspect testing process (which it hasn't/won't), I see no reason to restrict nominations to only qualified voters.

[Fireburn]

I agree with Jabba's sentiments, I'm really not seeing how these are problems as the community has honestly shown the contrary.

[Seven Deadly Sins]

Originally Posted by Fat Fireburn

I agree with Jabba's sentiments, I'm really not seeing how these are problems as the community has honestly shown the contrary.

This.

[Tobes]

Ok, fair enough, Jabba. I do see your point on non-voter nominations.

Paragraphs combined with blind nominations would defintely be a step in the right direction in my opinion, because it does help to curb bandwagoning, which I think we can agree is one of the bigger concerns for suspect testing.

This also leads me back to one of my other concerns though. We're attempting to reach a period of No Suspects before we end suspect testing, am I correct? (please correct me if I'm wrong) This seems unfeasible to me, since the community is rather split on whether or not weather is broken. A batch of nominations of "No Suspects" is highly unlikely. I think that a certain number of rounds where nothing is banned should also be a condition for ending suspect testing.

There has also been a lot of discussion in DST about a suspect ladder. I personally oppose this since in my opinion it promotes bias and unneccessary bans, but I'd be interested to hear the arguments of its proponents.

[Kevin Garrett]

I have always been a supporter of having paragraphs in the suspect process. Paragraphs have a negative connotation because some people think that means you need to write an essay. I think one or two well constructed paragraphs would be sufficient for what we're trying to achieve.

[shrang]

Originally Posted by Fat Kevin Garrett

I have always been a supporter of having paragraphs in the suspect process. Paragraphs have a negative connotation because some people think that means you need to write an essay. I think one or two well constructed paragraphs would be sufficient for what we're trying to achieve.

I agree with this. As long as you can get your point across a few sentences or paragraphs, I see nothing wrong with having paragraphs. Maybe do like a "Under 250 words per suspect, explain your vote" limit? It would make sure that the voters can actually show a reason on why they are voting what they are voting instead of just going purely on instinct or bandwagoning. The biggest problem with some of our voters at the moment is not the ability to play the metagame, but to be able to logically explain why they think X is broken or not, and this begs the question of if they understand the concept at all.

[mien]

I believe the main problem with paragraphs is that one will always subconsciously consider those who support the judge's own ideas of higher quality than those who oppose

The addition of paragraphs could give the judge the power to change the outcome of the vote towards his own liking.

[undisputed]

well theres a panel of 5 people running the process, so i doubt all of them are gonna gang up on some user just to keep them out. i think its pretty obvious to tell when someone knows what theyre talking about

[Smith]

Literally my only concern with paragraphs would be those with bad English because it isn't their first language or because they are just bad writers (or young ones). If you can't write well, then you oftentimes sound stupid even if you know what you're talking about. Otherwise I'm a huge proponent of paragraphs.

EDIT: Although I suppose simply bearing this in mind eliminates any harm that can cause. So I guess this is just a support post.

[reachzero]

I have always been a very outspoken supporter of paragraphs, and I still believe it would help the process. In terms of judging potential bias, I feel it would be valuable to officially publish all the paragraphs submitted, whether accepted or rejected. That would allow the community at large to hold the evaluators accountable. It terms of multiple nominations, I feel that if there is substantial variation in usage over time, it could make a difference. For instance, when BW first came out, Balloon Jolly Excadrill was the most commonly used Excadrill; now Life Orb Adamant Excadrill has significantly eclipsed it. At any rate, I agree that a metagame aspect that has multiple OU votes should be held to much, much higher standards in terms of nominations.

With regard to concern for non-English speakers, I don't really think it'll be a big problem. We have seen logical, cogent paragraphs submitted by non-native English speakers before (remember DarkLucario on the Salamence Council), and we're aren't grading based on grammar and diction anyway.

[JabbaTheGriffin*]

Yeah I think paragraphs are great. You cut the voter group down to people who are not only good, but clearly know the implications of their vote and can tell you how their votes will affect the metagame and etc.

I just don't like the idea of forcing even more work upon players. They already do enough work as it is (well maybe not "enough" but paragraphs are a ton of work on top of what they already do). I guess it comes down to what you think is better: the best voter pool possible, or simply a good/alright voter pool that requires less of the players. I'm personally a fan of the former.

[jrrrrrrr]

The reason why we stopped liking paragraphs was because everyone bitched if their vote got rejected. There were a small handful of people reading these essays and all they got in return was crap. So nobody liked writing the paragraphs and nobody liked judging them. I really don't see that changing if we go back to it, and that's ignoring the obvious potential for bias in the system.

With that said, I do like the idea of blind nominations. It prevents the bandwagon effect and makes the process more objective. We've already taken steps to reduce the impact of banhappy users which still exist in huge numbers by allowing mons to be unbanned, by making voting requirements tougher (although I still miss SEXP) and by making a majority of people nominate suspects instead of having one dedicated suspect per round.

I also like the idea of preventing things from being voted on if they're consistently voted OU. If something is voted OU twice in a row then it should not be eligible for one round.

Whatever we do, please let us not go back to a council.

Originally Posted by Fat JabbaTheGriffin

Smith brings up when people bandwagoned on my Reuniclus nomination (which isn't really the case, but let's just go with it). That round, reuniclus received an absolutely pitiful vote (iirc, it received the least Uber votes).

Originally Posted by Fat Fireburn

I agree with Jabba's sentiments, I'm really not seeing how these are problems as the community has honestly shown the contrary.

I just want to point out that while this might show that the community has been less banhappy as of late, it also shows that bandwagon nominations still happen and it's not even hard to accomplish. I think this thread is more about addressing potential disaster situations rather than saying that there is currently a problem.

Originally Posted by Fat Seven Deadly Sins

This.

Can you please elaborate

[shrang]

The reason why we stopped liking paragraphs was because everyone bitched if their vote got rejected. There were a small handful of people reading these essays and all they got in return was crap. So nobody liked writing the paragraphs and nobody liked judging them. I really don't see that changing if we go back to it, and that's ignoring the obvious potential for bias in the system.

Well, with a panel of five readers (just the people who are leading the OU nominations), the chance for bias is severely reduced. Face it, if all five of the readers are rejecting your paragraphs, chances are that you wrote a load of crap and you don't deserve to vote anyway. Just like the votes for the Pokemon themselves, you could probably put in a "supermajority" case, where the vote would be rejected if and only if >2/3 of the panel rejected it (in this case 4/5).

In short, if a person's vote got rejected, it won't be because they're vote is prejudiced against or whatever, but because it was clear by the paragraphs that they had no idea what they were on about.

[jrrrrrrr]

I'm aware of that but that won't stop accusations of bias, and there's no way to prove it one way or the other. That's why I only made a brief mention of the possibility of bias, the main thought behind my post was that we already tried forcing voters to write paragraphs and we didn't like it. Hearing Jumpman's horror stories made me cringe every time.

[Haunter**]

Originally Posted by Fat jrrrrrrr

I'm aware of that but that won't stop accusations of bias, and there's no way to prove it one way or the other.

This is absolutely correct. And expanding on this point: the accusation of bias we might receive after introducing paragraphs could be even heavier considering that some of the users who would be judging them aren't exactly neutral to some controversial Pokemon\abilities currently deemed "suspect".

In my opinion the voting process should be 100% objective and only subject to the achievement of a given rating.

As for blind nomination, I'm fine with those but still, if a Pokemon\ability gets voted OU twice in a row people should not be able to nominate it again, unless in the previous round we ban something that can potentially have a huge impact on the status of that Pokemon\ability.

[Chou Toshio]

Originally Posted by Fat Tobes

Another concern I have is the allowance of users who have not obtained voting reqs to nominate suspects. They haven't earned the right to vote, so why do they have the right to decide what is voted on? I know that the higher-ups are weeding out the poor nominations, but this still seems like something that should not be allowed to me.

This is just one point I wanted to get at but Tobes-- I think you have a misunderstanding of what an "ideal voter" is. Ideal voter =/= bad ass battler.

I think this is made obvious by the early attempts at 4th gen by Jumpman to do tiering by things like bold voting and paragraph writing.

The ideal voter (participant in tiering decision) is "someone with a deep understanding of the metagame, an understanding of the general consensus on suspects, has a vested interest in/concern for the tier, cares about Smogon as a site and truly wants to make Smogon's OU the best standard tier it can be, and finally has the intelligence to vote based on these motives."

...unfortunately, it was quickly made apparent that there was no easy way to figure out who exactly were these ideal voters-- bold voting or paragraph writing were way too subjective.

Since we can't figure out who the ideal voters are, we just use the "voting reqs" to give us a pool that will be a rough estimate, and at least likely to contain many "ideal voters."

That said, this obviously means many good users who would be good voters are inevitably not included. What should truly be tested in the process is the Pokemon, not the players. Because of this background, and the need to at least potentially give posters who can make an intelligent and convincing argument some footing, the process of open nominations weened by the tiering leaders has been established, and I think it's a good system.

So yeah, I completely disagree with removing it.

[khz]

Originally Posted by Fat Haunter

As for blind nomination, I'm fine with those but still, if a Pokemon\ability gets voted OU twice in a row people should not be able to nominate it again, unless in the previous round we ban something that can potentially have a huge impact on the status of that Pokemon\ability.

The only thing is I agree with the previous sentiments that its very hard to predict the metagame to the extent of saying "banning X won't impact on Y's tiering."

Actually to be honest I'm not convinced that given enough votes a suspect will eventually be voted Uber. If something is borderline OU (say 45%) then its fair enough to keep testing and nominating. But if something is clearly OU (like 20% vote to ban) then if anything "uninformed voters" will hop on the majority bandwagon.