1. Welcome to Smogon! Check out the Smogon Starters Hangout for everything you need to know about starting out in the community. Don't forget to introduce yourself in the Introduction and Hangout Thread, too!
  2. Welcome to Smogon Forums! Please take a minute to read the rules.

Potential Flaws with the Current Suspect System

Discussion in 'The Policy Review' started by Tobes, Jun 27, 2011.

  1. Tobes

    Tobes 「Cheezum」「Cripes」
    is a Site Staff Alumnusis a Battle Server Admin Alumnusis a Super Moderator Alumnusis a Community Contributor Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis a Smogon Media Contributor Alumnusis a Past WCoP Winner

    Joined:
    Jul 16, 2010
    Messages:
    2,570
    Decided to jump the gun a bit and post this now.

    My main concern is that I feel after a certain number of consecutive votes for OU status a suspect should not be elligible to be voted on again. Let's take a look at Excadrill. Nomiated 4 times, voted on 4 times, OU 4 times. It's clear that Excadrill is not likely to be banned in the near future, so why is it still being voted on? An unlimited number of votes is just asking for the voter pool to eventually shift to just the right number of voters in favor of banning Excadrill, when all previous votes show for the contrary. There would of course, need to be cases for extenuating circumstances. If Excadrill was blocked from being a suspect and then Drizzle was banned, that would certainly be grounds (pun not intended) for it to be allowed as a suspect again.

    I would hope that this reasoning would also apply for weathers.

    Another concern I have is the allowance of users who have not obtained voting reqs to nominate suspects. They haven't earned the right to vote, so why do they have the right to decide what is voted on? I know that the higher-ups are weeding out the poor nominations, but this still seems like something that should not be allowed to me.

    Discuss away. If you feel the system is fine as is, explain why. If you have other options you wish to discuss, then go for it, but please keep this from being a "current system vs council" thread, because that is not what this is about.
  2. M Dragon

    M Dragon The north wind
    is a Tournament Directoris a Forum Moderatoris a Community Contributoris a Tiering Contributoris a Tutor Alumnusis a Battle Server Moderator Alumnusis Smogon Frontier's Arena Tycoonis the Smogon Tour Season 17 Champion
    Moderator

    Joined:
    Jun 12, 2008
    Messages:
    1,569
    I agree with you.
    Right now, the current suspect system seems like "vote a pokemon suspect n number of times until it gets voted Uber".

    If a pokemon is voted OU several times, it shouldn't be a suspect again, unless a big change happens in the metagame, or if there is a really good reason to make that pokemon suspect again (new broken set discovered, like SD yache chomp in DP or something like that, etc).
  3. Kevin Garrett

    Kevin Garrett is a competitor
    is a Tutor Alumnusis a Tournament Director Alumnusis a Site Staff Alumnusis an Artist Alumnusis a Super Moderator Alumnusis a Tiering Contributor Alumnusis a Smogon Media Contributor Alumnusis a Battle Server Moderator Alumnusis the Smogon Tour Season 12 Championis a SPL Winner

    Joined:
    Jan 5, 2008
    Messages:
    3,307
    As I said in the OU suspect thread, if a suspect is unable to achieve two simple majorities, it should not be elligible to be a suspect unless one of its major checks or counters is banned or the mechanics change from a new game. I think it would remove subjectivity from the process. As it stands now, people can keep nominating whatever they feel like and get some bandwagon votes. If the community repeatedly votes the same thing, that should be it unless something changes within the metagame that directly impacts the suspect.
  4. franky

    franky
    is a Team Rater Alumnusis a Forum Moderator Alumnusis a Contributor Alumnus

    Joined:
    Apr 12, 2009
    Messages:
    2,486
    Metagame shifts.
  5. Haunter

    Haunter 100% avocado
    is an official Team Rateris a Battle Server Administratoris a Community Contributoris a Tiering Contributoris an Administratoris a Tutor Alumnus
    Líder máximo

    Joined:
    Aug 6, 2007
    Messages:
    5,923
    Where are these huge metagame shifts between the last suspect test and the current one? Seriously, somebody please explain me how Garchomp's ban is supposed to affect the status of Exadrill or Drizzle. I completely agree with KG's proposal.
  6. Smith

    Smith is a 90's bitch
    is a Team Rater Alumnus

    Joined:
    Nov 23, 2009
    Messages:
    1,341
    I agree with franky in that Excadrill absolutely has the right to be voted on every single round. Every month the metagame shifts in incredibly unpredictable ways, and we have no idea how banning certain things will impact his performance. For example, we might think that the banning of Thundurus (if it were to occur) would not impact Excadrill's performance- but if Quagsire or Scarf Politoed usage were to suddenly drop, then Excadrill might be a lot more anti-metagame. We can't pretend to understand a beast as complex as the OU metagame well enough to automatically foresee how banning certain things will affect our metagame, and our suspect voters should not have to decide every round if we can vote on Excadrill again. If the people upstairs tell the voters that they can't vote on something "just cause" then we have defeated the purpose of having a public suspect process completely.

    Another thing we should do is post our nominations in a "blind voting" type scenario, or perhaps submit them by PM (although reach will probably nix that). If anybody remembers a while back to when reuniclus was damn near uber, I think the only person who was concerned with him before the nominations thread was Jabba and that everybody kind of hopped on the bandwagon (although this certainly has the potential to be inaccurate so if I simply have my facts wrong here, please tell me). It would also prevent people from just posting 'BAN THUNDURUS' and potentially influencing a voter who somehow hadn't seen a lot of thundurus in his ladder run. If you have laddered up to 1450 then you should have an opinion about every suspect, and if you don't then Abstain from that suspect instead of just listening to every other person that posts in the suspect nominations thread who feels entitled to an opinion. By association I do agree that non-voters should not be able to nominate suspects.
  7. Oglemi

    Oglemi gimme
    is a Tournament Directoris a member of the Site Staffis a Community Contributoris a Pokemon Researcheris a Contributor to Smogonis a Smogon Media Contributoris an Administratoris a Tiering Contributor Alumnus
    C&C Leader

    Joined:
    Oct 13, 2009
    Messages:
    9,075
    Having blind nominations isn't that bad of an idea tbh

    Though I mostly agree with KG's and Haunter's sentiments.
  8. Mario With Lasers

    Mario With Lasers Self-proclaimed DEAD king
    is a Forum Moderator Alumnusis a CAP Contributor Alumnus

    Joined:
    Nov 15, 2006
    Messages:
    7,303
    I think non-voters should have the right to nominate, but either a) make those nominations blind or b) why the hell can people post "ban thundurus" and have it count as a nomination ?__? We could have something similar to UU paragraphs/sentences, in which people will have to elaborate as to why they're nominating a certain element of the metagame if they aren't going to vote in that Round.
  9. JabbaTheGriffin

    JabbaTheGriffin Stormblessed
    is a Tutor Alumnusis a Super Moderator Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis a Smogon Media Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    Jan 21, 2007
    Messages:
    3,045
    Voting on a mon numerous times probably isn't as dangerous as it seems on paper (just look at the Excadrill votes since R1). But I definitely think nominations for these mons should be held to a higher standard and people should have to explain why, regardless of past votes for OU, that mon deserves to be voted on again. I definitely think it'd be bad to drop a blanket statement that something can't be voted on again if it was voted OU a few times, but at the same time I do agree that it shouldn't be voted on every round unless there's good reason to vote on it again (like franky said, "metagame shifts").

    As for the second "flaw," I don't see how this is a problem. Generally, what's voted on would be voted on regardless of who does the nominating. Maybe if one round all the nominations from users who didn't qualify contained Pikachu, I'd see the problem. But even then, it wouldn't do any harm because the voters would undoubtedly not ban it (well maybe Pikachu is a bad example...so strong). Now in a banhappy community, I could see something like that being a problem. Where no matter what the general voter pool sees in front of them, they just put a ban next to it. However, this is clearly not the case in this community. Smith brings up when people bandwagoned on my Reuniclus nomination (which isn't really the case, but let's just go with it). That round, reuniclus received an absolutely pitiful vote (iirc, it received the least Uber votes). I'm personally a fan of allowing maximum community involvement in the suspect testing process. The best way to do that is to allow for nominations from everyone. As long as it does no tangible harm to the suspect testing process (which it hasn't/won't), I see no reason to restrict nominations to only qualified voters.
  10. Fireburn

    Fireburn BARN ALL
    is a member of the Site Staffis a Forum Moderatoris a Community Contributoris a Tiering Contributoris a Contributor to Smogon
    Ubers Co-Leader

    Joined:
    Jan 18, 2009
    Messages:
    2,280
    I agree with Jabba's sentiments, I'm really not seeing how these are problems as the community has honestly shown the contrary.
  11. Seven Deadly Sins

    Seven Deadly Sins ~hallelujah~
    is a Site Staff Alumnusis a Forum Moderator Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis a Smogon Media Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    May 29, 2008
    Messages:
    4,267
    This.
  12. Tobes

    Tobes 「Cheezum」「Cripes」
    is a Site Staff Alumnusis a Battle Server Admin Alumnusis a Super Moderator Alumnusis a Community Contributor Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis a Smogon Media Contributor Alumnusis a Past WCoP Winner

    Joined:
    Jul 16, 2010
    Messages:
    2,570
    Ok, fair enough, Jabba. I do see your point on non-voter nominations.

    Paragraphs combined with blind nominations would defintely be a step in the right direction in my opinion, because it does help to curb bandwagoning, which I think we can agree is one of the bigger concerns for suspect testing.

    This also leads me back to one of my other concerns though. We're attempting to reach a period of No Suspects before we end suspect testing, am I correct? (please correct me if I'm wrong) This seems unfeasible to me, since the community is rather split on whether or not weather is broken. A batch of nominations of "No Suspects" is highly unlikely. I think that a certain number of rounds where nothing is banned should also be a condition for ending suspect testing.

    There has also been a lot of discussion in DST about a suspect ladder. I personally oppose this since in my opinion it promotes bias and unneccessary bans, but I'd be interested to hear the arguments of its proponents.
  13. Kevin Garrett

    Kevin Garrett is a competitor
    is a Tutor Alumnusis a Tournament Director Alumnusis a Site Staff Alumnusis an Artist Alumnusis a Super Moderator Alumnusis a Tiering Contributor Alumnusis a Smogon Media Contributor Alumnusis a Battle Server Moderator Alumnusis the Smogon Tour Season 12 Championis a SPL Winner

    Joined:
    Jan 5, 2008
    Messages:
    3,307
    I have always been a supporter of having paragraphs in the suspect process. Paragraphs have a negative connotation because some people think that means you need to write an essay. I think one or two well constructed paragraphs would be sufficient for what we're trying to achieve.
  14. shrang

    shrang Go to your room
    is a Contributor to Smogonis a Battle Server Moderatoris a Tiering Contributor Alumnus

    Joined:
    Jul 24, 2009
    Messages:
    4,376
    I agree with this. As long as you can get your point across a few sentences or paragraphs, I see nothing wrong with having paragraphs. Maybe do like a "Under 250 words per suspect, explain your vote" limit? It would make sure that the voters can actually show a reason on why they are voting what they are voting instead of just going purely on instinct or bandwagoning. The biggest problem with some of our voters at the moment is not the ability to play the metagame, but to be able to logically explain why they think X is broken or not, and this begs the question of if they understand the concept at all.
  15. mien

    mien
    is a Contributor to Smogonis a Forum Moderator Alumnusis a Researcher Alumnus

    Joined:
    Jul 13, 2008
    Messages:
    732
    I believe the main problem with paragraphs is that one will always subconsciously consider those who support the judge's own ideas of higher quality than those who oppose

    The addition of paragraphs could give the judge the power to change the outcome of the vote towards his own liking.
  16. undisputed

    undisputed
    is a Tutor Alumnusis a Tiering Contributor Alumnusis a Battle Server Moderator Alumnusis Smogon Frontier's Frontier Lackeyis a Past WCoP Winner

    Joined:
    Aug 16, 2009
    Messages:
    2,119
    well theres a panel of 5 people running the process, so i doubt all of them are gonna gang up on some user just to keep them out. i think its pretty obvious to tell when someone knows what theyre talking about
  17. Smith

    Smith is a 90's bitch
    is a Team Rater Alumnus

    Joined:
    Nov 23, 2009
    Messages:
    1,341
    Literally my only concern with paragraphs would be those with bad English because it isn't their first language or because they are just bad writers (or young ones). If you can't write well, then you oftentimes sound stupid even if you know what you're talking about. Otherwise I'm a huge proponent of paragraphs.

    EDIT: Although I suppose simply bearing this in mind eliminates any harm that can cause. So I guess this is just a support post.
  18. reachzero

    reachzero the pastor of disaster
    is a Super Moderator Alumnusis a CAP Contributor Alumnusis a Tiering Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    Oct 18, 2008
    Messages:
    1,524
    I have always been a very outspoken supporter of paragraphs, and I still believe it would help the process. In terms of judging potential bias, I feel it would be valuable to officially publish all the paragraphs submitted, whether accepted or rejected. That would allow the community at large to hold the evaluators accountable. It terms of multiple nominations, I feel that if there is substantial variation in usage over time, it could make a difference. For instance, when BW first came out, Balloon Jolly Excadrill was the most commonly used Excadrill; now Life Orb Adamant Excadrill has significantly eclipsed it. At any rate, I agree that a metagame aspect that has multiple OU votes should be held to much, much higher standards in terms of nominations.

    With regard to concern for non-English speakers, I don't really think it'll be a big problem. We have seen logical, cogent paragraphs submitted by non-native English speakers before (remember DarkLucario on the Salamence Council), and we're aren't grading based on grammar and diction anyway.
  19. JabbaTheGriffin

    JabbaTheGriffin Stormblessed
    is a Tutor Alumnusis a Super Moderator Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis a Smogon Media Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    Jan 21, 2007
    Messages:
    3,045
    Yeah I think paragraphs are great. You cut the voter group down to people who are not only good, but clearly know the implications of their vote and can tell you how their votes will affect the metagame and etc.

    I just don't like the idea of forcing even more work upon players. They already do enough work as it is (well maybe not "enough" but paragraphs are a ton of work on top of what they already do). I guess it comes down to what you think is better: the best voter pool possible, or simply a good/alright voter pool that requires less of the players. I'm personally a fan of the former.
  20. jrrrrrrr

    jrrrrrrr wubwubwub
    is a Tiering Contributoris a Forum Moderator Alumnusis a Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    May 23, 2006
    Messages:
    3,171
    The reason why we stopped liking paragraphs was because everyone bitched if their vote got rejected. There were a small handful of people reading these essays and all they got in return was crap. So nobody liked writing the paragraphs and nobody liked judging them. I really don't see that changing if we go back to it, and that's ignoring the obvious potential for bias in the system.

    With that said, I do like the idea of blind nominations. It prevents the bandwagon effect and makes the process more objective. We've already taken steps to reduce the impact of banhappy users which still exist in huge numbers by allowing mons to be unbanned, by making voting requirements tougher (although I still miss SEXP) and by making a majority of people nominate suspects instead of having one dedicated suspect per round.

    I also like the idea of preventing things from being voted on if they're consistently voted OU. If something is voted OU twice in a row then it should not be eligible for one round.

    Whatever we do, please let us not go back to a council.

    I just want to point out that while this might show that the community has been less banhappy as of late, it also shows that bandwagon nominations still happen and it's not even hard to accomplish. I think this thread is more about addressing potential disaster situations rather than saying that there is currently a problem.

    Can you please elaborate
  21. shrang

    shrang Go to your room
    is a Contributor to Smogonis a Battle Server Moderatoris a Tiering Contributor Alumnus

    Joined:
    Jul 24, 2009
    Messages:
    4,376
    Well, with a panel of five readers (just the people who are leading the OU nominations), the chance for bias is severely reduced. Face it, if all five of the readers are rejecting your paragraphs, chances are that you wrote a load of crap and you don't deserve to vote anyway. Just like the votes for the Pokemon themselves, you could probably put in a "supermajority" case, where the vote would be rejected if and only if >2/3 of the panel rejected it (in this case 4/5).

    In short, if a person's vote got rejected, it won't be because they're vote is prejudiced against or whatever, but because it was clear by the paragraphs that they had no idea what they were on about.
  22. jrrrrrrr

    jrrrrrrr wubwubwub
    is a Tiering Contributoris a Forum Moderator Alumnusis a Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    May 23, 2006
    Messages:
    3,171
    I'm aware of that but that won't stop accusations of bias, and there's no way to prove it one way or the other. That's why I only made a brief mention of the possibility of bias, the main thought behind my post was that we already tried forcing voters to write paragraphs and we didn't like it. Hearing Jumpman's horror stories made me cringe every time.
  23. Haunter

    Haunter 100% avocado
    is an official Team Rateris a Battle Server Administratoris a Community Contributoris a Tiering Contributoris an Administratoris a Tutor Alumnus
    Líder máximo

    Joined:
    Aug 6, 2007
    Messages:
    5,923
    This is absolutely correct. And expanding on this point: the accusation of bias we might receive after introducing paragraphs could be even heavier considering that some of the users who would be judging them aren't exactly neutral to some controversial Pokemon\abilities currently deemed "suspect".

    In my opinion the voting process should be 100% objective and only subject to the achievement of a given rating.

    As for blind nomination, I'm fine with those but still, if a Pokemon\ability gets voted OU twice in a row people should not be able to nominate it again, unless in the previous round we ban something that can potentially have a huge impact on the status of that Pokemon\ability.
  24. Chou Toshio

    Chou Toshio @Fighting Necktie
    is an Artist Alumnusis a Forum Moderator Alumnusis a Community Contributor Alumnusis a Contributor Alumnusis a Smogon Media Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    Aug 16, 2007
    Messages:
    8,251
    This is just one point I wanted to get at but Tobes-- I think you have a misunderstanding of what an "ideal voter" is. Ideal voter =/= bad ass battler.

    I think this is made obvious by the early attempts at 4th gen by Jumpman to do tiering by things like bold voting and paragraph writing.

    The ideal voter (participant in tiering decision) is "someone with a deep understanding of the metagame, an understanding of the general consensus on suspects, has a vested interest in/concern for the tier, cares about Smogon as a site and truly wants to make Smogon's OU the best standard tier it can be, and finally has the intelligence to vote based on these motives."

    ...unfortunately, it was quickly made apparent that there was no easy way to figure out who exactly were these ideal voters-- bold voting or paragraph writing were way too subjective.

    Since we can't figure out who the ideal voters are, we just use the "voting reqs" to give us a pool that will be a rough estimate, and at least likely to contain many "ideal voters."

    That said, this obviously means many good users who would be good voters are inevitably not included. What should truly be tested in the process is the Pokemon, not the players. Because of this background, and the need to at least potentially give posters who can make an intelligent and convincing argument some footing, the process of open nominations weened by the tiering leaders has been established, and I think it's a good system.

    So yeah, I completely disagree with removing it.
  25. khz

    khz

    Joined:
    May 20, 2010
    Messages:
    264
    The only thing is I agree with the previous sentiments that its very hard to predict the metagame to the extent of saying "banning X won't impact on Y's tiering."

    Actually to be honest I'm not convinced that given enough votes a suspect will eventually be voted Uber. If something is borderline OU (say 45%) then its fair enough to keep testing and nominating. But if something is clearly OU (like 20% vote to ban) then if anything "uninformed voters" will hop on the majority bandwagon.

Users Viewing Thread (Users: 0, Guests: 0)