Rejected Suspect Test Qualification Reform

[Rigas]

This feels like we’re chasing solutions for a problem that doesn’t exist.

...

Every time this topic comes up, no one can ever really point to a good reason why we should raise/segregate requirements, other than “well I think so”.

I agree with these statements.

To address another thing in the OP and in some of the replies: those who believe 1750 elo + 80% GXE is "too easy" have a skewed perspective, likely as a result of engaging primarily with the top echelon of Pokemon Showdown players (such as active Smogon tournament players) and not fully grasping that there's a vast, vast playerbase below. I'm not saying this as a condemnation; it's understandable that it happens. But it needs to be kept in mind. I don't see any cause for concern that "unqualified voters" will create trouble. To be clear, I'm not saying that suspect reqs should be achieveable by every single one of the tens of thousands of people on the ladder; I'm saying that someone who reaches 1750 elo + 80% GXE is qualified to have a vote about the metagame.

Plus, while I don't agree with the entirety of the comment an excerpt from which I'm about to quote, I believe this passage is strong:

If someone can consistently hold a high enough elo to be well outside of low ladder, they should have some kind of voice in the tier's rules. You don't have to be a top player to have a stake in the game. Frankly, someone playing at that level has a decent chance of beating a top player in any given game, anyway. Pokemon is swingy like that.

As such, I am not convinced that anyone who can achieve 1750 elo + 80% GXE (I believe 80% GXE is the harder part) should get anything less than a "full" vote, which is what the OP is proposing with the tiered system.

 

[Tree69420]

To address another thing in the OP and in some of the replies: those who believe 1750 elo + 80% GXE is "too easy" have a skewed perspective, likely as a result of engaging primarily with the top echelon of Pokemon Showdown players (such as active Smogon tournament players) and not fully grasping that there's a vast, vast playerbase below. I'm not saying this as a condemnation; it's understandable that it happens. But it needs to be kept in mind. I don't see any cause for concern that "unqualified voters" will create trouble. To be clear, I'm not saying that suspect reqs should be achieveable by every single one of the tens of thousands of people on the ladder; I'm saying that someone who reaches 1750 elo + 80% GXE is qualified to have a vote about the metagame.

Even though someone who can reach 1750 + 80% GXE may be in the top 0.1% of all players who play, doesn't make them necessarily qualified to vote on the metagame. For example, someone who goes 1-2 at a major in smash is probably in the top 1% and would easily beat the vast majority of players, but they still don't understand the metagame in comparison to professional players who can consistently place top 64, and they clearly shouldn't have the same level of authority to vote.

However, I believe the current status quo is ultimately the best option, as increasing the difficulty more would hurt the time-sink more than it would filter out unqualified players imo. Giving reqs to top tournament players would be positive but not that impactful of a change.

Different tiers of votes doesn't feel like a good change to me because it just rewards people who have time to nolife ladder, and tournament reqs having more weight seems also unfair to people who simply didn't have the opportunity to get these reqs.

 

[Blizihguh]

I'm going to pitch the OP's proposal in a slightly different way. I hope this will make more sense to people.

We set reqs with the idea that, if someone does well consistently on ladder, they must understand the metagame. But what consititutes "enough understanding" is subjective! And for for that matter, so is "what should be banned." Any hard cutoff you set is a value judgment, and it's imprecise. Saying "people under this line don't understand the metagame, people over this line do" is going to get some people wrong in either direction. It's also much much simpler and easier to stomach than trying to decide subjectively, which is why we do it. But the OP's proposal softens the cutoff somewhat, which has benefits for people on both sides of the issue.

Many people think that reqs are too easy, over-representing low-level players. Likewise, many people think reqs are too hard, over-representing the players who are most willing to grind for them. By removing the hard cutoff between "no vote" and "vote," we can alleviate both of those concerns. We can give one vote to people who play the game basically competently, by setting the lowest level of reqs lower than they're normally set. For players who play the game very well (say, to a "normal" level of reqs), we give two votes. For players who play extremely well (much better than typical reqs), we give three votes. For players who don't want to grind, it's much easier to stomach "I only get 1 vote" than "I don't get any vote." You can recognize that you're good enough at the game to have a say, but not dedicated enough to grind for 3 votes. On the other hand, for players who think reqs are too easy, the best players' voices now count more. How can you complain about bad players ruining the vote when good players count 2-3 times as much?

If there's not a big discrepancy between how the different tiers vote, it doesn't make a difference to the outcome, but it gives everybody a way to feel better about it. If there is a big discrepancy, then it might affect the outcome, if there are many more voters in the lower brackets -- but this already happens in our current system sometimes (see FayaWizard's post). This is the most contentious point, but I think this is a fair outcome. If the people who play a game overwhelmingly feel that something should go, then it should probably go, even if the top players think it's manageable.

We're playing this game for fun, not for any kind of real stakes. What really matters with these votes is investment. Someone who's more invested in the metagame should have their voice matter more than someone who doesn't care that much. But if a lot of players are invested, and they overwhelmingly feel a certain way -- why should a few players who care an inordinate amount get to overrule them? The goal of suspects is to make the game the best it can be, not to make it as satisfying as possible to the top 0.1% of players. A game that's better to the top 0.1% of players, and worse for 99.9% of people, is not the best it can be.

I have some thoughts about other posts but I'm just gonna omit them, I feel like this says what I want to say anyway. I genuinely think the main issue with this proposal is that we would probably have to rework TC, but I'm nowhere near qualified to figure out how that should work.

 

[Tree69420]

We're playing this game for fun, not for any kind of real stakes. What really matters with these votes is investment. Someone who's more invested in the metagame should have their voice matter more than someone who doesn't care that much. But if a lot of players are invested, and they overwhelmingly feel a certain way -- why should a few players who care an inordinate amount get to overrule them? The goal of suspects is to make the game the best it can be, not to make it as satisfying as possible to the top 0.1% of players. A game that's better to the top 0.1% of players, and worse for 99.9% of people, is not the best it can be.

I have some thoughts about other posts but I'm just gonna omit them, I feel like this says what I want to say anyway. I genuinely think the main issue with this proposal is that we would probably have to rework TC, but I'm nowhere near qualified to figure out how that should work.

I'm not necessarily against the idea that players who are less informed about/skilled in the metagame should not have an ability to vote, however I view it like this. Imagine you are a governmental body trying to work towards solving the issue of climate change. The vast majority of people, including regular citizens as well as climate scientists, have the same end goal, that being solving or reversing climate change. However, not everyone should have the same ability to vote regarding this. An ordinary citizen, though well meaning, may be very vocal about ineffective solutions peddled by the oil and gas industries, such as the idea that every person must be personally responsible regarding their carbon footprint. Asking climate scientists, even less dedicated ones, will likely result in a more effective solution to the problem than asking ordinary people. In this way, voting requirements are akin to a test you must take, with say 100 questions, to prove that you understand the problem. However, it is possible to pass this test despite not understanding the root of the problem, just by memorizing answers. By increasing the test length, it will be more effective at weeding out people who are less informed, but at the cost of many people who are well-informed but cannot sit through such a tedious test.

Similarly, the top players may have differing opinions on exact solutions to improving the meta, but they are much more likely to be able to diagnose the problem and present a solution than the average player. I heavily disagree with the idea that a metagame can be great for the top 0.01% but bad for the other 99.99%, as a good metagame, with diversity in sets, pokemon, and team styles, will be a good and healthy metagame regardless of how good you are at the game. A 1500s player may be well-meaning, but ultimately incapable of analyzing the metagame and improving it, which is why we only open votes to the top percentage of players who can achieve voting requirements. Brushing off metagame concerns as "just for fun" misunderstands the end goal of providing a healthy, balanced, and fun metagame for all players, regardless of skill level.

If we had a real way to figure out how much skill/understanding someone had of the game, then different tiers of voting could possibly work, but as increasing the difficulty of tests much more so increases the tediousness, this solution is not very effective in my opinion. Solutions such as giving each user only 1 account could potentially solve this issue.

 

[Ren]

idk how to say this and im largely retired now but i heavily dislike any proposal that says "people can vote but some people's votes matter more than others". all votes should be equal if people care enough to earn the right to cast one and discussion + a bigtime player's opinion is enough additional sway, use your words to change minds and the outcomes will follow. or they wont and thats still fine because thats how group decision making works. no opinion on any proposal that changes or adds ways for people to earn a vote, suspect laddering is bad for life havers

 

[FlamPoke]

I think the OP by Nat has good intent on making the Smogon tiering process more effective. There are also some good dissenting opinions about not wanting to make suspects more grindy in nature, which I can respect, but I think the general premise of improving the tiering process is a good one and the way to move forward with that seems pretty clear to me.

Providing the casual ladder player an opportunity to impact Smogon tiering via ladder reqs was a good decision overall, but not having a mechanism to emphasize the less-casual tournament player's opinion is doing a disservice to the site, I believe. I won't get into the actual requirements of the suspect in depth as that isn't relevant to this discussion, though I will just voice my opinion that ladder reqs are probably too easy across the site as a whole. What is more important to discuss here is that tournament players should have more influence on tiering by virtue of being involved in the tier at a higher level than the casual player.

I also won't touch on why surveys are probably more bad than good... but the surveys do split opinions into two groups: a general player base and a qualified player base. The fact that councils separate the two is proof of the belief that people who play tournaments are likely more inclined to understand the tier at a nuanced level, which should be a concept most people can at least understand if not align with. Placing an emphasis on their opinions to impact tiering decisions, such as what mons are worth suspecting in the first place, but not giving any sort of respect to their knowledge without requiring them to also get ladder reqs manually is redundant.

I think providing reqs to players who had good showings in recent tournaments is a direct improvement to the tiering process and does very little to hinder the casual player from still having input into the suspect. The list by Greybaum showing how few tournament players actually participate in suspect laddering should prove beyond a reasonable doubt that some of the most involved, influential players aren't really making a difference on the tier because it's just a redundant task to ladder for a tier you already have to be super involved with for the tournament scene.

And lastly, the point Nat made in the OP; having multiple tiers of qualification for ladder reqs. It still requires the grind but incentivizes people willing to slave away on ladder to go further and get more out of the suspect process by demonstrating their capabilities with a higher elo/gxe peak. I think this is an interesting approach to providing more opportunities for people who are good to influence tiering, but have no clue how to define how many tiers / what elo/gxe gaps need to be / etc.

tldr
Should reqs be harder? In my opinion, yes.
Should tournament players be given reqs based on being "qualified"? In my opinion, yes, but I don't really know how to quantify "qualified" besides w/l or placement most likely (like surveys do).
Is Nat's initial suggestion of having tier-based ladder reqs worth pursuing? Maybe, as it does offer a little more proof of your capabilities at the cost of having to grind ladder harder which is going to still make the tiering process in favor of those with time > skill, though it also requires a hard skill check that current reqs kinda don't. I'd want two tiers personally, maybe like 1750/80 and 1900/85.

 

[Giagantic]

This isn't some official response but...

Reqs should be left alone, you can fiddle with the Coil all you want but the process of getting a vote is perfectly fine. If you can't be bothered to go through with this process you simply don't have the right to vote. No one should be able to avoid this, you ladder or win a suspect tour to earn the requirements to vote. If you can't be bothered to go get the reqs, too bad, you clearly didn't care enough to put in a bit of effort.

Skipping going through this process is absolutely a terrible idea. It creates a clique centred around whatever metric you use to get said status.

Furthermore there is no good argument from my perspective in which you can justify one vote having more weight then another, this is simply antithetical to a democratic voting system. Every vote is equal is a fundamental aspect to this system. Doesn't matter if you are the greatest player to ever walk this earth, or the worst, if you went and got the reqs, then your vote is the same.

 

[termi]

Furthermore there is no good argument from my perspective in which you can justify one vote having more weight then another, this is simply antithetical to a democratic voting system. Every vote is equal is a fundamental aspect to this system. Doesn't matter if you are the greatest player to ever walk this earth, or the worst, if you went and got the reqs, then your vote is the same.

There are perfectly legitimate reasons to oppose weighted voting but let's not pretend Smogon is a democracy. You and pretty much everyone on this site agrees that you are not automatically bestowed the right to vote simply by being a member of the community but that you need to put in some degree of effort to show that you are invested enough in the metagame and have enough of a grasp of it to come to an informed judgment. This is antithetical to actual democracy, where being a member of the community (i.e. a citizen) does automatically give you the right to vote and your interest in or knowledge about the political system is completely irrelevant. Having a voting system =/= democracy.

Any discussion regarding the parameters of suspect tests should center around utility-based reasoning. While recognizing that it is impossible to objectively define what constitutes an informed opinion, let alone create a system that only takes such informed opinions into account, we should aim to create a system that maximizes engagement in the tiering process by those people who are more likely to have an informed opinion (i.e. better players) and thereby we hopefully create more desirable tiering outcomes. At the same time, we should be careful not to create a system that is too elitist, as it may have undesirable side effects (e.g. reducing the pool of voters to such a degree that it is no longer representative, or alienating players from Smogon and the tiering process entirely due to perceived elitism).

With that being said, I think it should be recognized that suspect ladders are undesirable as the sole mechanism by which to gain the right to vote (irrespective of what reqs you set), but it is also undesirable to do away with them. I think any system that, as Greybaum shows, discourages engagement from most top players is clearly insufficient. I can only echo what others have already said: suspect laddering is tedious, RNG-based, and teaches you practically nothing useful about the meta. It also will discourage those who have less free time to spend on something they don't even like doing from engaging in the tiering process and that doesn't seem quite right. Suspect tours are one alternative but are clearly not sufficient due to being too sparse in number and too dedicated to particular time slots that again disadvantage those with less free time. Recent (official) tournament results are, I think, one good way to include a bunch of top players in the voting pool that otherwise wouldn't bother. Suspect ladders, however, still remain the way for a greater mass of people to be able to vote while demonstrating at least some level of skill and involvement. The main question lies in what parameters are most optimal for qualifying, whether it be through tournament play or suspect laddering.

 

[Lambovino]

I agree with everything that termi said, I just want to add that some people might not know the state that some lower tiers ladders can be in sometimes. The PU ladder can be pretty dead, and when it's not you still find mostly low rated players that have all or nothing teams (with a higher power level than they do in OU relative to the tier they're in, because of the mons available). It can sometimes be tiring to fight those teams with the limited ressources that the tier offers, or sometimes you could just get lucky and fight the same 2 players for the whole day while having a good MU against them. The variance is huge, which makes it not ideal at all.

And the thing is, you can't really play with the COIL requirements, if you lower it it'll really become a filter on who can spam games and not on who knows the tier enough (which is already a bit the case), and if you make it higher it'll be impossible for classic ladder players to get reqs (it's already almost the case too, we always get only a few ladder only players each time).

The number of people that take the time to go through this trial in PU goes lower and lower, so we should just be stubborn and accept doing suspects with like only 15 voters in the future (if the current trend continues) ?
It's pretty clear that we need more ways to give the right to vote so it's more representative of the playerbase

If you can't be bothered to go get the reqs, too bad, you clearly didn't care enough to put in a bit of effort.

Also what the fuck looool
People don't always have the time to spam games on the ladder, when I have a busier schedule I can't choose to spend a whole day laddering to vote, despite caring a lot about the tier

 

[eeriespell]

This isn't some official response but...

Reqs should be left alone, you can fiddle with the Coil all you want but the process of getting a vote is perfectly fine. If you can't be bothered to go through with this process you simply don't have the right to vote. No one should be able to avoid this, you ladder or win a suspect tour to earn the requirements to vote. If you can't be bothered to go get the reqs, too bad, you clearly didn't care enough to put in a bit of effort.

Skipping going through this process is absolutely a terrible idea. It creates a clique centred around whatever metric you use to get said status.

I hate to be belligerent in threads like these, but this is just wrong. Not everyone has the time to ladder, a lot of us are people who have jobs, friends, families, and other hobbies. Furthermore, especially in regards to low tiers, ladders are not always active, and it's not a large stretch of the imagination to assume that with the variance of timezones and personal schedules around the world that a player's free time would fall in a time frame where they simply can not get a reasonable amount of games on ladder. This has nothing to do with how much they care and everything to do with the fact that, intentionally or not, the system benefits those who have the free time and have that time at the proper hours to sit down at the computer for a few hours and slug it out. Furthermore, while I understand having reservations around it being considered cliquey, especially when Smogon already is a victim of being perceived by the general public as some shadowy cabal that's dedicated to eradicating the concept of fun, I do not think that "these players have played the metagame at the top level of competition we offer, and therefore have earned the right to vote" is even a remotely indefensible position. We already use that metric for tiering decisions in some capacity, surely applying it to current gen isn't that great of a misstep. Additionally, would you still say that these players who have gone through the grind to get to where they are and play in these top level tournaments "don't care"? I fully understand the desire to make sure that voters qualify themselves, but the argument here does feel a bit silly.

 

[Giagantic]

I hate to be belligerent in threads like these, but this is just wrong. Not everyone has the time to ladder, a lot of us are people who have jobs, friends, families, and other hobbies. Furthermore, especially in regards to low tiers, ladders are not always active, and it's not a large stretch of the imagination to assume that with the variance of timezones and personal schedules around the world that a player's free time would fall in a time frame where they simply can not get a reasonable amount of games on ladder. This has nothing to do with how much they care and everything to do with the fact that, intentionally or not, the system benefits those who have the free time and have that time at the proper hours to sit down at the computer for a few hours and slug it out. Furthermore, while I understand having reservations around it being considered cliquey, especially when Smogon already is a victim of being perceived by the general public as some shadowy cabal that's dedicated to eradicating the concept of fun, I do not think that "these players have played the metagame at the top level of competition we offer, and therefore have earned the right to vote" is even a remotely indefensible position. We already use that metric for tiering decisions in some capacity, surely applying it to current gen isn't that great of a misstep. Additionally, would you still say that these players who have gone through the grind to get to where they are and play in these top level tournaments "don't care"? I fully understand the desire to make sure that voters qualify themselves, but the argument here does feel a bit silly.

I am keenly aware of the woes of and problems of our system as OM Leader where numbers regarding suspect tests is a huge topic in our section as player participation varies massively. The thing about reqs is you don't have to do them in one go, this is a fatal flaw that a lot of people (myself included) kinda delude ourselves into thinking which can result in tilting and exhaustion. You can just play a game here and there and generally get reqs within the time frame. This being said, increasing the time allotted to a suspect to enable it so everyone can more easily attain reqs over the course of a long time is a fair argument. The OM section also holds suspect tours and I am a firm believer that these are a fair metric to gauge current metagame awareness and skill whilst bolstering the number of voters.

I admit my wording was harsh, and it wasn't meant to be offensive, more me being overly blunt. If the general consensus is that there needs to be more ways of attaining suspect votes, I wouldn't get in the way of it. This is more my personal opinion as an individual not as Tier Leader of a section.

An idea like making it X far in a official tour gives you a temporary ticket to vote that remains usable for a year (so we avoid people just stocking up and not playing the metagame) is something I wouldn't be opposed too.

This all being said, I am thoroughly against any form of weighting regarding the value of a vote. This is my stance on a personal level and a official level as TL of a section. Votes should always count the same regardless of individual.

 

[FlamPoke]

The OM section also holds suspect tours and I am a firm believer that these are a fair metric to gauge current metagame awareness and skill whilst bolstering the number of voters.

Slightly unrelated to OP, but suspect tours might unequivocally be the worst thing Smogon does for the tiering process. Winning ~1-5 games against people who just happen to be in that PS! room at the time does not prove competency or metagame knowledge. This is mostly an OM gripe because I know some tours give reqs to both finalists regardless of overall turnout, but I know PU does suspect tours too, and despite their generally higher turnout, it's still like 5 games at most you have to win to get reqs.

No clue how winning ~5 games during a live tour demonstrates "current metagame awareness" but doing well in an official tour or subforum PL doesn't...

 

[Giagantic]

Slightly unrelated to OP, but suspect tours might unequivocally be the worst thing Smogon does for the tiering process. Winning ~1-5 games against people who just happen to be in that PS! room at the time does not prove competency or metagame knowledge. This is mostly an OM gripe because I know some tours give reqs to both finalists regardless of overall turnout, but I know PU does suspect tours too, and despite their generally higher turnout, it's still like 5 games at most you have to win to get reqs.

No clue how winning ~5 games during a live tour demonstrates "current metagame awareness" but doing well in an official tour or subforum PL doesn't...

I was moreso against top tour players automatically getting the ability to vote, a system whereby they earn it and it is only temporary is fine by me. Generally Suspect Tours have a fairly high barrier to entry since it is fairly coveted being able to skip grinding ladder by playing 5 or so matches though sometimes as you said they can be really overly easy. A lot of it depends on advertising the suspect tours and the turn out.

 

[Pais]

Speaking, respectfully, in my opinion, about things that should probably change in suspect tests.
The main idea behind my post is to suggest reintroducing tournament requirements as an alternative way to qualify for suspect tests, alongside traditional ladder requirements.

Quoting shiloh's thread of Tiering Policy Framework:

II.) We cater to both ladder players (the higher end of the ladder) and tournament players.

• Tiering actions should not be taken based solely on one group’s experience (ladder or tournament) unless the issue is egregious and clear-cut.
• Stats for both will be highly emphasized, but not a sole determining factor.

III.) The onus of providing justification is on the side of changing the status quo.

• The status quo can be changed in certain cases, such as new game releases.
• If a proposal is made to ban or unban a Pokemon, ability, item, or move, the side suggesting this must demonstrate why this is necessary and how it affects the ladder and the tournament scene, as well as provide evidence for both.
• The argument that removing an element of play does not have a downside is not enough; we need good reasons to take action, not reasons not to do so.

For some reason the Current Gen setting is the only one where the suspects are made only via Ladder Reqs and not also via Tour Reqs.
I think it's time we bring back Tour Reqs alongside Ladder Reqs.
I'm quoting exactly the shiloh's thread because it states that they cather both ladder players and tournament players but in this current scenario I feel like only ladder players are represented and even in this case it's not totally true since Ladder Reqs are way too easy so you just end up going to represent the mid-ladder playerbase.
I think the goal of every suspect should be to have an outcome that really represents the opinion of both tournament and ladder playerbase. It is actually true that accomplished tournament players should be able to take the ladder reqs with ease, but at the same time it is just another tedious process that accomplished players should be able to skip.
On a more subjective note I think I prefer more the evaluation and the opinion of someone which has a certain win rate across Team Tournaments or who made a certain number of points in Stour or who qualified for OLT or Circuit Top 16 etc etc than someone who made 1750 of ELO in ladder which is not a huge accomplishment in terms of Elite Skill in Pokemon.
It's ok to give to less proven players the chance to vote and express their opinions through ladder reqs but it is a bit disrespectful to an established playerbase asking to do 35-40 games in a ladder in which even at 1900 you face sometimes people with questionable gameplay ability and I'm not saying it to sound offensive, but just because I find absurd that the only way to prove that you're worth of voting is reaching 1750 of ELO which is not some crazy accomplishment by any mean.

Something like that was implemented back in the days during Zygarde suspect test in SM and for some crazy reason it was never implemented again:

Announcement Thread 'np: SM OU Suspect Process, Round 6 - A Thousand Miles [Zygarde is now Banned]'

Dec 15, 2018

​

Hey everybody, the OU tiering council has decided to test Zygarde.

Zygarde has been one of the most prominent threats throughout the entirety of the generation. This can be attributed to its bulk, typing, and variety of moveset options, including the infamous thousand arrows. A reliable STAB move with no immunities and scarce resists allows much freedom for...

• ABR
• Replies: 130
• Forum: SM OU

• 

This allows to have major advantages:
1) Having more accomplished voters implies a more accurate outcome in the Suspect Process since likely you have the opinion of both ladder players and tournament players
2) There is an incentive for tournament players to partecipate more in Tiering Actions. This is not about willing to have time to grind the ladder but it is about considering a honor that Tournament Players give their inputs on the tier. The Current Gen is always the flagship tier of Smogon and for this reason we should do our best to preserve a quality also in terms of playerbase when it comes down to suspect tests.

 

[Finchinator]

Council plans on posting a thread about tour reqs after the ongoing suspect; we've been discussing it for a while -- def on the table, just want to do it the right way and handle this suspect first

 

[Finchinator]

While the intent of the OP was good, we are not going to entertain weighted voter reqs. Thanks to everyone for posting.

The council will have a thread (for discussion -- not strict implementation) on the potential for tournament reqs for SV OU in the immediate future.