Improving Suspect Testing

[Evan.]

I know this is probably a touchy subject, especially because suspect testing is such an important part of Smogon and Smogon culture, but I feel like it's time to take a second look at the process and see if we can improve it. The reason I think now is a good time is because all the singles metagames are as stable as they've been since the beginning of ORAS, and if we can figure out a way to improve the system now, it will pave the way for better suspect results leading into Gen VII (also this thread in IS started drifting towards this topic and I thought it would be a good idea to bring it up in here as well).

I think the main problem that people have with the current suspect testing process is that people feel like some voters in suspect tests, frankly aren't qualified enough to make an educated vote towards the future of a tier. Many people (myself included) feel that just "getting reqs" on the ladder, doesn't give a player who has had little to no experience in ___ tier, enough information in order to make an informed decision about the suspect at hand. This is also amplified when you have a test similar to the Victini re-test, where the metagame during the suspect test is different from the normal UU metagame, as an uninformed voter will have no idea whether the re-introduction of a Victini, or the lack of a Mega Pidgeot has improved the tier or made it worse. Ideally, a player will be well-versed in the conversation taking place on the forums as well as having made reqs, but in reality, a large number of the voters have little to no forum presence, and base their opinion solely on their laddering experience. To give an example, in the last UU suspect test, 33 of the 97 voters had under 50 posts on the forums, and if you bump it up to less than 100 posts, 48 of the 97 votes equaled that number. That equates to about 33 and 50 percent of voters, which is an astonishingly high number.

In my opinion, an ideal suspecting process would combine both of the following aspects:

• All the voters in a suspect test know enough about a tier in order to not just have an opinion about whether to ban or keep a pokemon, but to be able to back up that opinion with knowledge of how suspect X affects the tier
• Everyone should feel like they are able to participate in a suspect test, and feel like they have an opportunity to voice their opinion, regardless of their standing in said tier. (this is one of the great things about smogon that we have to make sure is always applicable to any suspect test)

Now obviously these two points can be construed as somewhat opposite to each other, but I feel like there are ways that we can make it happen. I've been talking with a few people and we've put together an idea or two that can at least serve as discussion points moving forward. My idea involves 3 groups of people, the actual council, a group of exemplary posters, and a bunch of ladderers. In this idea, there will be a set limit of voters for each test (probably 50, maybe smaller idk but for this example we're gonna go with 50) and divide it up so that 10 of the voters are the permanent council, the resident TL's would pick a group of 5-10 exemplary posters in the suspect thread (kind of like a rotating council), and the remaining votes would be divided up in an OLT-like fashion, where the top 15 COILs are counted and given reqs for the 1st week of the suspect test, and the top 15 remaining COILs are given reqs for the 2nd week. I feel like this method guarantees that the majority, if not all the voters are educated enough on the tier to where they can make an informed vote, yet it also gives everyone an opportunity to vote, and makes anyone feel like they can be a part of the voting process. Another idea was brought up by my friend Finchinator, and it involves weighting the votes differently for different voters. He posted about it a bit here, but the gist of his post was that votes from established members of the community are worth more than votes from people who are newer. I'm not sure exactly how the weighting would work or who would decide the weighting but I guess that's something we can talk about in the thread.

Again, I feel like now is probably the best time to try and work out a solution to improving suspect testing, since it gives us enough time to properly discuss and sort everything any issues people may have so that we can be ready for the impending arrival of Gen VII with a stronger suspecting system than we had before.

 

[AM]

Tesung, myself, and Mcmeghan agree that the tiering process needs to be changed it's been pretty busy though to keep everything in track. Most of it has been brainstorms so fortunately this has been brought up at the right time.

I like the prospect of this idea and the way tiering is done needs to change that I can definitely say, and soon. The pool of players established in the OP I think is a very legitimate starting point. I guess the logistics of it all number wise can be determined but moving forward with an idea such as this is something I would consider implementing.

 

[Antar]

I'd personally like to see suspect decisions decided at least in part based on some objective metrics. We obviously can't quantify whether a metagame is more "enjoyable" (aside from measuring popularity), but we can look at centralization, playstyle diversity (is X Pokemon keeping stall from being dominant?), competitiveness (how likely is it that a better player will win against a weaker one?) and a few other things. I also really think we need to be doing a better job of performing A/B-style testing. This policy of having suspect ladders replace non-suspect ladders is, I feel, a mistake. What I would advocate instead (and which I've been told is possible by the PS staff) is having it so that someone using a team that's legal in both suspect- and non-suspect metagames would have the option (perhaps default?) of getting the first available battle on either ladder (Slayer95 was working on this, I was told). This would solve any "low activity" concerns (there is not a single official metagame that cannot afford to have its activity cut in half for a month) and allow us to directly compare statistics over a set amount of time.

There's also some stuff I'm working on that should help algorithmically identify potential suspects. I'll make sure to keep people posted.

 

[CyclicCompound]

Antar, while I definitely agree that analyzing simulator data can be a very useful resource, I don't know how far some of those objective metrics you mentioned above can go without making rather subjective decisions about what's good and what's bad. For example, centralization is not always automatically unhealthy, and playstyle diversity brings up a myriad of discussions of what strategies should ideally be viable. With that said, I'm no expert on statistics, and I'd love to be proven wrong if it can be shown that certain trends almost always correlate to a suspect/ban or something like that.

Getting back to the main point of discussion, I think Can-Eh-Dian's proposal sounds quite good. I definitely agree with the sentiment that players who don't usually play the metagame and jump in just for the suspect tests likely don't have as good of a grasp as they should of the metagame. While I'm sure that there are players who have a good enough grasp at the game that they can not only jump into a new meta with a team and win but also immediately identify what's unhealthy (this is mainly referring to people with council/official tournament experience), I'm sure there are lots of voters who aren't quite so knowledgeable and should ideally sit with a tier for a while and really learn it before making these decisions (like myself). Looking at Can-Eh-Dian's proposal, I think the first two groups (council members and exemplary posters) don't really need further qualification because being in either one of those categories essentially proves you really know the tier well. However, I'm still a little iffy on top ladderers.

I think one really good way of proving if you're knowledgeable enough in a meta to vote on its future is not only successfully laddering in said meta but also doing it with a team you built yourself. I know from hanging around IRC and talking with others that it's not uncommon for people to grab a team from someone better and then ladder to the top fairly easily, breezing by the entire lower-to-mid ladder by virtue of using a well-built team. I think if we had some way of verifying that the ladderers at the very least understand their teams well enough to have plausibly constructed it themselves (perhaps by having them submit a short write-up of their team), and have been proven to have an appropriate amount of battling skill by making it to the top, then we can more easily trust that they know the tier well enough to vote.

It's entirely possible that I'm not trusting ladderers enough or that requiring ladderers to write up their teams would be a deterrent or useless, but it's just a suggestion. I don't know if this is true but I heard some people achieved reqs in OU using one of DeNiSsSs's mono-BP teams before the latest BP ban, and I think with some kind of "team verification" we could have justified that the people who achieved reqs with a cheap, outsourced team really couldn't be trusted to make a tiering decision.

Either way, I like Can-Eh-Dian's solution and I think it should really be considered.

 

[ryan]

I have absolutely not read the vast majority of this thread, and I'm not sure when I will get the chance to. However, this bit is funny to me:

It's entirely possible that I'm not trusting ladderers enough or that requiring ladderers to write up their teams would be a deterrent or useless, but it's just a suggestion. I don't know if this is true but I heard some people achieved reqs in OU using one of DeNiSsSs's mono-BP teams before the latest BP ban, and I think with some kind of "team verification" we could have justified that the people who achieved reqs with a cheap, outsourced team really couldn't be trusted to make a tiering decision.

Do you know how often people use teams they built themselves to play in major tournaments? A ton of top tournament players rarely or never build their own teams and instead use teams their friends make them and do successfully with them. If the best of our best don't even build their own teams, why should we require suspect ladderers not to use cheap, outsourced teams? They don't even get the extra knowledge of how to play the team most of the time because lots of those people just rip teams from RMTs and get reqs with them.

Also, how is it a bad thing that people use a team that is considered "cheap" if the team is legal? Why not use BP to get reqs if it's easy to do so? I've used all kinds of "cheap" shit to get reqs because reqs are a retarded, unnecessary grind for anybody with remote competence in competitive Pokemon. Anything to make it easier for the ladderers should absolutely be utilized. -Tsunami- has gotten OU reqs with RU pride. Should he not be able to vote in suspects when he does that? It's "cheap" and sort of outsourced too actually since people tell him what to use lol

 

[CyclicCompound]

Oh, I understand how often that happens in tournaments - and that's totally fine. I mean absolutely no disrespect to anyone who does that in tournaments, nor do I think it's wrong, and I think tournaments are a completely different thing than laddering for reqs. My main concern is that ladder randoms who qualify for reqs with teams they got off of a featured RMT for a tier they don't play can influence the direction of that tier. Like you said, laddering for reqs is a time-intensive and often boring process, but it's not particularly difficult with a good team even if you're not knowledgeable of the tier.

And going by Can-Eh-Dian's proposal, it's very easy for established players like you to qualify for voting without having to go through the grueling process of achieving reqs, especially if you're already a part of whatever community is suspecting something. What I'm proposing is strictly an add-on to what Can-Eh-Dian said, and it's intended to help verify that players who choose to qualify for reqs primarily through laddering rather than the alternatives (be a part of council, make exemplary posts) are actually knowledgeable enough of the tier.

Also, the reason why people shouldn't have been able to use BP to get reqs is that ANYONE could have plausibly achieved reqs with BP without much actual skill or knowledge of the metagame - hence defeating the entire purpose of making people ladder to vote. As I understand it, the point of this thread is to find alternate solutions to (in my opinion) the flawed and inefficient system of laddering for reqs.

(Sorry if this post or my previous one was confusing. I'm having some trouble properly explaining my concerns. Long story short: I think solely laddering for reqs is unnecessary for people obviously fit to make tiering decisions, but not indicative enough of metagame knowledge for people who aren't.)

 

[Oglemi]

This thread has been moved to PR to garner input from a wider audience.

 

[Sturdynips]

I think the main problem that people have with the current suspect testing process is that people feel like some voters in suspect tests, frankly aren't qualified enough to make an educated vote towards the future of a tier. Many people (myself included) feel that just "getting reqs" on the ladder, doesn't give a player who has had little to no experience in ___ tier, enough information in order to make an informed decision about the suspect at hand. This is also amplified when you have a test similar to the Victini re-test, where the metagame during the suspect test is different from the normal UU metagame, as an uninformed voter will have no idea whether the re-introduction of a Victini, or the lack of a Mega Pidgeot has improved the tier or made it worse. Ideally, a player will be well-versed in the conversation taking place on the forums as well as having made reqs, but in reality, a large number of the voters have little to no forum presence, and base their opinion solely on their laddering experience. To give an example, in the last UU suspect test, 33 of the 97 voters had under 50 posts on the forums, and if you bump it up to less than 100 posts, 48 of the 97 votes equaled that number. That equates to about 33 and 50 percent of voters, which is an astonishingly high number.

I don't want to stray too far from the topic at hand, but I feel as though the booming growth of PS is quite a considerable factor in the trend toward less forum-active voters. People are finding PS before Smogon and I can't see how that will change in the future. That said, our current suspect process is designed to make it harder for "uninformed" voters (in this case, voters lacking much forum presence) to vote, as they will have to play more games in order to qualify. I can't see a problem with this; I don't think it's right to use the top players as the benchmark for the rest to be judged and if someone wants to prove they are competent enough to vote by playing twice as many matches to get reqs, then good on them.

My idea involves 3 groups of people, the actual council, a group of exemplary posters, and a bunch of ladderers. In this idea, there will be a set limit of voters for each test (probably 50, maybe smaller idk but for this example we're gonna go with 50) and divide it up so that 10 of the voters are the permanent council, the resident TL's would pick a group of 5-10 exemplary posters in the suspect thread (kind of like a rotating council), and the remaining votes would be divided up in an OLT-like fashion, where the top 15 COILs are counted and given reqs for the 1st week of the suspect test, and the top 15 remaining COILs are given reqs for the 2nd week. I feel like this method guarantees that the majority, if not all the voters are educated enough on the tier to where they can make an informed vote, yet it also gives everyone an opportunity to vote, and makes anyone feel like they can be a part of the voting process.

This idea hinges solely on some arbitrary restriction on the amount of voters, which in my opinion, is the opposite of what we should be looking for. Larger numbers of suspect voters can't possibly be a bad thing if the suspect test process itself is working as it should. To me it looks like this is just elitism for no apparent reason other than some concept that people who have qualified under the system are not "good enough" because they haven't made a post on the forums. Please. Forum presence is a laughable concept at even our top levels of play; we wouldn't stop people entering Stour because they didn't have enough posts, why stop them from having their vote? The standard is set in our tests so that a player must be "competent" to vote. Our current system is fine for this.

 

[YABO]

So it appears that part of the issue brought up here is that a user will not get the proper experience in a metagame where the suspect is both allowed and one where the suspect is disallowed. As mentioned earlier on, this occurred during the UU tests where Pidgeot and Victini were allowed without being able to judge the effect of their possible departure before the vote. In this scenario, is it unreasonable to have 2 suspect ladders? One where the suspect is allowed and the other where the suspect is banned. This lets a potential voter experience both sides of the metagame and would likely spurn better conversation in forum posts as well since it is far easier to compare and contrast with hard experience instead of conjecture. So what I'm saying is, would it be possible to make 2 separate ladders with a coil requirement for each to vote? This may run into an issue with ladder activity but that's something I'm not versed in which is why I'm posting the question here.

 

[Magnemite]

I honestly think that ladder reqs are a terrible method of determining who should be allowed to vote in suspect tests. There's a few reasons for this, but the main one I'd like to focus on is the low quality of the ladder. This may just be a PU specific thing as I don't have much experience on other ladders (though I doubt it based on what I've heard), but the ladder is a horrible representation of the metagame. While playing my 20 games on the ladder as part of council reqs, I faced either of the two suspects a grand total of one time total, and that one time was a mighty max HP Barbaracle, which hardly even counts. However, almost every team I faced had either unviable Pokemon or weird and horrible sets on viable Pokemon, to go along with the fact that there were many other top tier threats I either rarely or never saw. While some of these games were against the lower ladder, there wasn't any noticeable improvement in the quality of opponents as I got higher on the ladder. I've talked to people who laddered for full reqs, and they haven't given me any indication that they got many good opponents either. The other problem with the ladder as a means of determining skill is something which stems directly from this. Laddering for suspect reqs is much less about being a good player who has metagame knowledge and much more about who has the time to actually spend grinding away at the ladder. With such low ladder quality, it isn't hard for any random player to achieve reqs if they have the time and energy to do so.

Unlike others who have posted in the thread, the reason I take issue with this is not so much that this leads to uninformed voters (though this is still a concern), it's the fact that there isn't really anything gained out of the laddering process, especially not for players who are already known to be competent enough to vote. Playing 30-40 low level opponents on the ladder isn't going to give anyone a good idea of how to vote on the suspect, nor is it going to give players who are new to the tier and genuinely want to learn it a valid picture of it.

Now obviously we can't just have entirely subjective reqs, but there are ways of objectively proving oneself that aren't pointless grindfests. The main way I can think of is to rely entirely on suspect tours, but make it so that just doing well across several different suspect tours is enough to qualify, probably with some kind of point system with non-single elimination tours. This way, it's less boring to get reqs, harder for bad players to achieve reqs, easier for good players to do so, and doesn't involve seeing more Gothorita and Meditite than the Pokemon being suspected.

 

[Lemonade]

I think most suspect tests are a bit backwards. See, I would assume everyone on the council plays their tier at a pretty high level (say, tournament level). Therefore, it would make sense for them to be deciding if a Pokemon should be banned or not. However, current councils are deciding what Pokemon should be suspected, and leaving the banning to anyone who qualifies. The issue here is the qualification is ladder-based, which I don't think anyone can say is quite tournament level. I don't see the logic behind this.

Since I say this is backwards, the simple fix is just swap the roles. The ladder qualification lets users say what they think is broken, and the council votes on whether these threats should be banned or not. This makes way more sense to me. Everyone gets to share their experience, but the ultimate decider of what the meta will look like is people good at their tier. Add some rotating council or whatever to catch people who are good / intelligent but not on council or whatever.

One important difference in the discussion with this reversed system is that players can actually share their own experiences. I can't really think of a good way to say this, but basically I've gotten the impression that a lot of experience is thrown out because "it's the ladder" or "you're not thinking about Pokemon X at a high level" or "you haven't used other playstyles so you don't have a good grasp on what this Pokemon really does" (I haven't really read recent np threads so idk if this is still a thing). However, when the general population is deciding what should be suspected, people's experiences sort of combine. For example, Pokemon X might wreck all playstyles. Then, people who play stall, balance, and offense would have all experienced this, and Pokemon X would be brought up for suspect. But maybe Pokemon X only wrecks certain stall builds. Then, only a small proportion of the players would vote for this Mon to be suspected, and even if it made it to council, they would still have the final say. This also eliminates the "well I didn't have any trouble with Pokemon X because my team consisted of 6 checks to it."

So
Pros

• everyone gets to participate and have a say
• the actual decision is made by people who are skilled players and knowledgeable about their tier

Cons

• you (still) have to play a lot of ladder matches
• Smogon (still) gets flak for banning stuff

tags: LC suspect test

Could replace council with suspect tour results and let them nominate suspects and vote ban or no ban, while laddering on lets you nominate suspects. Once again though, this goes back to having the resources to run a suitable amount of suspect tours.

As a last note, you can't accommodate both the ladder and official tournaments with balancing.

 

[Mazinger]

From what I see, the main problem here is that while the quality of voters is easily improved by just going back to council votes, leaving all tiering decisions to a handful of people is way too exclusive for most of us, so we've kinda been going in place trying to find a compromise between quality and quantity of voters. This leaves us with this hybrid system where the very exclusive council decides the direction of tiering while ladder reqs are a nuisance for good players and easily exploited by uninformed voters. This doesn't seem to really satisfy anyone.

I think a better solution than combining small councils with ladder req votes is simply ditching the ladder and significantly expanding the councils, criteria for selection being participation in tournaments, for example playing in wcop/spl/lower tier leagues, reaching a certain round in ost/opens/cups, a point total in tour, playoffs in olt, maybe some easy ones for people who already made it onto the council once. This solves quite a few problems at once; you guarantee that the council members actually compete at the highest level without stopping anyone from qualifying if they put in the effort, you're not just leaving everything up to a few people picked by upper staff, aside from olt the required experience actually comes from tournament play and not the (very different) ladder metagame, vacant spots are a non-issue, and you still have flexibility in how you select suspects, whether it's through the existing small councils or the public as Lemonade. suggested above.

This is actually something I came up with just now at 5 am so it probably has a hole or two but I think it's worth considering.

 

[jpw234]

Just to clarify - is Smogon the biggest competitive Pokemon community in the world, or is it the biggest Pokemon tournament organizing group in the world? Because if it's the latter, I support restricting important tiering decisions to only tour players; but if we do that, we kinda have to drop the former.

 

[Mazinger]

?

I'm not sure if you're talking to me but by that logic smogon would currently be "the biggest Pokemon ladder organizing group in the world", since success in tournaments does nothing for you in terms of tiering decisions. It's not about taking anything away from ladder players, it's about optimizing the tiering process. Unless you go with a completely free vote, you're going to have people left out either way, and my proposed solution doesn't exclude anyone beyond those that don't have the results, which is no different from ladder reqs. Also note that you still have olt as a factor and could include some more seats for proven ladder players.

 

[YABO]

The issue I see with exclusively tournament reqs is that tournament play and ladder play are fundamentally different in what they emphasize. Ladder play is heavily skewed towards building teams that are very consistent and can deal with the vast majority of teams that you will find randomly. Tournament play revolves around short series or even best of 1 in some cases. Because of this, tours are less focused on consistency and more prone to wild trends rising out of nowhere. The question we have to ask is which style is better suited to producing a playable and enjoyable environment, an environment based on consistency or one based around dealing with one match.

 

[dice]

i disagree with your contention. due to the bo1 nature of tournaments in settings like spl, gimmicks and more peculiar sets are commonplace, yes. however, teams that are build without the players' own fundamental sense of 'soundness' are few and far in between. in these tournaments, players with cheese teams that have no semblance of consistency are rare. the best teams and metagame development originate from spl or smogon tour for a reason. some sets are more often catered to your opponent, but there are just minor nuances of counterstyling as opposed to complete desecration of 'sound building'.

 

[nyttyn]

You're never going to get a suspect test system ideal for tournament play by making it based on ladder users, and you're never going to make a suspect system ideal for ladder play by making it based on tournament users.

jpw234 raises a valid point, I think. And I think the current system should be changed to be more inclusive of ladder players, as they comprise the majority of what smogon's actual userbase is. As I see it, the current push toward council and tournament players is extremely flawed, as important as tournaments are, they only represent the game at its highest level and in a considerably different enviroment, not competitive play as a whole. Users have already gone on the fudnamental differences between Ladder and Competitive play, so I won't write a mountain of text on that, but as the two are fundamentally different, suspect tests are inevitably going to cater to either one group or the other. Pokemon that are seen as gamebreaking in tournaments are not necessarily dangerous on the ladder, and vice-versa. As a result, I think the vastly larger playerbase should be catered toward the larger playerbase - and that would be the ladder.

This does not mean I think 6 Eeveeloution teams who can't peak past 1000 should have a say in how the tier is run, and I don't think that Tournament players should be completely disregarded either, but as things stand it seems incredibly foolish to push a system that effects the masses closer to being based upon only a smaller subset of users. In addition, it makes no sense to continue to ramp up requirements while still allowing anyone to vote via ladder requirements, since that still allows ladder players to have a say, regardless of how high those metrics are pushed. As such, I feel that the ladder requirements should simply be lowered to a point that is reasonable for the average 'skilled' player to reach, as the current repeatedly escalating grind only discourages legitimate voters. I don't think 'unskilled' players making these reasonable requirements would be an issue either - as I see it, the system should be 'you care enough about competitive pokemon to showcase a reasonable level of experience with the metagame, you get a say.' Even if you think unskilled players getting reqs is a problem though, they're already 'allegedly' getting them even with requirements being pushed as high as they currently are, these users have a lot of time on their hands (seeing as how the primary userbase of pokemon, competitive or otherwise, is children to teens who have enormous amounts of free time) and will manage to make any requirement that isn't completely unreasonable for even the best of users to make.



Having said my personal opinion on the matter of what would be best for the vision of further suspect tests, by all means, if the intent is to continue pushing toward making Tournaments the only thing that matters for suspect tests, and attempting to completely disregard the ladder as much as possible is a continued desire, suspect votes should entirely be moved toward invite-only councils. You're only going to make both groups upset if you try to cater to both, and so long as ladder players are 'allowed' to vote on things that directly affect them, you're going to continue to have players who manage to BS their way past the requirements. It's been shown time and time again, and even adding paragraphs as an attempt to increase the skill requirement just leads to people regurgitating the arguments and words of others.

The current system only harms both groups. A vision needs to be established - either suspect tests cater toward ladder players who've evolved past the point of understanding Hyper Beam is not the best move in the game, or they cater towards tournament players. Smogon's ladder-only playerbase has simply grown by such an exponential factor that any system that attempts to cater to both is doomed to failure.


That or you could attempt to separate ladder and tournament bans but I don't think such a system would ever be popular, thanks to the barrier of entry it imposes.

 

[BKC]

you guys are taking this ladder vs. tournament thing to a ridiculous extreme. there's no need to choose to cater to only one of them. there's no such thing as a pokemon that you whip out in an important tour that can't also be part of a good ladder team. it's the same game fellas. just because you can't coast with a tour-winning team that was built for a specific opponent thus meaning it may have chosen to overlook some stuff to really pound on other things doesn't mean it isn't. plus you'll rarely go the distance with one team anyway, and you'd be surprised at how much these re-used tour teams still do work, despite these supposed negatives, see olt so far; again, same game.

don't really have an opinion on this thread as a whole but to reiterate, ~ladder versus tournaments~ is not the problem

 

[jpw234]

?

I'm not sure if you're talking to me but by that logic smogon would currently be "the biggest Pokemon ladder organizing group in the world", since success in tournaments does nothing for you in terms of tiering decisions. It's not about taking anything away from ladder players, it's about optimizing the tiering process. Unless you go with a completely free vote, you're going to have people left out either way, and my proposed solution doesn't exclude anyone beyond those that don't have the results, which is no different from ladder reqs. Also note that you still have olt as a factor and could include some more seats for proven ladder players.

I would contend that far more of Smogon's userbase uses the ladder than any of our tournaments, and correspondingly, that achievement on the ladder is a more representative metric for qualification than achievement in tournaments. Again, though, this is dependent on who we are actually catering to. Do we care about the opinion of the general Smogon userbase? Do we care exclusively about the opinion of tournament players?

 

[Mazinger]

I would contend that far more of Smogon's userbase uses the ladder than any of our tournaments, and correspondingly, that achievement on the ladder is a more representative metric for qualification than achievement in tournaments. Again, though, this is dependent on who we are actually catering to. Do we care about the opinion of the general Smogon userbase? Do we care exclusively about the opinion of tournament players?

I obviously can't prove this but I'm pretty sure the vast majority of people with experience in both fields will tell you that tournaments are a much more competitive playing field. There might be more ladder players, so it would make sense to call for their votes if you're not looking for a particular qualification, but suggesting that they're generally more knowledgeable/skilled than tournament players is quite simply not based on reality. If you mean something else by "qualification" then please elaborate. Either way, your orignal claim that we would have to drop the "competitive community part" if we chose to select voters based on tournament play is bogus.

That said, if you don't think we should base our tiers on the highest level of play then that's your opinion and I can't tell you you're wrong, so I agree that the general goal of the tiering process would have to be discussed and agreed on before we move on to a specific system. I was posting under the assumption that we're still strictly doing the "ban what's broken/bad for the metagame" (I would argue that this implies high level play because if something is only a problem for bad players the answer is simply improving your play) thing but after talking to people that apparently isn't the case, so we're gonna need more opinions on what we actually want to achieve, since the only thing that seems like general consensus is that the current system is severely flawed. Some sort of official word would be appreciated too.

Also agreeing w/ BKC

 

[jpw234]

I obviously can't prove this but I'm pretty sure the vast majority of people with experience in both fields will tell you that tournaments are a much more competitive playing field. There might be more ladder players, so it would make sense to call for their votes if you're not looking for a particular qualification, but suggesting that they're generally more knowledgeable/skilled than tournament players is quite simply not based on reality. If you mean something else by "qualification" then please elaborate. Either way, your orignal claim that we would have to drop the "competitive community part" if we chose to select voters based on tournament play is bogus.

Tiering decisions are the most important things that go on on this website, given that they shape play for everyone who uses our system. I would say that people who are locked out of that decision process cannot meaningfully be called members of the community. A system that restricts access to suspect voting to tournament players would, in my opinion, restrict the community to tournament players.

That said, if you don't think we should base our tiers on the highest level of play then that's your opinion and I can't tell you you're wrong, so I agree that the general goal of the tiering process would have to be discussed and agreed on before we move on to a specific system. I was posting under the assumption that we're still strictly doing the "ban what's broken/bad for the metagame" (I would argue that this implies high level play because if something is only a problem for bad players the answer is simply improving your play) thing but after talking to people that apparently isn't the case, so we're gonna need more opinions on what we actually want to achieve, since the only thing that seems like general consensus is that the current system is severely flawed. Some sort of official word would be appreciated too.

Also agreeing w/ BKC

Banning what's broken/bad for the metagame and allowing users who aren't tournament stars to vote is not mutually exclusive? Something that's become aggressively clear during this generation is that ultimately suspect policy is a matter of opinion and there is almost never an objectively correct decision in a suspect. It seems to me that plenty of tournament players held the opinion that Mega-Metagross should have been banned and used the result of the suspect test as evidence that the current suspect system is flawed. That's ultimately an argument that the opinion of tournament players should be what matters in our suspect policy, but let's not sit here and say that that's the only right way to ban what's broken.

 

[Pocket]

if you're a good ladder player, doing decently enough on tours to earn reqs shouldn't be an issue. There are many examples of strong tournament players who were formerly known for their high laddering skills. It's one of those questionable players who are good enough for ez ladder reqs, but bad enough to be eliminated early in tours, that are weeded out, which I find desirable

 

[Mazinger]

Tiering decisions are the most important things that go on on this website, given that they shape play for everyone who uses our system. I would say that people who are locked out of that decision process cannot meaningfully be called members of the community. A system that restricts access to suspect voting to tournament players would, in my opinion, restrict the community to tournament players.

Nobody is locked out; anyone who wants to participate can join the competition, just like it is now. If your suggestion isn't that literally everyone gets a vote, your argument doesn't make any sense unless you consider the current community resitrcted to ladder players, which is what I told you originally.

Banning what's broken/bad for the metagame and allowing users who aren't tournament stars to vote is not mutually exclusive? Something that's become aggressively clear during this generation is that ultimately suspect policy is a matter of opinion and there is almost never an objectively correct decision in a suspect.

Huh? If you're acting within a system that assumes all of this is a matter of opinion anyway, then the logical conclusion of that is that the terms "broken" and "bad for the metagame" don't have any meaning within that system. You pretty much said this yourself in this very paragraph. If, on the other hand, the system tries to eliminate broken threats, then it defines "broken" as some sort of property that has to be discovered, which requires knowledge and understanding of the game, ie skill. Of course you could technically create a system that acknowledges this and still allows random rookies to vote, but why would you ever do that?

It seems to me that plenty of tournament players held the opinion that Mega-Metagross should have been banned and used the result of the suspect test as evidence that the current suspect system is flawed. That's ultimately an argument that the opinion of tournament players should be what matters in our suspect policy, but let's not sit here and say that that's the only right way to ban what's broken.

Considering that I never made any of those claims, I have no idea why you're telling me this. I'm not even sure if you're reading my posts.

 

[jpw234]

Nobody is locked out; anyone who wants to participate can join the competition, just like it is now. If your suggestion isn't that literally everyone gets a vote, your argument doesn't make any sense unless you consider the current community resitrcted to ladder players, which is what I told you originally.

ditching the ladder and significantly expanding the councils, criteria for selection being participation in tournaments, for example playing in wcop/spl/lower tier leagues, reaching a certain round in ost/opens/cups, a point total in tour, playoffs in olt, maybe some easy ones for people who already made it onto the council once.

This was I believe your original proposal, which is a fixed number of players that make a certain spot in certain tournaments. So, unless I've misinterpreted your proposal, not everybody who wants to participate can join.

Huh? If you're acting within a system that assumes all of this is a matter of opinion anyway, then the logical conclusion of that is that the terms "broken" and "bad for the metagame" don't have any meaning within that system. You pretty much said this yourself in this very paragraph. If, on the other hand, the system tries to eliminate broken threats, then it defines "broken" as some sort of property that has to be discovered, which requires knowledge and understanding of the game, ie skill. Of course you could technically create a system that acknowledges this and still allows random rookies to vote, but why would you ever do that?

Sure, brokenness is some property that has to be discovered, but there is no accepted, objective definition of brokenness. Even very skilled players can differ widely in what they believe is broken. If the idea is that restricting the pool of voters to better players is going to more accurately measure what is truly broken you're doomed to fail because even top tournament players will have different opinions.

Considering that I never made any of those claims, I have no idea why you're telling me this. I'm not even sure if you're reading my posts.

Wasn't referring to you specifically with that, just observations I made, specific example I remember was like I believe Clone's post where he was arguing that too many people were able to vote based on the Metagross results.

 

[toshimelonhead]

Suspect testing problems differ amongst the tiers. For ORAS OU, the metagame is as stable as it has ever been. Future suspect testing will require much more judgement compared to prior suspect testing. It didn't take a rocket scientist to figure out Mega Kangaskhan had no business in OU, and that's true of many of the earlier OU suspects. This also made it easier for newer players to earn reqs: spam the suspect. In that case, that's sign of the suspect breaking the metagame, not necessarily the fault of the player. I also want to add that in the most controversial suspect test, Mega Metagross, the "masses" ended up getting this one "right". While SPL players started to reach gentlemen's agreements not to use M-Metagross, the general Smogon community voted to keep Mega Metagross OU. Megagross is still a top tier threat, but the meta has adapted to it quite well despite no new organic introduction of any Megagross counters. I bring this up because the "ladder vs tournament" player debate in my eyes can seem condescending towards ladderers - the vast majority of great tournament players started out as ladder players, and to discredit their judgment in a suspect test creates a huge rift between good tournament level players and good players that might never get a chance to get in the tournament scene. If it's too easy to get reqs, raise them or limit the number of battles to reach reqs.

Other metagames have to handle suspect testing differently. Ubers and Doubles need to do some soul searching if they haven't already to figure out what they want their respective metagames to look like. In this case, I'm not quite sure what suspect testing does. It's not fair to strongman people out of voting because of subjective reasonings to get the result tier leaders want. Nor is it fair for people to delegitimize player's accomplishments in a tier because of "hax".

For lower tiers (UU, RU, NU, and PU), many suspect tests could be avoided altogether by banning drops until holding a live suspect tournament containing the drops. After the tournament, a committee containing both respected players in the tier and those successful in the live tournament should decide whether to have a suspect test, quickban something (by a supermajority), or have no additional suspect testing. The standard suspect test still works in a pinch, but it shouldn't be the first tool to reach for here.

The overall conflict with suspect testing is dealing with an "Animal Farm" style of conflict: whether all votes really matter, or whether all votes matter, but some matter more than others.