questioning the process itself

[alex]

kd24, while I agree with the premise that the committee should largely be the best and most knowledgeable, insightful battlers (the last two points are key), the problem I have with what you suggested is that you are saying that the top 15 people on the leaderboard are the top 15 battlers which just isn't at all accurate. the people in charge of the process - not the ladder - should decide the committee. will there be overlap? maybe, and they could even use the ladder as a potential guide when appropriate, but it should be purely up to their discretion.

even if the ladder were a perfect representation of skill, which it will never be, it's still not enough to only take the most skilled; with a committee like this, you need an appropriate balance of playing ability and ability to make thoughtful, logical, intelligent, substantiated arguments, and we need to make sure that we trust the people in charge to make the calls on who those people are

 

[cantab]

Yeah, I completely agree that a concrete ruleset is important. I'd like to hear the opinions of those who think otherwise, to be honest.

The problem I see is that if we say the tiers are "set in stone" barring GameFreak additions or clear "emergencies", we then have to make sure we get them right first time. That tends to make the whole process take longer. Will a council necessarily help things? And if the final vote is 8-7, should a concrete ruleset that won't be looked at again for months if not years be made based on that vote? (That is assuming you intend the council to vote, and decide things on a simple majority. You could expect supermajorities, or even unanimity, but either of those will readily lead to deadlocks).

If it is decided at some point that the tiers are fine - that nothing in OU remotely meets any Uber characteristic, and everything in Uber we know will be too powerful in OU, then indeed there is no need for testing. If that happens, great. But I don't think we should force it. I don't think "a concrete ruleset is important" is a good enough reason to have perfectly balanced Pokemon banned, or worse overpowered Pokemon remain present.

And indeed, some take the view that changes to tiering are a good thing because they promote change and head off stagnation of the metagame. (This is unlikely to be an issue with a young generation though.) Some people don't want a concrete ruleset. Obviously we don't want it changing every two weeks. But UU has been quite happy to have three or four tier changes per year, indefinitely - why would it be bad if OU experienced similar? I'm not saying we should go banning Salamence and Scizor and letting in Manaphy and Deoxys-D solely to "shake up the meta", but if we have reasonable suspicion a Pokemon is in the wrong tier we should do something about it.

Allowing some members into the council by ladder ranking is a very reasonable way to do things. There are a couple of flaws. The present system means that achieving a high ladder ranking requires skills totally unrelated to actual battling - things like using alts for testing teams, battling enough (but not too much) per day, and so on. But that may change with Shoddy 2. Also, assuming ladder ranking is not the only way to get on the council (and I think there should be multiple means), then either you let very few people from the ladder into the council, or you have quite a large council. Top 15 of the ladder, all the Gen 4 tiering contributors who are still active battlers, maybe some other people - you're getting up to around 50 there.

 

[Seven Deadly Sins]

After a long, mostly civil debate in #stark, we (alex, me, Blue Kirby, Gouki, Erazor, Synre, and others whose names I cannot recall) kinda centered on the following proposition:

Community Nomination, Committee Decision

Basically, it would run similarly to how the UU test is run in that the community would decide on which Pokemon should be considered "suspect", and should be either bumped out of OU into Ubers or vice versa. The committee would parse these nominations, then make their decision based on their own opinions and those of the people who post nominations. When we talked, we decided it would probably be optimal to have the Policy Review-eligible posters do the nominating, to at least have a filter for the community writ large, but obviously there are other ways it could be handled.

The nominations would be opened up at regular intervals (1-2 months, though a special nomination could be called if something goes horribly wrong, I suppose), and at that time everyone capable of nominating has the option of doing so. The committee would be selected at the discretion of Aeolus and Jumpman16, and would as large as they saw fit (at this point, 9 people).

It has all of the ideas that we're going for:

• Community involvement - While decisions are still "made" by the committee, the committee's actions will be dictated and heavily influenced by members of the community outside of the committee.
• Less bureaucracy - No more "testing for a month, then running a bunch of stats and sifting through paragraph submissions just so we can vote on one Pokemon". Instead, it's a streamlined process where nominations are posted and then immediately decided on.
• Less time - Decisions on the metagame in general would be made much faster than in the current system, since "testing periods" would effectively not exist.

It doesn't get much better than this, folks.

 

[alex]

After a long, mostly civil debate in #stark, we (alex, me, Blue Kirby, Gouki, Erazor, Synre, and others whose names I cannot recall) kinda centered on the following proposition:

Community Nomination, Committee Decision

Basically, it would run similarly to how the UU test is run in that the community would decide on which Pokemon should be considered "suspect", and should be either bumped out of OU into Ubers or vice versa. The committee would parse these nominations, then make their decision based on their own opinions and those of the people who post nominations. When we talked, we decided it would probably be optimal to have the Policy Review-eligible posters do the nominating, to at least have a filter for the community writ large, but obviously there are other ways it could be handled.

The nominations would be opened up at regular intervals (1-2 months, though a special nomination could be called if something goes horribly wrong, I suppose), and at that time everyone capable of nominating has the option of doing so. The committee would be selected at the discretion of Aeolus and Jumpman16, and would as large as they saw fit (at this point, 9 people).

It has all of the ideas that we're going for:

• Community involvement - While decisions are still "made" by the committee, the committee's actions will be dictated and heavily influenced by members of the community outside of the committee.
• Less bureaucracy - No more "testing for a month, then running a bunch of stats and sifting through paragraph submissions just so we can vote on one Pokemon". Instead, it's a streamlined process where nominations are posted and then immediately decided on.
• Less time - Decisions on the metagame in general would be made much faster than in the current system, since "testing periods" would effectively not exist.

It doesn't get much better than this, folks.

i'm "alex" and i approve this message

 

[Blue Kirby]

I enjoy aforementioned idea and even more so due to the fact that a debate ended productively for once, as alex mentioned in #stark.

 

[Erazor]

After a long, mostly civil debate in #stark, we (alex, me, Blue Kirby, Gouki, Erazor, Synre, and others whose names I cannot recall) kinda centered on the following proposition:

Community Nomination, Committee Decision

Basically, it would run similarly to how the UU test is run in that the community would decide on which Pokemon should be considered "suspect", and should be either bumped out of OU into Ubers or vice versa. The committee would parse these nominations, then make their decision based on their own opinions and those of the people who post nominations. When we talked, we decided it would probably be optimal to have the Policy Review-eligible posters do the nominating, to at least have a filter for the community writ large, but obviously there are other ways it could be handled.

The nominations would be opened up at regular intervals (1-2 months, though a special nomination could be called if something goes horribly wrong, I suppose), and at that time everyone capable of nominating has the option of doing so. The committee would be selected at the discretion of Aeolus and Jumpman16, and would as large as they saw fit (at this point, 9 people).

It has all of the ideas that we're going for:

• Community involvement - While decisions are still "made" by the committee, the committee's actions will be dictated and heavily influenced by members of the community outside of the committee.
• Less bureaucracy - No more "testing for a month, then running a bunch of stats and sifting through paragraph submissions just so we can vote on one Pokemon". Instead, it's a streamlined process where nominations are posted and then immediately decided on.
• Less time - Decisions on the metagame in general would be made much faster than in the current system, since "testing periods" would effectively not exist.

It doesn't get much better than this, folks.

I agree fully with this.

That was a nice productive debate we had btw.

 

[Gouki]

Yeah I fully support this idea as well as any others that might mention me. It was nice to see a discussion get somewhere before flaming out into procrastination and spite.

 

[Synre]

oh was i supposed to post in this thread



In agreement with the above, obviously.

 

[cantab]

I (who was not party to the IRC discussion) will back SDS's proposal, but do suggest one note - the committee should be free to run playtests if it feels they are needed to help it come to a good decision. (If you're considering banning a current OU you probably know enough from regular ladder play, but if you're considering letting an existing Uber into OU then if you don't actually play with it you've only got theorymon.)

 

[Seven Deadly Sins]

Actually, the point is to AVOID playtesting, considering that's a major part of what eats up all our time.

As I said, the system will have a built-in "we fucked up" clause that deals with unforeseen situations like what happened with Deoxys-E.

I also assume that there would be a significant amount of caution regarding dropping Ubers, but I also assume that whoever is selected for the committee will be capable of correctly assessing potential ramifications regarding the dropping of Ubers.

 

[Jumpman16]

I suppose it would look weird if I didn't post here even if I think this isn't going to be relevant for another six months. I understand that many (all) of you have felt that the Suspect Test Process has taken too long. Please realize that, as Aeolus noted, no one is more cognizant of this than him, myself, and Doug. Aeolus and I have shared many private conversations in which we've discussed streamlining the Suspect Test Process long before rolling out ideas to the community.

The concept of SEXP is one such example of our internal realization and handling of a problem of which some implied we were either not aware or unwilling to address the root. The streamlining of the "Suspect Test Process", in whatever way it will be recognized, is something that is a chief goal of Aeolus and me, and should be desired by everyone else involved, which I'm glad to see is largely the case. As far as your broader (and most relevant question), Chou:

We kind of have to figure out what it is we want ultimately. Are we doing the tests for the purpose of coming to some ultimate list? Or do the tests in and of themselves, some value (in terms of fun/stimulation) to the players?

As I stated in my Smog Article about the Suspect Test Process:

My goal has always been to include the community in the process of making and maintaining our competitive tiers, even though it would have been much faster to simply poll the opinions of a few of our tenured, well-respected and battle-tested members instead. At once you can see why this examination extends not only to Pokémon but to the greater community of Smogon.

While Aeolus and I may decide to "go in a different direction" in Generation V and beyond, it should be made clear what my vision for this test was initially and has been throughout.

 

[Chou Toshio]

Thanks a lot Jump. :D

The problem I see is that if we say the tiers are "set in stone" barring GameFreak additions or clear "emergencies", we then have to make sure we get them right first time.

Oh please, do not kid yourself in thinking that there is a "right" and a "wrong" in any of this. What makes for an ideal metagame is largely subjective, and what decisions will get us to such a metagame is in part guess work.

Everyone has their own opinions on what's "right," and no list will ever be perfect for everyone. If I think that bulky offense should be punished, than I would believe Shaymin-S should be OU (which I do in fact). If I believed Wall Breakers were completely unfair to stall, I'd ban Salamence and Infernape. There's no right or wrong-- only a matter of making a list that the majority of players will accept as reasonable and we as a community are proud (or at least not embarrassed) to stand by.

There's no such thing as a "perfect" nervana of tiering, so the value of concreteness and stability (which is real) trumps any argument for "correctness" (which is sketchy at the very best). I'm sure the vast majority of smogonites will be happy to frankly accept a reasonable looking list from a group of respected players.

Frankly speaking, if I was asked "Would you be happy (or at least willing to accept) a 1-off Uber list written by [Syberia] next gen?" [] insert ipl, earthworm, Jumpman (frankly not really caring whether the person is respected via administration or the ladder), I'd frankly say "Yeah sure. Cool." If there's a list that can be depended on, than people will learn to adapt to whatever meta it becomes. This should be the mind set-- not pampering our players by trying to make everyone happy with a given list, but rather saying

"Here you go guys, go at it. If you don't like it-- suck it up." That's what we'd have to say at the end of delivering any list we could think of anyway!


. . . on another note, I am all for including (and will support) the inclusion of the "We fucked up" clause.

 

[Aeolus]

After a long, mostly civil debate in #stark, we (alex, me, Blue Kirby, Gouki, Erazor, Synre, and others whose names I cannot recall) kinda centered on the following proposition:

Community Nomination, Committee Decision

Basically, it would run similarly to how the UU test is run in that the community would decide on which Pokemon should be considered "suspect", and should be either bumped out of OU into Ubers or vice versa. The committee would parse these nominations, then make their decision based on their own opinions and those of the people who post nominations. When we talked, we decided it would probably be optimal to have the Policy Review-eligible posters do the nominating, to at least have a filter for the community writ large, but obviously there are other ways it could be handled.

The nominations would be opened up at regular intervals (1-2 months, though a special nomination could be called if something goes horribly wrong, I suppose), and at that time everyone capable of nominating has the option of doing so. The committee would be selected at the discretion of Aeolus and Jumpman16, and would as large as they saw fit (at this point, 9 people).

It has all of the ideas that we're going for:

• Community involvement - While decisions are still "made" by the committee, the committee's actions will be dictated and heavily influenced by members of the community outside of the committee.
• Less bureaucracy - No more "testing for a month, then running a bunch of stats and sifting through paragraph submissions just so we can vote on one Pokemon". Instead, it's a streamlined process where nominations are posted and then immediately decided on.
• Less time - Decisions on the metagame in general would be made much faster than in the current system, since "testing periods" would effectively not exist.

It doesn't get much better than this, folks.

I'm generally fine with this... though I don't really want the periodic review. I want a stable tier list unless a problem arises. I say the community nom's when the gen first comes out... and we have an initial vote of the council to tier. After that, a review should only happen if there is an event or discovery that ruins things. There isn't really any need to convene every 4 or 8 weeks. As has been eloquently stated before, there is no "right" in the absolute sense when it comes to this stuff. Once a reasonable decision is made, people are welcome to accept it and adapt.

 

[Seven Deadly Sins]

As much as I agree, it's obvious that metagame shifts at least warrant a review... thus why I considered something like an 8 week "review". You could even just extend it, and just do it every 4 months when OU is calculated. However, I'm wary of just ignoring the effect that metagame shifts have on the viability or "uber-ness" of certain Pokemon. As much as I agree that "static tier lists" are an ideal, I also think that just "avoiding change" doesn't seem fair, especially when the ramifications of said change are minimal, and the process is streamlined and bureaucracy-free.

EDIT: I only said "review" because I assume that the committee is smart enough to decide when something is needed and not just act on a random whim. The way I foresee it, 90% of these "reviews" would end the same way: nothing changes. However, if the committee is doing its job right, that 10% of the time will be the "right decision" for the metagame. There's no real reason to just say "review will be done when we feel it needs to" or something like that. Also, note that said "review" would last maybe one or two weeks maximum and wouldn't affect the majority of players.

 

[mtr]

No, I agree with it being reviewed every 2 months in OU. Also, for what it's worth I support the idea.

 

[goofball]

I disagree, there shouldn't be a need for a meeting every few months. After all, the council will be members that are actively battling within the metagame and active in the community as well. If they feel that something has to be done, they can simply organize a meeting within a few days and discuss the problem. Like you said, a majority of the time, the "reviews" won't even matter. There's no reason for the council to hold these reviews, if nothing has to be done. In my opinion, it is more effective to "review things when there is a need to."

 

[Chou Toshio]

^Exactly. To be honest, I think Aeolus' plan is the best.

Remember, if we include a "We fucked up" clause, the council members can convene at ANY time if they see fit-- and it's likely they will convene if there is an obvious reason ("YacheChomp" second coming).

With a group of active players taking charge, if there is a real need (emergency), it should be felt by the meta at large, and most obviously to the kind of players we are going to pick to take those roles.

Considering all that, forcing them to convene for arbitrary reviews seems arbitrary, and also the simple existence of the reviews built into the system discredits the image of stability we try to give to the tier list.

 

[lilyhollow]

I agree with Aeolus, goofball and Chou, especially Chou's "image of stability" bit, which I think we shouldn't understate (though "why would we have regular reviews if we don't expect to gain anything from them" is certainly solid reasoning all its own). I don't like the idea of having a community that feels its focus is on scrutinizing the metagame's balance so they can nominate things or write articles proving a Pokemon's brokenness or whatever. As long as there aren't any major problems to be dealt with, they should be focused on playing and winning.

Actually Aeolus, could you elaborate more on your position here? Your post seems to suggest limiting community involvement (meaning nominations of Pokemon) to the beginning of the process, in which case I completely advocate it. But if you think community/PR/whoever nominations would still have a place, I'd love to know exactly why that is. I don't think I see as much value in community involvement as others do.


edit: Yes, by "community involvement" I'm referring to anything that extends beyond the Council and administrators, sorry that wasn't clear.

 

[Aeolus]

Well, Blame Game, I actually would consider the Council itself to be "community involvement". If you mean the larger universe of PR contributors, then yes, I think it is generally wise to limit involvement of that large group to the very first, initial tiering discussion. Also, SDS, when I say I do not want periodic reviews, I mean I do not want scheduled periodic reviews of every X weeks/months. Obviously, Jumpman and I would convene a Council every time the "facts on the ground" make it prudent to do so.

 

[Seven Deadly Sins]

What exactly would "facts on the ground" mean?

 

[alex]

I think the frequency of meetings is a fairly unimportant issue. Having a fixed schedule is probably going a bit overboard, in retrospect - letting either a majority of people on the committee (think SCOTUS writs of cert) or the people at the head of the process "convene" a meeting whenever it's deemed necessary is more than likely sufficient. Overall though, ironing out these kinds of details is probably not something even worth worrying about so far in advance. Things like the nomination process and who will be officially deciding the tiers and how should take precedent until the committee's actually formed.

 

[Aeolus]

I think the frequency of meetings is a fairly unimportant issue. Having a fixed schedule is probably going a bit overboard, in retrospect - letting either a majority of people on the committee (think SCOTUS writs of cert) or the people at the head of the process "convene" a meeting whenever it's deemed necessary is more than likely sufficient. Overall though, ironing out these kinds of details is probably not something even worth worrying about so far in advance. Things like the nomination process and who will be officially deciding the tiers and how should take precedent until the committee's actually formed.

I agree with basically all of this. If I were to criticize the value of this forum, I would say that it makes people feel like they should be commenting on every minute detail of every possible decision. This is not how anything works in the real world or on the internet.

I think the greatest possible value of this thread has already been achieved. We've set a basic framework and even begun to hear suggestions on how to flesh it out. I'm not willing to post here and go back and forth debating every tiny detail with every single user who has access to this forum.

If you have a major objection to the idea of a "council" in general, great, let's hear it. If you have a beef with "how often will they vote" or "what users will vote exactly" or "other detail x" then that isn't relevant now so you can keep that to yourself. It appears to me that this framework has broad support. The actual execution of these general ideas should rest with the people running the process. It is incredibly inefficient and unproductive to make tiny decisions by committee.

As far as the nominations of suspects goes, I've already outlined a sensible plan on that. And I'm sorry SDS, I don't feel like it is necessary to further explain what I stated earlier as "facts on the ground". Everyone knows what is meant by that; it is not helpful or productive to haggle about the common meaning of phrases.

 

[Erazor]

There's only one problem that can arise with the council idea, and that's the bias factor. More specifically, that the admins would "stack the council" in order to achieve a particular outcome. Now, I have full faith in Jumpman and Aeolus. They have shown as little bias as possible during the whole suspect process. However, the council members themselves may - may - not be as impartial.

Now, I know that there will always be some measure of bias. This is pokemon, it's never going to be completely objective. However, we need to make sure that the committee isn't formed with an "agenda" in mind, so to speak.

This is sort of hard to explain... we were having this discussion in #stark, in which I said that I though Salamence wasn't broken, and someone replied "Clearly Erazor shouldn't be on this council". Now, I know that this was a joke, but what if someone actually has this mentality? That you must believe that X is Uber or Y is OU to be worthy of a seat on this council? This is something that we should make sure never happens, although I doubt that this will happen in the first place, since I have great faith in Smogon to do the right things.

I'm sure a lot of people have/had this fear, and that's primarily why I posted this. Personally, I think that the council will work out beautifully, but others may not think so.

 

[Chou Toshio]

Erazor, I think you (and actually most members) are blowing the importance of tiering out of proportion. At the end of the day, the Uber tier is just a given list of pokemon we will not include in standard play, that is to some degree-- arbitrary.

Especially getting near the bottom of Ubers, top of OU, it's not like the world is going to end based on whether or not the likes of Manaphy or Latias are going to be OU or not. Frankly speaking, if there's no OBVIOUS problem, than whether the pokemon is Uber or not is pretty arbitrary. Lead Deoxys and Yache Chomp made obvious situations-- in the case of things like Manaphy, Skymin or Latias, I don't think people are going to (or should) riot one way or another.

So what if Salamence is Uber? So what if Latias or Manaphy is OU? Unless it causes big problems in the system, it frankly doesn't matter all that much.

The thing is that even if you pick a council meant to give you a list smogon thinks is great-- you've biased the tiering right there! In fact, I certainly want them to be biased towards making a list that most smogonites can at least respect. A small group of our best players will certainly not embarrass us.

It's not like they're going to go on a spree of banning Vaporeon and Flygon. When it comes to pokemon that border between Uber and OU-- frankly it really doesn't matter if they end up going one way or another.

I don't like the idea of having a community that feels its focus is on scrutinizing the metagame's balance so they can nominate things or write articles proving a Pokemon's brokenness or whatever. As long as there aren't any major problems to be dealt with, they should be focused on playing and winning.

THIS

If there is one thing that 4th gen and all this testing has done, is blown the importance of tiering way out of proportion.

At the end of the day, Ubers is just a list, that is innately arbitrary/subjective/biased to a certain degree. Just 1 tiny part of the ruleset. What really should matter is not what pokemon are in OU, but rather who finds the best ways of using them.

Really, all this tiering stuff makes it seems like the environment is at fault for winning and losing, but as a competitive community what should really be important is the players themselves-- not the ruleset.

 

[Jumpman16]

I disagree, there shouldn't be a need for a meeting every few months. After all, the council will be members that are actively battling within the metagame and active in the community as well. If they feel that something has to be done, they can simply organize a meeting within a few days and discuss the problem. Like you said, a majority of the time, the "reviews" won't even matter. There's no reason for the council to hold these reviews, if nothing has to be done. In my opinion, it is more effective to "review things when there is a need to."

I don't think we need to look much farther than Latias herself to be reminded of why this is a little off. The same future "council" members, who will somehow be more capable of raising an issue of importance regarding tiers, have for whatever reason not done so regarding Latias over the past four or five or six months that she has evidently been still very much a Suspect by definition, yet allowed in Standard play and the Smogon Tour and most tournaments.

It's a little disconcerting that others, however active and respected, feel they need conferred upon them some special title or badge to feel that they can organize their thoughts and discuss an important problem, especially when this forum exists and has existed for years almost exclusively for this purpose. And I think it's fair to assume that most if not all members of such a council either currently have a Badge that grants a de facto permission to post in this forum or have been granted posting privileges here regardless (though the Stark Mountain forum and Aeolus and my PM boxes have been long accessible to everyone).

Right now, you're either surprised at the relatively sudden belief that Latias is uber, (relative to how long Latias has been in OU, to be clear), or you're of the so-called "active" community who has kept largely mum about how broken she is in Standard play for months and months. I realize that this development, however drastic, may be owing in large part to natural metagame shifts that sometimes take months and months to develop themselves. But even right now, one could make the argument that Latias should not be on the Standard Ladder, today.

I'm willing, however disappointedly, to accept that maybe many of you do need a certain status or title to feel you're allowed to voice opinions and concerns—or allowed to be prompted by others to do so, according to a certain schedule—regardless of the wide extent to which you were respected all along (again, we're talking about the same people on a hypothetical "council"). But this doesn't mean we should forget that the ability to voice such insight on problems and issues has long since been possessed and underutilized by the very same people. So in a few regards, I therefore respect Syberia more than many of you for at least attempting to voice his concerns, in a forum created for exactly such concerns, and a forum to which I personally granted him and many of you posting privileges years ago.

I think the frequency of meetings is a fairly unimportant issue. Having a fixed schedule is probably going a bit overboard, in retrospect - letting either a majority of people on the committee (think SCOTUS writs of cert) or the people at the head of the process "convene" a meeting whenever it's deemed necessary is more than likely sufficient. Overall though, ironing out these kinds of details is probably not something even worth worrying about so far in advance. Things like the nomination process and who will be officially deciding the tiers and how should take precedent until the committee's actually formed.

I would argue that perhaps the biggest problem with the Suspect Test Process has been asking just two people in Aeolus and myself to do all the necessary "deeming" despite the stated importance upon being a prolific, skilled battler. This problem is of course inversely proportional to whatever extent none of the "would-be-council" members have not been or felt responsible for the duration of Gen IV competitive play.

Erazor, I think you (and actually most members) are blowing the importance of tiering out of proportion. At the end of the day, the Uber tier is just a given list of pokemon we will not include in standard play, that is to some degree-- arbitrary.

I think you're really off base here—tiering is incredibly important to almost every single aspect of competitive pokemon. To whatever degree it is "arbitrary" has no bearing on how many modes of competitive play and how many other communities depend upon the decisions of a competitive community they expect to get it as close as possible to "correct" as we can.