Policy Review Stage Order in Concept Assessment

quziel

I am the Scientist now
is a Site Content Manageris a Forum Moderatoris a Community Contributoris a Smogon Discord Contributoris a Top CAP Contributoris a Contributor to Smogonis a member of the Battle Simulator Staff
Moderator
Approved by Jho
This is a significant revision and improvement to the ideas in a previous policy review thread I posted (https://www.smogon.com/forums/threads/defining-moves.3659891/).

It is no secret that every single concept is different in terms of aim, execution, and constraints, from Equilibra's concept being entirely designed around the question of "How do you make Doom Desire good" to the Move-Ability interactions of our starter trio. However, despite very large differences in those factors, the ordering of the stages has remained largely the same across most of our recent caps.

My proposal is then as such: We should allow the stage ordering to be discussed and modified by community consensus during Concept Assessment. Hypotheticals are bad, but, for example, I believe that the starter trio (and arguably Jumbao as well) would have had a lot more success if Moves and Abilities were far closer together than they were in practice, while Equilibra and CAP 27 were served well by the current staging. We should be biased towards the current staging because it has worked well for a lot of concepts, but I believe this discussion could help to serve some of the weirder, more specific, or more inventive concepts to reach fruition, and to fully fulfill their concept.

I guess that's about all I have to say, thank you for reading. (apologies for recent examples, but I wasn't around for earlier CAPs)
 
Last edited:

Bughouse

Like ships in the night, you're passing me by
is a Site Content Manageris a Forum Moderator Alumnusis a CAP Contributor Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnus
I could definitely see value in this for extreme concepts. Taking the idea to the extreme, I could imagine a concept about a Pokemon that can ONLY learn STAB moves or a Pokemon that can ONLY learn 4 moves, period. In either of these cases, moves have been inflated to such a high degree of importance, that I could see either of them working better if movepool went second after typing, or even first overall in the case of the latter concept.

But I don't see how this would be a real necessity for most any less extreme concept. We've done several move focused concepts before and they've never really suffered imo due to movepool coming last. Even when the movepool focused concept is open-ended, as it was for Cawmodore (which was very open - "A good user of moves with effects not frequently used in the OU metagame"), and to some extent, Equilibra (which offered only 2 choices). In these cases, what we've generally done is had an extended concept assessment where we nailed down the essential early, i.e. what move will it be? but otherwise left everything else in the regular order. Other examples of these sort of two-parter concept assessments that moved up big decisions (whether movepool or not) include Naviathan, Pajantom, and Necturna. Probably Voodoom and Volkraken (picking the partner/core)? I'm sure there's been many of these. I think we just need to be more open to multi-part concept assessment threads, and even continuations of the "concept assessment" (or whatever we want to call these bonus stages) that happen midprocess.

I do think there's something to be said for a concept about move-ability interactions to be more closely linked. Obviously we don't want to select Technician with Bonemerang in mind as a powered-up Earthquake, but then give an Attack stat that is too high for this option to seem palatable. If the point of Technician was Bonemerang, then Bonemerang should be able to get locked in before stats, and it's a shame that this didn't happen.

That said, there's a reason we use the process order we do. It generally works for any concept and follows a pretty good chain of logic to build from the more general to the more specific. I think it's rather difficult to do a movepool stage without stats. It's hard enough to do stats without movepool (though one has to go first!), but we can generally make a few reasonable assumptions about the STAB attacks and even a few coverage moves that a CAP will get later on. I think it's a lot harder to do this in reverse. We might not have even decided an attacking bias yet if we did moves before stats. Doing the Stats stage first informs both offensive and defensive calcs, while movepool first overwhelmingly only informs offensive calcs and leaves all the defensive decisions til later.

My thought is that instead we could just have had an ad-hoc step that says "hey! our concept is about movepool-ability interactions and we just selected Technician - let's discuss the moves we want to use that interact with Technician". Not the whole moveset, and certainly not the whole movepool. But something that made clear we picked Technician with the moves Flame Charge, Bonemerang, and Mach Punch in mind. This would effectively be a concept assessment 2, just held rather late, after abilities. From there, we'd be able to do stats with those specific moves in mind and when we get to the full movepool discussions, these moves are locked in and appropriate with the stats selected.

I think this would be a pretty natural extension of what we already do to try to promote flexibility around concepts, while being less radical about the process as a whole.
 

quziel

I am the Scientist now
is a Site Content Manageris a Forum Moderatoris a Community Contributoris a Smogon Discord Contributoris a Top CAP Contributoris a Contributor to Smogonis a member of the Battle Simulator Staff
Moderator
This proposal was definitely targeted to those more extreme cases; ideally you wouldn't need to diverge from the standard order unless there's true need, as in your extreme cases, or you have something proper weird, eg the Starters (even then you could use the Defining Moves proposal https://www.smogon.com/forums/threads/defining-moves.3659891/).

I do agree that the Concept Assessment Stage, and well, similar stages, can definitely be used more throughout the process. Having a secondary concept assessment stage after Ability to see if there are any moves that are super relevant to how the mon could function could prove pivotal to really fulfilling some concepts. My ideal would be to only really invoke this proposal as a formality, if even as a formality, for most concepts, and to use a Defining Moves stage for other concepts where we need to ensure that specific concept relevant moves aren't made overpowered by stats.
 

Birkal

We have the technology.
is a Top Artistis a Top CAP Contributoris a Top Smogon Media Contributoris a Site Content Manager Alumnusis a Battle Simulator Admin Alumnusis a Senior Staff Member Alumnusis a Community Contributor Alumnus
This is a pretty significant change. It puts a lot of power into the hands of the TL specifically. I'm going to tag DougJustDoug because he may want to comment on the historical contexts of why the CAP process is set up in the rigid structure it has. However, I'll keep my response brief. I think this is a good idea, with additions. We should probably write a guide for when TLs would want to change the order (and maybe make a guide in general), we should emphasize in concept submission that specifying a change in order during the process is polljumping, and that changing the order HAS to happen in concept assessment and not midstream.

Good idea here!
 

DougJustDoug

Knows the great enthusiasms
is a Site Content Manageris a Top Artistis a Programmeris a Forum Moderatoris a Top CAP Contributoris a Battle Simulator Admin Alumnusis a Smogon Discord Contributor Alumnusis a Top Tiering Contributor Alumnusis an Administrator Alumnus
Moderator
This is a good discussion to have. I have a lot of thoughts on this, both for and against. I'm not really leaning pro or con, so I am just dumping my thoughts into a long post on the subject of CAP process ordering and then we can all sort it out together. Writing the post is taking longer than I expected, so please don't conclude this thread until I get the post finished. I'll replace this post as soon as I am done, and we'll take it from there.
 

DougJustDoug

Knows the great enthusiasms
is a Site Content Manageris a Top Artistis a Programmeris a Forum Moderatoris a Top CAP Contributoris a Battle Simulator Admin Alumnusis a Smogon Discord Contributor Alumnusis a Top Tiering Contributor Alumnusis an Administrator Alumnus
Moderator
Sorry for taking so long to get this post together. I have a lot of thoughts on this proposal, both for and against. I'm not really sure whether I am leaning pro or con, so I'll just dump my thoughts and then we can sort it out.

I think this is a good discussion to have, since we haven't talked about the subject of stage ordering in a while. It is a controversial topic, and there has been a lot of debate about stage ordering over the history of the CAP project, particularly in the first few years as we were struggling to figure out the best way to, y'know... create a pokemon. And, like many things in CAP, our stage order and our policies towards stage ordering are not perfect. In fact, you could argue they are not even very good. But over time we settled on the current order and policies, mostly as a solution that was not necessarily good, but better than all the other alternatives. The CAP project has changed a lot over the course of 13 years and almost 30 creations, so talking about this now is worthwhile to see if changes might help us in the future.

With that said, let me remind everyone of some key fundamental principles of the CAP project that are directly applicable to this discussion. The fundamental principles I will be referring to below, are so important that they are in fact, the first principles listed in the "Fundamental Operating Principles" of the CAP Project as explained in the CAP Leadership Compendium which is stickied here in the CAP PR subforum. That compendium is a super-long post, so I'll excerpt the relevant principles here. It refers to the inevitable, unavoidable, maddening "Dichotomy" of the Create-A-Pokemon project:

CAP Leadership Compendium
Fundamental Operating Principles

Dichotomy Part A: This is a terrible way to build a good pokemon


If you really want to make a good pokemon, don't invite a few hundred people together to make decisions in a rigid step-by-step manner and use majority vote for every significant decision. Design-by-committee sucks. If you want to make a good pokemon, get a few talented people together in a room, and make it an iterative effort. Building a pokemon en masse waterfall-style is fraught with problems and is virtually guaranteed to have many stumbles and mistakes along the way. Popularity voting is a horrible way to get anything of quality, because propaganda and bandwagoning tend to overshadow "doing the right thing" almost every time. All in all, the CAP process is pretty much the worst way to make a high-quality pokemon you could possibly conceive.

So get used to it.

We are NEVER going to encourage a few supposed "experts" to go off in a private room and make their own wonderful pokemon, and then reveal it to the CAP masses and expect the rest of the community to call it "our creation". It is not going to happen. If you even imply it -- you are threatening the community aspect of this project and you are missing the whole point of Create-A-Pokemon. If we don't do this thing as a big collaborative group -- then we are not doing "Create-A-Pokemon".

The internet is filled with thousands of fakemon created by individuals or very small groups. It is so common and uninteresting, it's not even funny. Do a quick Google search on the term "fakemon". Do you honestly give a shit about the stat spread or movepool of any of the thousands of pokemon you see in the list that comes up? Do you have even the slightest desire to actually work to implement any of those pokemon and use them in real battles? Don't get me wrong, some of those fakemon look cool as hell and you'd probably try them out if you could easily battle with them. But would you actually expend any of your personal time and effort to determine if those pokemon were competitively viable, and then work to implement them on a simulator to use them in battle?

Hell no, you wouldn't.

Because all those personal fakemon creations are a dime a dozen. They saturate pokemon fandom so completely that they are irrelevant at best, and annoying at worst. That is not the goal to which Create-A-Pokemon aspires.

I don't care what you personally know about making awesome pokemon, and what kind of amazing pokemon you might make if you had a chance to do it on your own. Let me repeat that for emphasis -- I DON'T CARE. And neither does anyone else. And that is exactly the point and key differentiator between your perfect personal creations, and our horribly flawed group creations. People CARE about our pokemon creations, and they don't care about yours. And if ours is flawed and yours is perfect, what does that tell you about how quality relates to interest and involvement? END RESULT QUALITY DOES NOT MATTER MUCH AT ALL. What matters is that we all get to play a part in the creation of our pokemon, and we all feel like we own a little piece of it.

Create-A-Pokemon gets many people invested in the creation process, and it is precisely because of that investment that our project stands head and shoulders above all the other personal fakemon projects out there. Yes, our process makes it very hard to make a good pokemon. But it is the ONLY way to get everyone involved, and THAT is the reason the project exists.

So when I see individual CAP participants getting pissed off about our pokemon being "ruined" by people that "don't know what they are doing"; or when I see individuals trying to hog threads, manipulate polls, or otherwise short-circuit the community in order to make what they consider to be a "better pokemon" -- those people just don't get it. In fact, as a community leader, you should see it as your job to protect the community from the influence of people like that. So if you really can't stand to be part of a project that knowingly and consistently makes less-than-perfect pokemon -- then get out of here and don't come back. Because you are not going to be happy here, and we're not going to be happy with you while you are here.

On the other hand, if you like being part of big groups, if you like pitting your ideas against others and vying for your ideas to be selected over others, if you enjoy the crucible of creating something and exploring ideas with others -- then the CAP project is the place for you.


Dichotomy Part B: CAP pokemon are important and our decisions matter

This may seem completely contrary to the point I just made above, but CAP pokemon are "important" and our decisions have important consequences. I put the word "important" in quotes, because I don't mean they are literally important. We are just making fake pokemon for crissakes. But we need to treat each step of the process as important, otherwise everyone will make a half-assed effort on the project. Although we must accept that the result of all our hard work will likely not be very good, we still need to encourage a project where everyone gives their best effort.

People don't work hard on anything if they don't feel like the results of their effort have any real impact. So it is imperative that CAP takes each project seriously and we consider each decision carefully. We encourage participants to make good arguments. We encourage submitters to work their ass off on their submissions and to put their best foot forward. We celebrate winners, and we console losers. Each step of the CAP process is treated with gravity and importance -- and that is BY DESIGN. It makes everyone step up their game and work hard to make the best possible decision on every step. It's what makes each CAP pokemon as good as it reasonably can be.

Deal with the dichotomy

If you want to be a leader around here; if you want to truly "get it" in terms of the inner workings of Create-A-Pokemon -- you need to synthesize the two principles I just presented above. On the one hand, you have to accept that our creations will most likely never be very "good". But you must also be absolutely committed to encouraging us to make it as "good as it can be", taking into account the considerable limitations imposed by the project structure and community creation process. Most people tend to fall in one camp or the other, but never really figure out how to promote both principles. If you understand the importance of both, then you are well on your way to being a true CAP leader, and not just a person who posts a lot in CAP threads.
The dichotomy of CAP rears its head in almost every aspect of the project, but it is particularly frustrating when talking about Concept Assessment and Process Order.

Since even that Dichotomy excerpt is pretty long, I will enumerate a few key points for TLDR:
1) If you really want to make a good pokemon, don't invite a few hundred people together to make decisions in a rigid step-by-step manner and use majority vote for every significant decision.​
2) We are NEVER going to encourage a few supposed "experts" to go off in a private room and make their own wonderful pokemon, and then reveal it to the CAP masses and expect the rest of the community to call it "our creation".​
3) Each step of the CAP process is treated with gravity and importance -- and that is BY DESIGN. It makes everyone step up their game and work hard to make the best possible decision on every step.​
4) On the one hand, you have to accept that our creations will most likely never be very "good". But you must also be absolutely committed to encouraging us to make it as "good as it can be", taking into account the considerable limitations imposed by the project structure and community creation process.​


The whole idea of "Concepts" for CAP is, in some ways, antithetical to CAP principles of community step-by-step creation. Because a Concept is essentially a vague description of a fully completed pokemon. And, as we have have clearly established, CAP pokemon are not created by a small group of experts, and certainly not created by some individual Concept author.

That is the reason we have many rules about what constitutes a valid concept. Because if any aspect of a Concept is too prescriptive or too narrowly defined, then it undercuts the ability for the community to participate as a whole. We do NOT want steps of the CAP process where there are a small number of legitimately possible options to discuss. We need broad discussions that allow different ideas and opinions to be presented. Those are the kinds of discussions that make community members feel like they are truly part of the creation process, and not simply voters choosing from the options given to them by the TLT.

But we need Concepts and other mechanisms to keep the discussions from being too broad, and more importantly, to help encourage each step of the process to proceed towards creating a somewhat cohesive pokemon in the end. I specifically use the words "encourage" instead of "guarantee", and "somewhat cohesive" instead of "highly cohesive", because CAP often falls into the trap of trying to force certain outcomes while trying to make good competitive pokemon. And we just can't do that, as explained with the whole Dichotomy of CAP above.

And that brings me to the heart of this PR thread. Essentially, we are saying that in order to make sure certain Concepts get executed in a more cohesive way, we want to tailor the process order to cater to the needs of the Concept. Taken at face value, I don't have a problem with changing the process order. Theoretically, we can execute the process in almost any order we want.

The four basic competitive aspects of a Pokemon -- Typing, Stats, Abilities, and Movepool -- are completely interchangeable in the process order. There is nothing mechanically that dictates that any basic step needs to be done before another. And, in fact, there is nothing mechanically that dictates the steps need to be interrelated at all! I could literally randomly select each basic aspect from a list of all possibilities and put it all together, and call it a pokemon, without violating any mechanical rules of the game. I'm sure there are some exceptions, but ykwim.

The only reason we have all these rules and order for our steps is to give ourselves the best chance to have some cohesiveness in our final creation. For example, for most Concepts, we all tend to agree that Typing is such a big deal on a pokemon, that almost everyone thinks it should be decided first. But for the other three basic aspects (Stats, Abilities, and Movepool), it is almost completely subjective as to which one is next most important to be decided after Typing. And the arguments for or against any particular ordering of the next three are all over the place. There simply is no objectively best order for the other three, IF we are truly adhering to CAP principles.

But that's a very big "if"...

In the past, the main reason we have had arguments to decide something earlier in the process than the normal ordering dictates, is because some people really don't want the "wrong thing" to be chosen for whatever step is being targeted to be decided earlier than normal. They feel like a later step is critically important for the Concept, and therefore some decision needs to be made earlier. To be more specific, they feel that a certain specific outcome of a later step is critically important for the successful implementation of the Concept - and they want to guarantee that specific outcome in order to give the Concept the best chance for success.

Of course, most people don’t actually argue that they want a specific outcome of a later step. Usually the argument is along the lines of:

“We need to know if CAP X is going to have Ability Z or not, because if it gets Ability Z, then the rest of the pokemon will be built very differently than it will be if it doesn’t get Ability Z. “

And, of course, there is typically a ton of off-forum chatter about Ability Z. Which means if the Ability step were actually moved earlier in the process, the step would not really be a discussion. It would be a binary decision about Ability Z or Not Ability Z. And, with all the chatter and drama about Ability Z, it all but guarantees that Ability Z would win.

You can replace the Ability Z example above with “Speed greater than 120” or “Reliable Healing Move” or whatever. The point is that there is some specific pivotal outcomes of steps that are believed to be make-or-break decisions for the Concept. And people want it to "make", not "break".

With all the polljumping Discord and Showdown discussions happening outside the forum, there is often a bit of polljump railroading happening all the time in CAP, even without changing the process ordering for a given CAP. I don’t know if that is good or bad for the quality of discussion in the project. Sometimes we open a discussion thread and it feels like the step is a foregone conclusion, simply because it has already been discussed so much off-forum and the CAP diehards have collectively already determined the best outcome for the Concept. The posts in the thread sometimes appear to be almost obligatory going through the motions, rather than actual idea proposals and debates.

So based on that line of reasoning, I would be opposed to allowing the process order to be changed for a given CAP Concept. Changing the process ordering would probably NOT be a process change to facilitate a good discussion and debate about a step. It would be a way for the off-forum predetermined outcome to be cemented in place in a more predictable fashion. And that’s bad for CAP. Because “going through the motions” threads are not just boring, they undercut the premise that this is a community endeavor and that the true home base for our decision-making is the forum project.

HOWEVER... that isn’t the only way of looking at this proposal...

The fact is, over the course of almost 30 CAP creations, we have covered a lot of territory when it comes to Concepts. And with several hundred real pokemon in the real Pokemon game, all the so-called low-hanging fruit has been done already, either by GameFreak or by CAP. I am not saying we have done everything. But it is becoming damn hard to come up with an original “good Concept” that can actually be executed successfully while adhering closely to traditional CAP principles.

For CAP to move into new territory in terms of Concepts, one could argue we need to move into narrower niches of the game and pokemon with more specialized roles or more subtle implementations. And traditionally CAP just can’t do “subtle”. Our process isn’t made for that. It’s just the nature of the beast here in CAP, as has already been said in the whole Dichotomy of CAP stuff above. So, if CAP really has run out of room for good Concepts with the current process — maybe it’s time to change the process a bit to give us more room. This proposal is a possibly a good way to do that, if that is what we want to do.

Like I said at the beginning of this post, I can see pros and cons of this proposal. If this ends up becoming a thinly-veiled way for the TLT or the TL to railroad certain steps by moving them earlier in the process, I think it would be a bad thing for CAP. If this legitimately allows CAP to have better discussions and better creations, by making the most important decisions first, I think that would be good for CAP.

I tend to think this is probably really only a problem for more subtle, specific, or extreme Concepts. And traditionally I have been opposed to CAP considering those sorts of things. But times change, and perhaps CAP needs to change too, to keep things fresh and interesting for the entire community. But if this really is a change that will allow a different breed of Concepts in the future — then we should be open that we want to go there. And then figure out how to best do that in the most community-oriented way possible.
 
Last edited:

Users Who Are Viewing This Thread (Users: 1, Guests: 0)

Top