Policy Review Policy Review - Playtesting Ladder

[DougJustDoug]

If you are not an experienced member of the CAP community, it is strongly recommended that you do not post in this thread.

This thread is intended to contain intelligent discussion and commentary by experienced members of the CAP project regarding CAP policy, process, and rules. As such, the content of this thread will be moderated more strictly than other threads on the forum. The posting rules for Policy Review threads are contained here.

I've been carrying this idea for a while, and I've discussed it on the server over the past few weeks.

At the conclusion of each CAP project, I would like to do the following for playtesting our new creation:

• Close the CAP ladder temporarily
  
  
• Open a Playtesting ladder for two weeks. The Playtesting ladder would only allow the NEWEST CAP pokemon to be used. All past CAP pokemon would be disallowed.
  
  
• The Playtesting ladder ratings would always be reset at the beginning of playtesting.
  
  
• At the conclusion of the two-week playtesting period, the highest ranked player would be crowned the "CAP x Playtesting Champion". Perhaps I would give them a custom title on their profile, that they could carry until the next CAP playtesting champion is crowned.
  
  
• After the playtesting period, the Playtesting ladder would be closed and the regular CAP ladder would be reinstated, with all CAP pokemon eligible to play, including the latest creation. Ratings would remain the same on the CAP ladder as they were prior to the playtesting period.

If this proposal is implemented, I think it would yield several benefits:

• It will encourage participation in playtesting the latest creation. Since the playtesting ladder would consist ONLY of OU pokemon and the latest CAP creation, it would be very accessible to anyone who joined the CAP project during the latest creation. Essentially, it provides a perfect "jumping-in point" for new forum members to try out the server, since there is no metagame baggage to carry from previous CAP's. Each new CAP pokemon will receive focused attention, and hopefully more play and feedback.
  
  
• The lure of a little forum e-fame should draw in some new battlers -- perhaps even people that weren't involved in the CAP forum during the creation project. Any good OU battler should be able to pick up one new pokemon quickly, and since the new pokemon is "new" to all battlers -- no one has a "head start". Since all Playtesting ratings will be reset, there will be no ratings baggage either. People could literally join the server on the first day of playtesting, and have almost no playing disadvantage to longtime CAP battlers.
  
  
• It will allow us to see the effect of our creation on the current OU metagame. I've never been terribly interested in making a new metagame completely. Just read the CAP mission statement, to see where I stand on that. However, it is an inevitable consequence of making a bunch of new pokemon -- that a new metagame will emerge. While some people may like it, I think it distances us from learning about the current metagame, which I consider to be a fundamental goal of the project. By having some time to test the new pokemon in isolation from other CAP's, I think we can learn more about the current metagame than we can by testing new CAPs with all the old CAPs as well.
  
  
• It will act as a "hook" for new long-term members of the server community. After the playtesting period is over, presumably some of the people who showed up for playtesting will realize how much fun it is to play on the CAP server, and they will stick around. Hopefully they will take the time to learn more about the other CAP pokemon, and try to ladder up the regular CAP ladder. That's good for the server and good for the project overall. Right now, it is hard to hook new players, because of the learning curve associated with all the past CAP pokemon. This proposal would go a long way towards alleviating that.

Response to this idea has been positive on the server. I'd really like to try this out with Kitsunoh. That's why I am holding this Policy Review in the middle of a current CAP project, rather than waiting until after CAP 7 completes.

I look forward to feedback from members of the Policy Committee and other experienced members of the CAP forum and/or server community.

 

[Deck Knight]

I support this because it provides some direction. Eventually there comes a point where we're at CAP 12 and the metagame is barely recognizable, let alone capable of drawing inferences to standard.

Question:

I notice my rating goes down if I don't play on a particular Ladder for a few days. Is it really possible to implement a ratings freeze?

 

[sbc]

I generally agree with this.

Also DK, I think that would be counterproductive in getting people to regularly play matches.

 

[Infiltrator]

I support this, because:

It will encourage participation in playtesting the latest creation. Since the playtesting ladder would consist ONLY of OU pokemon and the latest CAP creation, it would be very accessible to anyone who joined the CAP project during the latest creation. Essentially, it provides a perfect "jumping-in point" for new forum members to try out the server, since there is no metagame baggage to carry from previous CAP's. Each new CAP pokemon will receive focused attention, and hopefully more play and feedback.

 

[Plus]

I'm not really seeing much bad in this proposition, and although many CaP regulars are completely fine with the metagame as it is now, newcomers will have a hard time trying to learn 7 or more completely different pokemon and will not attract many users to come in. This ladder will not only act as a deterrent to the overwhelming difference in the metagame we MAY create in the future, but it will also give us a clear understanding on how one pokemon alone can affect the OU metagame, which is, obviously, what we want to see in CaP.


Question:

I've actually shared laddering accounts with people such as Mag. Would that be allowed, or is that something that should be avoided?

 

[Colloquy]

I agree with this, as it would encourage people to come on and test the Pokemon, making them want to play the metagame. More ladders couldn't hurt, too.

 

[Magmortified]

When this was brought up on the server, I thought it was a pretty good idea. It solves the accessibility problem I brought up a while back (for the most part), and allows us to see how a Pokemon works in a pure-OU environment.

My concern was primarily if whether the information we're going to be gleaning here will turn out as accurate as it should be. It takes a while to adjust to a changed metagame - and this metagame will remove six different top-OU Pokemon and add one. Will the information we get from this, in just two weeks, be as accurate as that in a more stable time period (assuming we collect usage stats)? We're most likely not going to be able to reach an approximation of the standard OU metagame at the exact moment the other CAPs are removed and Kitsunoh is introduced. Adding this to a relatively-limited two week timespan... are we really getting accurate information?

Ideally, a short timespan is good because, for the most part, we're going to be eager to play with the other CAPs. So we're kind of stuck between the accessibility thing, and reliable information. Not to mention the continual disturbance of the metagame. Once after a new CAP is made, and then again two weeks later.

As with the stance we took in the Accessibility PR thread: should accessibility be a factor in discussion if we're not sure if it's actually affecting the server right now?

 

[Comatose811]

I know I have never posted in a Policy Review topic, and if I don't know anything, feel free to ignore me, but what happens when we decide upon a Anti-CAP concept that has been thrown out there a couple times, or if we make another Arghonaut. By placing him into a different metagame to start with, we lose the ability to gauge his performance in the metagame he was created for, and subsequently lose the ability to accurately decide upon the movesets to put into the analysis, a very important tool for incoming players.

 

[RBG]

THe way I see it, CAP is supposed to fill niches that are absent in the current OU metagame. What better way to test this than by having it on a ladder with the regular non cap OU?

 

[billymills]

I'm not exactly sure this would solve a long-term accessibility problem. Seeing as 80% of the time, there would be 7+ CAPs. It sounds like a good idea, and I'd support it, but there still is need for accessibility on the other 8 weeks of the cycle.

 

[cyberzero]

I don't see many drawbacks that seem important here as long as the rankings for the regular CAP OU ladder remain frozen. I would definitely support this, but Comatose has a point. We'll have to reevaluate our goals for the concept submissions, meaning that decentralizing and other similar goals to help the metagame won't be viable.

 

[DougJustDoug]

The fact that we have seriously considered an "Anti-CAP Pokemon" is strong evidence about how far CAP has diverged from OU. One aspect of this proposal is to steer the CAP project back on course -- which is to learn more about the OU metagame. If along the way, we create some stuff that might be interesting as a metagame unto itself -- that's fine. But, it's not our mission. And if the CAP metagame gets unwieldy in the future (too many pokemon, broken concepts, etc) -- then we can always scrap it or change it.

I don't like focusing too much on the CAP metagame itself. It may be interesting at times, but I don't think it is viable long-term, for reasons I have posted at length in the past. If anything, I am more concerned about the point billy raised -- that even if we adopt this proposal, the server will still be focused on the "full CAP metagame" most of the time.

This proposal is not the definitive solution to all CAP's accessibility problems -- it's a "step in the right direction". I don't want to implement big sweeping changes that may or may not help. At worst, I don't think this proposal could do much harm. But, at best, I think it could help quite a bit. After we have tried it out, we can re-assess and determine if we need to take further steps.

 

[jagged_angel]

I am very happy with this proposal and fully support it. I think that two weeks is an appropriate length of time for Kitsunoh testing but that depending on the outcome, it could be lengthened in the future, for example if the statistics are very inconsistent over a short period of time.

 

[Kaxtar]

I am not that expirenced with all the policy review stuff, but I do know the CAP metagame. I like this idea, and think we should go through with it. When I started battling on the CAP server, Stratagem had just been finished. It took me about 45 minutes to understand how to adapt to this metagame, and with in 3 days made a first team adaapted to the OU+5 CAPs. But now with all the new changes such as at least 2 new CAPs and recent changes to OU itself, this might give noobs to CAP a better learning expirence. But I do have a few questions/changes to this process. First, would this same "transition phase" be used for EVO? Or another way of phrasing that, would it be necessary? Next, how would the analysis step work? Would we make a preliminary analysis and later edit/add on to the first 2 week analysis after we learn how the new CAP reacts/interacts with the other projects? Or would we just stick with 2 week first one? Or heck, not even take the 2 week period into consideration? Now for a possible change, how about having the OU+ most recent CAP and OU+ all CAPs have two sepret ladders that both are usuable at the same time? I mean, suspect tests and normal OU operate at the same time on the regular smogon server, so why shouldn't that be the same for CAP? We could encourage both old and new players to play on the 2 week ladder, and for people who like all the CAPs, have them play on the regular ladder. The last thing we want is a shortage of users on the 2 week ladder. So all in all, a few suggestions on how we could improve this step. My two cents.

 

[Magmortified]

The fact that we have seriously considered an "Anti-CAP Pokemon" is strong evidence about how far CAP has diverged from OU. One aspect of this proposal is to steer the CAP project back on course -- which is to learn more about the OU metagame. If along the way, we create some stuff that might be interesting as a metagame unto itself -- that's fine. But, it's not our mission. And if the CAP metagame gets unwieldy in the future (too many pokemon, broken concepts, etc) -- then we can always scrap it or change it.

I really like the idea of testing how singular Pokemon apply to OU. I'm just not sure if we'll be able to learn from it in a two-week time period. A regular Suspect test for a Pokemon takes much longer, and even then it can be difficult to fully agree on something. Stability is the key issue here: Is two weeks long enough to reach a stable metagame that we can look at objectively? If not, then we're putting a Pokemon by itself for no reason. Our goal is to observe, but if we can't observe well...

I don't like focusing too much on the CAP metagame itself. It may be interesting at times, but I don't think it is viable long-term, for reasons I have posted at length in the past. If anything, I am more concerned about the point billy raised -- that even if we adopt this proposal, the server will still be focused on the "full CAP metagame" most of the time.

Short of complete elimination of previous CAPs (which, long-term, doesn't help us learn very much) or a simultaneously-run ladder, I don't think there's much to do on this here. A half-and-half scenario would have problems of its own. Particularly in that it's confusing to have to constantly shift metagames at regular intervals (one CAP's single-time, all of the CAPs, then a different CAP's single-time... ugh). Spending 80% of the time on a single-CAP ladder leads us to questioning what we're learning from the 20% of the rest of the time.

Personally, I feel that a metagame with multiple CAPs allows us to see how different roles can interact with each other. The metagame's not made up of singular niches. It's made up of a bunch of different niches having an effect on each other. To see the culmination of a number of different effects on a metagame could lead to interesting results. Which is kinda the reason I suggested a ladder with only a certain number of CAPs - not one, not all - back then.

EDIT: And Kaxtar brings up a good point. With the possibility of EVO bringing in new CAPs at an even faster pace, the CAP metagame could end up chaotically unstable. I'd actually think that more of a turn-off for newcomers if they knew they wouldn't be able to settle in.

This proposal is not the definitive solution to all CAP's accessibility problems -- it's a "step in the right direction". I don't want to implement big sweeping changes that may or may not help. At worst, I don't think this proposal could do much harm. But, at best, I think it could help quite a bit. After we have tried it out, we can re-assess and determine if we need to take further steps.

Unless we're sure if we're looking through the right lenses, the face-value of the results gleaned from the two-week testing period may not be accurate. Which can be harmful in and of itself. I'd agree to "testing the test," but only if we look back on it comprehensively afterwards and revise where necessary.

 

[Bass]

When I first read this, I had some reservations about not allowing the other CAP Pokemon on the Playtesting Ladder for pretty much the same reasons I stated in Mag's "Accessibility" PR thread a little while ago. Essentially, I thought that limiting the number of CAP Pokemon allowed would not give battlers a representative view of the metagame.

However, when I read the rest of the OP, I completely agree with the reasoning for this limitation. We have never really developed a formal playtesting process, and I think creating a separate ladder where it is the only CAP Pokemon allowed would actually give us very good insights on how it affects just the OU metagame, and I feel that through this we will be able to see where we were successful, as well as where we went wrong much more clearly than just dumping it on the standard CAP ladder.

In short, I fully support this proposal and I am interested to see how it will turn out if we choose to implement it.

 

[hydrolphin]

This is a good idea but we also would need a follow through. If we reintroduce the other 6 CAPs after two weeks, some people will be overwhelmed and leave. This would work as a good lure to get people on the CAP server, but what we really need is a tutoring system, to teach newcomers the metagame. These two weeks could be an introduction to the CAP metgame, while during the next CAP process, we can hold a tutoring session, where we bring these people in to stay. These two weeks are a grace period for people wanting to learn. Then we teach them everything they need to know after they get hooked.

Oh, and I'm all for all that stuff about the purpose of CAP and the effects on the OU metagame.

 

[DougJustDoug]

Maybe two-weeks is too short, but I just don't think we can go much longer than that. I based my two-week proposal on the period of time that people generally seem to be "engaged" in playtesting following our CAP creations. I am not a definitive authority on this, and I welcome feedback from others on this point.

It seems like the community is very enthusiastic for about a week after a new CAP is implemented, and then it trails off over the second week. After two weeks, the "honeymoon" is over for the new CAP, and people start moaning for the next CAP to start, if it hasn't already. Posts in the analysis and/or playtesting threads dry up almost completely after two weeks, and everyone turns their attention to the next CAP. IMO, two weeks is the most I think we can expect from the community regarding "dedicated playtesting".

I agree with Mag that two weeks is hardly enough time to glean much meaningful information about a new pokemon or its impact on the OU metagame. But, unfortunately, it's probably all the time we can reasonably take.

If we had a LOT more traffic, perhaps we could sustain two CAP ladders -- one for all the old CAPs and one for the newest CAP. Such a dual-ladder system could be used for the entire duration of the subsequent CAP project, at which time the "isolation CAP" would be transitioned into the "full CAP ladder", and a new CAP would move into isolation. But, the CAP server simply cannot sustain two simultaneous ladders, particularly if one of those ladders only has a single pokemon to distinguish it from regular OU. Look no further than Smogon U's Suspect ladder to see how difficult it is to get ladder play on a ladder with one new pokemon. And SU is the most trafficked battle server in existence.

The CAP server can only viably maintain one active ladder. And at this point, even our single ladder needs a boost of adrenaline. That is one reason I am making this proposal. By implementing two ladders, and staggering their play -- with one ladder working more like an "extended tournament" -- I'm hoping we can improve the overall quality of playtesting and server participation.

 

[cyberzero]

Doug, you're right in saying that it does die down after a week or so, but if the Pokemon is viable and has succeeded in it's niche in standard OU, it will keep being used. If the usage dies down terribly, then we've learned that it's not sufficient.

I think that period of three weeks would be better suited. The first week for the fanboyism to calm down because everyone will be using it, and the second and third weeks for us to evaluate how well it's been doing its job according to usage statistics.

 

[foibles]

I would just like to say, that being a member that only started playing CAP after stratagem, that is is pretty hard to figure out all at once since coming form standard OU and BL. So this would be very nice for people in a similar position.

 

[beej]

I'm in full support of the new ladder for all the reasons posted throughout the thread. This will be a great way to get more people while making it easier to actually test our Pokemon's relevance in standard.

On the subject of the playtesting period, is there any real reason why this new ladder can't be operational even after the two weeks are up and the statistics have been collected? It seems like a rather arbitrary limitation, even if it's speculated that most of the hype will die down within that time. By keeping the ladder around after the test, I think that we would more effectively take advantage of its ability to make the game easily accessable to server visitors driven away by the difficulty of adapting to the new metagame.

 

[User 12783]

Doug, you're right in saying that it does die down after a week or so, but if the Pokemon is viable and has succeeded in it's niche in standard OU, it will keep being used. If the usage dies down terribly, then we've learned that it's not sufficient.

I think that period of three weeks would be better suited. The first week for the fanboyism to calm down because everyone will be using it, and the second and third weeks for us to evaluate how well it's been doing its job according to usage statistics.

I agree with your proposal Doug, as well as Cyberzero's extra tidbits. We need CaP tutors, I really want to learn, though I don't have much free time to go on the CaP server, I'll try to go on more often with Kitsunoh.

 

[DougJustDoug]

On the subject of the playtesting period, is there any real reason why this new ladder can't be operational even after the two weeks are up and the statistics have been collected? It seems like a rather arbitrary limitation, even if it's speculated that most of the hype will die down within that time. By keeping the ladder around after the test, I think that we would more effectively take advantage of its ability to make the game easily accessable to server visitors driven away by the difficulty of adapting to the new metagame.

We can't sustain two CAP ladders on the server. We can barely sustain one ladder today.

The point of making a temporary ladder, with a defined end after two weeks, and recognizing a champion -- is to create some excitement and get some traffic on the Playtesting ladder.

If I just add a second ladder to the server, and leave it around semi-permanently, I fear that two things will happen:

1) No one will play it after a short time, so it will become an advertisement of LACK of activity on the server. That's not a good thing for the server or the project.

2) No one will get excited about the ladder, because it isn't "new". If it is around all the time, then it will be, "Ho hum. The playtesting ladder... yawn...." By making the Playtesting ladder a special event, hopefully we can drive some more interest and excitement.
​

I have heard many proposals for interesting ladder concepts on the CAP server. All of those proposals are unrealistic for the simple fact that it takes a lot of traffic to make a ladder viable. Of course, we CAN make as many ladders as we want. But, if no one plays a ladder, then it is worthless. In fact, it becomes a bad thing because it makes the server look inactive. As I frequently say, "It's bad for business".

At the very least, the ladder's ratings become meaningless, because it is trivially easy to manipulate matchups on an inactive ladder. The whole purpose of a ladder is to track ratings. If the ratings are able to be easily manipulated, then why have the ladder?

Smogon U has a hard time sustaining new ladders, and SU hosts over 300,000 battles per month. Even a ladder like Ubers is very hard to sustain on SU -- and the Uber metagame is a lot more popular than CAP. CAP hosts roughly 3000 battles per month. That's barely enough battles to make a single ladder viable.

Sure, I'd love to have more than one ladder. But, I'm not going to keep ladders around that have no activity and meaningless ratings.

 

[MapleSandwich]

I agree wholeheartedly with this concept. The offer of fame on the Smogon forums should draw many people who were involved with the project, but who might not want to participate in battles, to come to the server.

Pros:

• More members of the CAP community
• More awareness of how the current CAP impacts the standard metagame
• A chance to define non-CAP counters to the new Pokémon

Cons:

• The CAP ladder might see less usage

Given that the pros far outweigh the possible con, and that doing nothing will be guaranteed to see little usage on the CAP ladder, I support this concept.

 

[Goodbar]

The CAP server can only viably maintain one active ladder. And at this point, even our single ladder needs a boost of adrenaline. That is one reason I am making this proposal. By implementing two ladders, and staggering their play -- with one ladder working more like an "extended tournament" -- I'm hoping we can improve the overall quality of playtesting and server participation.

For this I'll support it, but like already expressed I can't see much being learned about the OU metagame from this nor do I think it's possible to keep CAP focused on exploring OU at this point, but at least CAP explores how different niches affect each other and how certain sets develop from a new pokemon entering the metagame. I think though we could accomplish that goal however if that ladder stayed open after the two weeks and only included the latest CAP, but that would require more involvement... But anyways I do support this because CAP could use such a hook