Deck Knight
Blast Off At The Speed Of Light! That's Right!
A few months ago CAP started winding down in anticipation of the 5th Generation, however we still had many issues unresolved. Polling was a particularly sticky situation, which came to no real resolution. With the effective departure of tennisace and the increased responsibilities Doug has taken on, CAP finds itself with a more limited availability it leadership. Here is what I have been doing in the interim.
Over the past few weeks I have been conducting a survey of the top poll placers of the last 3 CAP projects to get insight into our longstanding polling problem. With CAP on the backburner from an overall site perspective, while response was not as strong as expected there was excellent feedback that I will share with you. If the authors wish to come forward they may do so, I have constructed the survey in a way to maximize objectivity and ensured their opinions would be protected.
In the interest of full disclosure of my intent and methodology, here is the original PM I sent:
Submitted for your discussion is the feedback. I have made a few edits where appropriate to keep identities secret.
___
Rather than tally any of these up, I present this to you as the feedback of those members who have taken the time to deliver quality contributions to CAP as well as the time to help out during this lull.
Particular notes for discussion:
Recruiting: It's always been a huge issue in polls and NO method of polling (including IRV) is immune to it. Where do we draw the line between trying to get support and actively seeking it at poll time. One of the better insights I thought was the notion that you get your feedback in the discussion and submission threads, the voting is not really that helpful in the grand scheme of things. If this is the case is there any way to aid that processwise?
Poll typing: I guess this was the whole reason for having the survey, but from what I've seen there's no clear-cut answer, and in the end it's only 10 responses to go on. I feel however that by selecting people who have done well (but not neccesarily won) it would control for if people feel slighted or thought a particular method they encountered was unfair. I'm willing to try IRV out as a "neutral option" if and when we get back on track. I am deeply appreciateive of the thought put into the responses.
Feedback Factor: This last one is especially important to pin down, since there's a great deal of divergence and different needs for the competitive and art portions. The way we've set up submissions and discussions has improved this I think, but I would be open to suggestions.
As always, thank you to everyone who still shows interest in CAP at this point. Sorry if it's been a little dead, I've been waiting for this to trickle in in order to spark your interest. If you happened to receive a PM and still want to respond, please feel free and I will add it to the feedback we already have.
Over the past few weeks I have been conducting a survey of the top poll placers of the last 3 CAP projects to get insight into our longstanding polling problem. With CAP on the backburner from an overall site perspective, while response was not as strong as expected there was excellent feedback that I will share with you. If the authors wish to come forward they may do so, I have constructed the survey in a way to maximize objectivity and ensured their opinions would be protected.
In the interest of full disclosure of my intent and methodology, here is the original PM I sent:
Deck Knight said:Hello everyone.
Recently we had a Policy Review in the CAP Forum regarding polling procedures that never came to an adequate conclusion. In order to gather better information regarding this decision I am sending a PM to all of you in order to get your opinions on the various polling methods we have employed in the past You have been selected because you were the top finishers or runners-up during the last few CAP projects.
What I want you to focus on specifically is which of these polling methods creates the best initial poll. Once we have the field narrowed down to five or less, we use single click polls. However, the initial poll is the most important determining factor in how many options get to that second stage.
In assessing the poll options, I want you to consider the following traits for determining the “best” polling method:
Fairness: Is this voting method fair to lesser known or new submitters? Does it ensure the most equal playing field among veterans and newcomers, such that the best submission will win regardless of who submits it?
Feedback: Does the voting method give adequate feedback to the submissions? Is it giving submitters a good grasp of whether they finished on top, in the middle, or at the bottom? More broadly, does knowing the top, middle, or bottom placement help improve later submission quality in your experience?
Ideally, the “best” polling method should have a balance between these two traits. We want the actual voting process to be fair to all entrants, but we also want entrants who don’t get to the top to figure out if or where they went wrong so their submission can improve the next time around.
Additionally an important issue that has come up in CAP Polls is the matter of manipulation and cheating, whose definitions are given below:
Manipulation - Intentionally submitting or altering a ballot specifically to affect the end poll result. It is not illegal, but somewhat unethical. Manipulation usually takes one of two forms:
- Recruiting - People voting that have no credible knowledge of the poll itself, and no meaningful participation in the project. They are "recruited" to vote in a poll by someone else, who wishes to inflate the vote total or ranking of one or more options in a poll.
- Gaming - Altering a ballot specifically to affect the end result totals of the poll. Basically, a person is "gaming the poll" if they are not voting in strict accordance with which option or options they personally favor the most. This usually takes the form of a person not voting for their favorite option because their favorite option "already has it won" or "has no chance to win", and they move their vote to some other option. Alternatively it can take a form similar to recruiting where an option is moved up or down in an existing vote to affect its final ranking.
The key difference between recruiting and gaming is that recruiting deals with new votes from outsiders while gaming deals with the alteration of existing votes. It is possible for a particularly adept manipulator to both recruit an outside voter and game their vote.
Cheating - Submitting one or more votes under false identity or pretenses. This also covers "buying votes," basically exchanging something of value for a vote on a poll. Cheating is illegal and grounds for banning.
In explaining which poll method you find to be the best initial poll, please express any concerns you believe the methods might have in allowing for manipulation and cheating, and whether that concern makes that poll style inherently nonviable or not. Unlike the fairness and feedback criteria, the possibility for cheating and manipulation can make the entire experience worthless, even if it would result in better fairness or feedback if everyone followed the rules. Please consider this is giving your opinions.
As a final note before presenting you the poll options, if you think there are substantial differences between the Sprite and Art Polls and the other competitive polls in using these methods, please differentiate. You have been selected from every poll category where a name is attached to the submission, and these polls operate somewhat differently in terms of the number of entries and subjectivity in voter determinations.
The poll types I would like your opinion on are:
Unweighted, infinite bold vote: this poll allows each voter to submit an unlimited amount of votes on their ballot that each have the same weight. If someone votes for only three submissions on their ballot and another person puts seven on theirs, each of those 10 votes will count equally.
Unweighted, finite bold vote: This poll allows each voter to submit a fixed amount of votes on their ballot that each have the same weight. All voters must adhere to that fixed number on their ballot or their ballot will be invalid.
Weighted, finite bold vote: This poll allows each voter to submit a fixed amount of votes on their ballot, but in order of their preference to advance. Votes listed higher on the ballot have a higher weight than those listed on the lower end of the ballot. All voters must adhere to that fixed number on their ballot or that ballot will be invalid.
Please comment on each of these poll methods as comprehensively as possible, and get them back to me in a timely manner. I will publish the results anonymously, but if you want me to include your name for any specific commentary please notify me in your response.
Thank you very much for your time. Your opinions will help improve CAP Policy going forward and are greatly appreciated.
___Sadly, my opinion varies depending on the poll. When it comes to things like movepool, I'd rather have Finite, but Art Submissions should be infinite due to the size of the slate compared to movepool. That said, in each circumstance, unweighted is definitely the better option, since I know when I submit ballots for Art Submissions I just put down the ones I like going from the top.
I prefer Finite for more of the process, but I think the number should never be one, since then that restricts things to the point where you are unsure if someone is cheating or not. With at least two votes, you give open ground for anyone to win. Having say, two or three ballot options allows others to see what the opportunity cost of that one user's vote is. Let's say X and I were in a poll and someone wrote down mine and X's for his options. It shows that ours were the two best options, although which one he likes more is unknown. Either way, it gives us both a chance to win, since if he's voting for the pair of us, it shows he's happy with either movepool.
___Weighted, finite bold vote
I can't make a judgment on the fairness of any of these three methods.
To be honest, I'm not sure if it's even necessary to find a way to 'level the playing field'. A quick look at our list of CAPs reveals the same names cropping up again and again responsible for the competitive elements. There may be an element of bias there, but who is anyone to say that they didn't submit the best possible selection of moves or stats given the structure of the process at that time? These names show up consistently because these people generally choose the most favored direction in the first place, and the winners are after selected based on a few speed points here or whether or not the CAP gets taunt.
The art polls actually already seem to demonstrate a reasonable amount of diversity. We have a number of CAPs whose designs were submitted by relative unknowns (who often don't show up again because they then become inactive around the forums anyway).
Concept polls have clearly the greatest diversity in winning submitters. It shouldn't come as a surprise that they are well respected community members who would already have the best idea of what a good concept looks like.
What I think the weighted poll really addresses are feedback and the various forms of manipulation and cheating.
In forming a vote, the member has to consciously order based on preference. While they don't necessarily voice their reasoning, it does provide at least some information to the submitter as to where they stand.
An argument I can see for Unweighted/Infinite is that it is at least inclusive and less discouraging to someone who might not see their name appear much at all in either of the finite methods. However, I don't think the voting booths is the place one should be getting their critique or a self-esteem boost. Ideally, a submitter would be getting all of their feedback in the submission thread, and it would be great if, as a community, we tried to give feedback to everyone. Even a passing comment can do much to stave off discouragement and is worth a lot more than a throwaway vote amongst many that won't really mean much in the following polls anyway.
I don't know enough about how early polls were affected by manipulation or cheating, but the very fact that it requires a specific ordering and number seems like it could be worthwhile in deterring such actions. A person may be discouraged from having to make a similarly ordered vote as someone else, and it's much easier to spot trends in such a case anyway. At least recruited voters may have to make an informed decision on the rest of their vote if they were recruited for one specific submission.
The method may be more vulnerable to gaming, but I don't see this as a huge problem. Is it dishonest to vote for your second or third favorite option if your first option really doesn't stand a chance? People are going to want to get the best possible result, and it's unreasonable to suggest people shouldn't make an attempt at that if their ideal choice is no longer an option.
Cheating may be easier to spot by comparing the order of lists. But, I think it's a greater issue in the later polls. It may do something to help weed it out early, though, which I don't see either of the other two poll options doing.
___To answer your request:
Unweighted, infinite bold vote
In my opinion, this is perhaps the most manipulatable form of voting because it can mean 10 votes for a single entry. From my limited knowledge of how much cheating was done during CAP 11's art polls, this could allow the small minority of cheaters/manipulators to have a large voice in the polls. For this reason, this is not my choice.
Unweighted, finite bold vote
This is seemingly the most manipulation-resistant option presented here. By forcing the voter to include a certain amount of other entries in addition to their favorite, it prevents some aspects of cheating. However, this could be circumvented by voting for the least popular entries in addition to your entry of choice. In addition, this form seems to be susceptible to swarming and band-wagoning because it fails to measure the intensity of the voter towards its favorite. The vote of an apathetic is equal to the vote of a fanatic. For these reasons, this is not my choice.
Weighted, finite bold vote
This is my choice. It allows for voters to show the strength of their opinions while forcing them to consider other entries as secondary candidates, making it less influenced by manipulation. Mmm compromise. Please don't make the polls heavily weighted though.
___Unweighted, infinite bold vote: this poll allows each voter to submit an unlimited amount of votes on their ballot that each have the same weight. If someone votes for only three submissions on their ballot and another person puts seven on theirs, each of those 10 votes will count equally.
The current method being used, but not the best. It's effective in keeping check of cheaters, but the individual voter could simply pick anything he wants or even go as far as to give one vote to every single participant.
Unweighted, finite bold vote: This poll allows each voter to submit a fixed amount of votes on their ballot that each have the same weight. All voters must adhere to that fixed number on their ballot or their ballot will be invalid.
Yes - I think this one is the best out of the three. Not only does it allow the artists to see more accurate feedback through the individual voters ranking their options, it does not give cheaters an easier time like the third option does. It makes the voter pick his options more carefully and not blindly pick one that he simply maybe likes or something along the lines of that.
Weighted, finite bold vote: This poll allows each voter to submit a fixed amount of votes on their ballot, but in order of their preference to advance. Votes listed higher on the ballot have a higher weight than those listed on the lower end of the ballot. All voters must adhere to that fixed number on their ballot or that ballot will be invalid.
No - Sorry, but in my opinion the fact that the poll would be weighted means the problem of manipulation and cheating would be even more apparent. Although on the good side the individual would have to consider his votes much more carefully as his polls are going to be weighted, this also means the people cheating on the polls or manipulating them would have an easier time allowing the ones they want to win.
___I would prefer the unweighted, infinite bold vote over the other two options listed. Mainly, over the finite bold vote, this option allows more freedom for a voter to "tell the truth" about his/her preferences, while over the weighted vote, there's less of an ability to game the polls.
I believe I raised a concern that a voter listing more options amounts to inflating the impact of his/her vote, but now I'm not so sure about that. It may only really have been a problem if a voter were allowed to vote more than once on the same option. Now I see this as a like/dislike vote for each poll choice.
The infinite bold vote is the best option in terms of fairness. With the finite bold vote, there are problems with determining how many votes should be enforced. Additionally, the like/dislike mentality in the infinite vote is preferrable for the "underdogs" over the "vote the best options" mentality in the finite vote that may inevitably push underdogs out in favour of the veterans.
The infinite bold vote is also the best option in terms of feedback. Since not voting for a choice amounts straight up to an expression of dislike. Feedback doesn't matter during the actual polling, only before and after. If a submission is eliminated, then too many people disliked it, plain and simple. I'm just not sure that exact feedback is that important in early rounds. (I'm assuming that the last poll(s) will be single-click, but I suppose that's not necessarily set in stone...)
In terms of manipulation and cheating, in the end I don't see how the other two options would be a satisfactory improvement over the single-click poll. A sufficiently weighted vote verges on being the same as the single-click (albeit with better feedback, I suppose), while a weakly weighted or unweighted finite vote only partially solves the problem of not voting for one's most preferred options.
The unweighted, infinite bold vote is the best option out of the three for maximizing fairness and the inclination of the voter to express his/her desires both accurately and precisely.
___I am a fan of unweighted, finite bold vote. Unlimited votes is a clusterfuck, but weighted votes can shut out some cool shit. I think it is a good medium between the two.
Recruiting is allowed, idk why it wouldn't be.
Cheating is a BIG problem, as has discussed in the past. idk how to fix or mitigate it, but imo it is the worst issue with cap atm.
[Deck Note: IRV is short for Instant Runoff Voting. It is basically a single preferential ballot that goes through multiple “rounds” of preference eliminating the least popular option and re-appropriating its votes until a winner or slate of winners is selected.]I think that Unweighted, Limited / Unlimited Voting are pretty legitimate polling methods, although they're outdone entirely by IRV (Which CAP should absolutely be using to expedite the polling process). Weighted Voting is absurd and should never be used. There's literally no advantage in this method other than catering to dishonest individuals.
I'd also like to propose that we use IRV over all of these methods. We can leave voting open for a day and easily vote for the final winner all in one fell swoop. I'd like to hear Doug's opinion on this voting, however, as it means that we skip the visible poll stage, which admittedly is a bit exciting for participants. It's possible that he favors that more than expediting the process and eliminating skewing or cheating.
___
___I don't feel I'm qualified enough to provide good commentary on the possible choices because I almost never participate in the competitive portion of the project. But I will say that the way feedback is received differs between the Art and Competitive Polls. In the competitive polls the submissions are all constrained by specific parameters, and if someone doesn't do well in the polls they can figure out why by looking at the choices that scored higher than theirs. But in the art polls the submissions will differ so much that comparing your submission to the ones that received more votes won't help much. While the competitive polls one submission may win because it's a stat spread with 101 speed and is very bulky and another submission loses because it has 120 speed and is frail, in the art poll one submission may win because it is done in the same art style as the official Sugimori art and one loses because it looks like a western comic. The competitive polls can give feedback through comparison, but the art polls require more individual feedback.
___I definitely think out of all three of these options, the Unweighted, Finite bold vote is the best choice, followed by Weighted, Finite and then lastly Unweighted, Infinite.
Unweighted, finite voting requires users to select a certain one or two people (whatever is allowed) to represent themselves and their own personal opinion. By limiting the amount of choices one can actually choose from, I believe that it helps to better specify what individuals want in terms of the project itself. Furthermore, the fact that each vote counts as one vote and/or one point, not more or less, creates a much more level playing field for all candidates. One person means one vote, plain and simple. It places a fair limit on our own opinions, which is exactly what this project should strive for.
Weighted, finite voting accomplishes the same thing as unweighted, finite voting in terms of limiting the candidates available to vote on in one sitting. However, by placing more power in the position of voting, we have made the playing field less even. For example, assuming 1ist place gets three points, and third place gets one, it's possible for someone who gets 15 first place votes to beat someone who gets 44 third place votes, even though it's possible that these people actually like the third option very much. While some may argue that the result is still determined by popularity, it definitely puts some at a disadvantage, since name recognition and personal inflection may play a part in the ordering and position of voting ballots. I wouldn't be terrified if this goes through, but it's not the best option.
I am a huge disbeliever in letting individuals vote for as many choices as possible, as while there is a chance that some responsible users will tactfully choose (and not choose) certain people, it gives an outlet for people to just choose people "for the heck of it" and lessen the concreteness between a voter and his or her opinion. The unweighted, infinite voting style does exactly this. How can we accurately judge the opinion of the public when we have individuals choosing applicants for small reasons as a result of "just being able to"? It becomes less of a "This person is the best and should win." contest and more of "Oh, well, I like this too. And this. And this." kind of a vote. It's by far the worst of the three, and should not be considered.
___Re: Fairness: In my opinion, I think that the votes are slanted towards the more well known artists a tiny bit. However, it really isn't a huge difference, and no matter what approach is taken for voting, this is bound to occur. For example, if we were to make all entries and votes anonymous, a chain reaction is almost bound to occur when one artist tells one user which one they made. That user tells another, etc, and they all figure out who made what, and then the point is lost. So, I do think that the table is slightly tilted in favor of better known submitters, I don't think it's a terribly pressing problem, and nothing really practical can be done about it.
Re: Feedback: Between the visibility of the number of votes one user gets on click polls, and the Topic Leader announcing which submissions are allowed to advance to the next round, I think it's pretty easy to see where you sit rank-wise (top, middle, bottom). If someone got cut out fairly quickly, it lets them know well enough to try a little harder the next time. As for more advanced placement details, I'd rather we not go very much further than what is already in place. I know that this is much more important issue when it comes to bold vote polls, but it's not terribly hard to keep track of how many votes there are total, and how many are for the submitter. I guess we could have someone (probably the Topic Leader or Assistant Topic Leader) periodically post the numbers of votes and percentages overall, but that just seems like dumping more work on them when the people who are really interested could just do it themselves.
Re: Manipulation, Gaming, and Cheating: Cheating is a very straightforward one, and it is fairly obvious that anyone caught cheating will be punished. (From what I can tell, the moderators have been pretty effective at catching cheaters in the past, so I trust them.) Gaming is much more difficult to regulate, and there are few options for it, the best of which is simply encouraging people to vote for their favorite option, and putting good effort in keeping up a positive attitude throughout the entire project to stop any grudges that could cause it to occur. Finally, recruiting is definitely the hardest one to stop. Users with little to no experience in the CAP project are very easily persuaded to click a button (clicky polls) or post some bold votes (postcount+) when asked. On the other hand, things like disallowing votes from users with less than x number of votes will slow interest in the project, likewise, having a select group of experienced CAP members vote doesn't really reach CAP's goal of being a community project. With those things in mind, I can't think of a compromise, but I'm sure others will have, and I hope to see what they think.
Lastly, of the three choices of votes listed, I would like the Unweighted, finite bold vote to be used. I think both of the unweighted votes are superior to the weighted version, because the weighted version encourages slightly more gaming and recruiting than actual just picking the choice the voter thinks is the best. With unweighted votes, the user can vote for their desired choice(s) more freely without gaming being as large of a factor. I also chose the finite bold over the infinite one, because it can be a lot of work for the TL to count up every vote made when all the choices are simultaneously selectable.
Rather than tally any of these up, I present this to you as the feedback of those members who have taken the time to deliver quality contributions to CAP as well as the time to help out during this lull.
Particular notes for discussion:
Recruiting: It's always been a huge issue in polls and NO method of polling (including IRV) is immune to it. Where do we draw the line between trying to get support and actively seeking it at poll time. One of the better insights I thought was the notion that you get your feedback in the discussion and submission threads, the voting is not really that helpful in the grand scheme of things. If this is the case is there any way to aid that processwise?
Poll typing: I guess this was the whole reason for having the survey, but from what I've seen there's no clear-cut answer, and in the end it's only 10 responses to go on. I feel however that by selecting people who have done well (but not neccesarily won) it would control for if people feel slighted or thought a particular method they encountered was unfair. I'm willing to try IRV out as a "neutral option" if and when we get back on track. I am deeply appreciateive of the thought put into the responses.
Feedback Factor: This last one is especially important to pin down, since there's a great deal of divergence and different needs for the competitive and art portions. The way we've set up submissions and discussions has improved this I think, but I would be open to suggestions.
As always, thank you to everyone who still shows interest in CAP at this point. Sorry if it's been a little dead, I've been waiting for this to trickle in in order to spark your interest. If you happened to receive a PM and still want to respond, please feel free and I will add it to the feedback we already have.