Make Sense Of Your Shoddy Battle Rating

Discussion in 'Shoddy Battle and Pokémon Lab' started by X-Act, May 28, 2009.

  1. X-Act

    X-Act np: Biffy Clyro - Shock Shock
    is a Site Staff Alumnusis a Programmer Alumnusis a CAP Contributor Alumnusis a Researcher Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis an Administrator Alumnus

    Joined:
    Feb 17, 2006
    Messages:
    4,675
    This thread explains clearly what the ShoddyBattle rating is and what you can use it for. It is displayed in your ladder record here:

    [​IMG]

    As you can see, the rating has a lower value L and an upper value U. From these two values, you can find two very important numbers related to your rating, called xbar and sigma:
    Code:
     xbar = (U+L)/2
    sigma = (U-L)/2
    xbar is your mean, or average, rating. sigma is a number signifying the uncertainty of your rating. The larger sigma is, the more uncertain your average rating xbar is. For example, a rating displayed as being 1200-2000 is a much more uncertain rating than one displayed as being 1500-1700, even though both have the same average rating (xbar) of 1600.

    The xbar value increases when you win games and decreases when you lose them. However, it does not always increase or decrease by the same amount, but proportional to how better or worse your opponent is. To put it in simple terms, if your opponent had a higher xbar value than yours before the start of the battle, your xbar value would increase by a lot if you win against him, but would only decrease by a little if you lose. Conversely, if your opponent had a lower xbar value than yours before the start of a battle, your xbar value would increase only slightly if you win against him, but would decrease drastically if you lose.

    The sigma value always decreases whenever you battle, whether you win or lose. This is because the more you battle, the more the rating system gains accurate information of your Pokemon battling skills (or lack of) from the results of said battles, and hence the uncertainty of your xbar value decreases. On the other hand, if you stop battling, the sigma value would increase, indicating that your battling skills are now more uncertain, and this is where the volatility value comes in, since it is the value that governs the speed of your increase in sigma once a player stops battling.

    The volatility is a measure of how consistent you are in your battles. If you tend to win against players you are expected to win against (i.e. having an xbar value lower than yours) and lose against players you are expected to lose against (i.e. having an xbar value higher than yours), your volatility would be low. A consistent player would thus have his sigma increase slower when he stops playing, because due to his previously consistent performances, his xbar value should become more uncertain at a slower rate.

    Finally, a player is ranked on the ladder by his Conservative Rating Estimate (CRE). This is found as follows:
    Code:
    CRE = xbar - 4 * sigma
    The Conservative Rating Estimate rewards players that have a high xbar (average rating) and low sigma (uncertainty of their average rating), by assigning them a larger number, and hence a higher rank on the ladder. However, while this ranking system is adequate, it has a few disadvantages, and should be replaced by a better one soon.

    Earlier, I mentioned that a player is 'better' or 'worse' according to their corresponding xbar's and sigma's. In this final part of this thread, I shall provide a relatively simple approximation (compared to how it's actually calculated) to quantify this. It is written underneath in step-by-step form:

    Step 1: Find the sum of the squares of both players' sigma's, add 100000 to this sum, and find the square root of the answer.
    Step 2: Find the difference of the players' xbar's. Always subtract your xbar from your opponent's. Afterwards, multiply the result by 0.79 and divide by the number found in Step 1.
    Step 3: Raise 10 to the power of the answer. Call this number G.
    Step 4: For every game you win against the opponent, your opponent would win G games against you. Alternatively, the probability that you win against the opponent is 1/(1+G), while the probability that you lose against the opponent is G/(1+G).

    Here's the equation for G in mathematical form:
    Code:
    G = 10^((OppXbar - YourXbar) * 0.79 / sqrt(100000 + YourSigma^2 + OppSigma^2))
    Example: You have a rating of 1650 - 1738. Your opponent has a rating of 1580 - 1704.

    YourXbar = (1738+1650)/2 = 1694
    YourSigma = (1738-1650)/2 = 44

    OppXbar = (1704+1580)/2 = 1642
    OppSigma = (1704-1580)/2 = 62

    Step 1: 44^2 + 62^2 = 1936 + 3844 = 5780. Adding 100000 to this we get 105780. The square root of this number is 325.238.
    Step 2: 1642 - 1694 = -52. -52 * 0.79 = -41.08. -41.08 / 325.238 = -0.1263.
    Step 3: 10^(-0.1263) = 0.7477. This is our G.
    Step 4: For every game you win against the opponent, your opponent would win 0.7477 games. Alternatively, the probability that I win against the opponent is 1/(1+0.7477) = 57.22%, while the probability that my opponent wins against me is 0.7477/(1+0.7477) = 42.78%.
  2. StarmanXL

    StarmanXL

    Joined:
    Jul 1, 2007
    Messages:
    1,443
    Yay, it's AP Stats again!

    Thanks for the info. For some reason, I feel like doing confidence intervals or something...
  3. iKitsune

    iKitsune

    Joined:
    Jul 8, 2008
    Messages:
    805
    Does Shoddy Battle filter your search results based on your rating?
    ie. Are higher ranked players more likely to be paired up with higher ranked players
  4. Caelum

    Caelum qibz official stalker
    is a Site Staff Alumnusis a Smogon IRC AOp Alumnusis a Forum Moderator Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    Apr 5, 2008
    Messages:
    1,656
    I believe the system organizes all the players currently looking for a match and pairs you with the person who's CRE is nearest to your own.
  5. X-Act

    X-Act np: Biffy Clyro - Shock Shock
    is a Site Staff Alumnusis a Programmer Alumnusis a CAP Contributor Alumnusis a Researcher Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis an Administrator Alumnus

    Joined:
    Feb 17, 2006
    Messages:
    4,675
    I can't edit the OP for some reason, but there are some slight mistakes there. In particular, the formula for G contains "/100000" at the end which shouldn't be there.
  6. Son_of_Shadoo

    Son_of_Shadoo

    Joined:
    Apr 13, 2009
    Messages:
    2,849
    X-Act, you're a genius. My head would just explode if I tried to do half the stuff you do.
  7. d2m

    d2m

    Joined:
    Apr 18, 2009
    Messages:
    753
    How do you even view your own rating? I can see everyone else's but my own when I'm on the server :(
  8. A.P.

    A.P.

    Joined:
    Jul 14, 2008
    Messages:
    731
    Are you planning on implanting GLIXARE anytime soon, or only on SB2?
  9. tab

    tab ʕ•͡ᴥ•ʔ
    is a Community Contributoris a Team Rater Alumnusis a Forum Moderator Alumnus

    Joined:
    Mar 4, 2008
    Messages:
    913
    d2m, type /record <your name> into chat to see it.
  10. Blue_Tornado

    Blue_Tornado

    Joined:
    Apr 14, 2009
    Messages:
    3,200
    Wow, great work on these. This should be a sticky (or kept somewhere) for information, since new battlers to shoddy would love to know what the rating stands for.
    Really? didn't know that. ;)
  11. X-Act

    X-Act np: Biffy Clyro - Shock Shock
    is a Site Staff Alumnusis a Programmer Alumnusis a CAP Contributor Alumnusis a Researcher Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis an Administrator Alumnus

    Joined:
    Feb 17, 2006
    Messages:
    4,675
    Doug should be implementing GLIXARE soon on ShoddyBattle 1. When that happens, this thread would be updated.
  12. TheFourthChaser

    TheFourthChaser #TimeForChange
    is a Smogon IRC AOPis a Tiering Contributoris a Battle Server Moderatoris a Past WCoP Winneris a SPL Winner

    Joined:
    Oct 13, 2008
    Messages:
    1,464
    So would all current CREs and Ratings be reset when this is implemented? (Seems like a stupid question to ask but I'm wondering)
  13. tab

    tab ʕ•͡ᴥ•ʔ
    is a Community Contributoris a Team Rater Alumnusis a Forum Moderator Alumnus

    Joined:
    Mar 4, 2008
    Messages:
    913
    I think GLIXARE just works from the current rating system to give a more accurate rating to base the ladder on than CRE.
  14. Myzozoa

    Myzozoa Throw-up on the internet, or get off on TV
    is a Tiering Contributoris a Past WCoP Winner

    Joined:
    Jan 31, 2009
    Messages:
    1,416
    I apologize if this question has already been answered:

    When GLIXARE is implemented will our ratings be reset?
  15. Cathy

    Cathy

    Joined:
    Jul 11, 2007
    Messages:
    1,062
    Tab already answered it above, but I will explain for greater certainty

    The rating system -- Glicko2 -- is not going to be changing any time soon. However, Glicko2 does not assign each player a single figure which can be used to rank the players; it is necessary to impose some total ordering, and the way we have accomplished that up to this point is by introducing the CRE. The predicative percentage win metric ("GLIXARE") is just a quantity used to impose a different total ordering on the set of players -- your rating is completely unaffected; the only thing that is changed is the ordering of the leaderboard.
  16. X-Act

    X-Act np: Biffy Clyro - Shock Shock
    is a Site Staff Alumnusis a Programmer Alumnusis a CAP Contributor Alumnusis a Researcher Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis an Administrator Alumnus

    Joined:
    Feb 17, 2006
    Messages:
    4,675
    Just to confirm what Colin and Tab already said...

    There's no need to reset the player's ratings when GLIXARE replaces CRE. Contrary to what many people seem to think, GLIXARE is NOT a rating system. It is just a way of interpreting your xbar and sigma as a single estimate for your rating... a way that is much closer to your actual playing strength than CRE. That's all. Your xbar and sigma will not be affected at all when GLIXARE is implemented, and hence the player's ratings do not need to be reset when it is implemented.

    Note that this doesn't mean that the ratings won't be reset - it just means that they don't need to be reset. If we decide, for whatever reason, to reset the players' ratings at any time, we'll do it. Moreover, a resetting of the player's ratings can happen even without a change in the rating estimation method.
  17. paramylodon

    paramylodon

    Joined:
    May 24, 2007
    Messages:
    238
    Well, I'm looking forward to seeing if my alt stays on the leaderboard or not once GLIXARE is implemented.

    Thanks for the information. I knew the basics (what things mean), but not how to actually calculate expected win rate.
    I also didn't know what volatility was based off of.
  18. DougJustDoug

    DougJustDoug Knows the great enthusiasms
    is a member of the Site Staffis an Artistis a Programmeris a CAP Contributoris an Administratoris a Battle Server Admin Alumnusis a Smogon IRC SOp Alumnusis a Tiering Contributor Alumnus
    CAP Leader

    Joined:
    Jun 26, 2007
    Messages:
    2,991
    Now that I have finished the Shoddy client update, and I've gotten May stats published -- I'm close to putting GLIXARE on Smogon University and CAP. You will see some public announcements when that happens.
  19. VKCA

    VKCA (Virtual Circus Kareoky Act)

    Joined:
    Jan 8, 2009
    Messages:
    3,638
    Nice man you read my mind, I was literally about to go on the server and ask what the differences were. Thanks alot!
  20. odisseas

    odisseas

    Joined:
    Jun 5, 2009
    Messages:
    48
    Thanks X-Act that just perfectly explains a lot! And i also have to imrove my rating now that i have learned how to find it!
  21. Kingdrom

    Kingdrom

    Joined:
    Oct 18, 2007
    Messages:
    381
    Thanks for explaining this X-Act! Do these rankings (xbar and sigma) determine how much your rating (and therefore CRE) changes with each battle?
  22. X-Act

    X-Act np: Biffy Clyro - Shock Shock
    is a Site Staff Alumnusis a Programmer Alumnusis a CAP Contributor Alumnusis a Researcher Alumnusis a Tiering Contributor Alumnusis a Contributor Alumnusis an Administrator Alumnus

    Joined:
    Feb 17, 2006
    Messages:
    4,675
    Well, they are not rankings, they're just values used to determine how good or bad you are. The xbar and sigma values and those of the opponents you play against are all used to determine how your rating changes.
  23. Cathy

    Cathy

    Joined:
    Jul 11, 2007
    Messages:
    1,062
    A lot of people ask me how much their CRE will decrease by if they don't play for a day. The answer is that your deviation changes (approximately) according to this formula:

    new deviation ≈ sqrt((old deviation)^2 + 30178 * (volatility)^2)

    Strictly speaking, your deviation increases by this much each day whether you play or not, but if you do play, the decrease from the games you play offsets it. However, after you play your first battle in a day, your CRE may sometimes decrease, and that's because you have not offset the daily deviation increase yet, which is applied in the rating window after your first battle in a rating period. After you play more in the same rating period, you will always experience a net increase in CRE.

    Here is a fully worked out example. Suppose a user has these stats:

    CRE: 1449
    Rating: 1521-1569
    Volatility: 0.051068226...

    What will his CRE be if he doesn't play for a day? First work out his deviation and mean rating:

    mean rating = (1521+1569)/2 = 1545
    rating deviation = (1569-1521)/2 = 24

    Now apply the formula above:

    new deviation = sqrt(24^2 + 30178 * 0.051068226^2)
    => new deviation ≈ 25.587167266882

    The mean rating is unaffected, so we use the formula for the CRE to find his new CRE.

    new CRE = 1545 - 4 * 25.587167266882 ≈ 1443

    So by not playing for a day, this user's CRE will decrease from 1449 to 1443. This should give you a tangible idea of what to expect with the daily deviation increases.
  24. PI-Dimension

    PI-Dimension

    Joined:
    Jul 31, 2009
    Messages:
    254
    I just have to say one thing, i'm disturbed that its not in LaTeX!

    Other than that, this is amazing! I always wondered what CRE really meant. It took me a while to get the percentage stuff, but it makes sense now.
  25. SkyDragon

    SkyDragon

    Joined:
    Sep 20, 2009
    Messages:
    7

    Go to the memberlist, search for your name, (the names are in alphabetical order) and then right click and click 'View Record''

Users Viewing Thread (Users: 0, Guests: 0)