Pending Registered bot Elo after the opt-in matchmaking change: evidence of a substantial population shift

[littlestar]

I previously posted about the difficulty registered bots have finding opponents after the new opt-in system. This is a separate follow-up about the rating behavior after the matchmaking population changed.

We have a frozen Gen 9 OU research bot, vadmiral_slowbro, that played both immediately before and after the Oct. 20 opt-in change.

Segment	Matchmaking	Games	W–L	Elo	GXE	Glicko	Peak Elo
Games 1–107	Before opt-in	107	79–28	1867	85.2	1830 ± 36	1895
Games 108–726	After opt-in	623	391–232	1637	79.8	1759 ± 25	1858

The most striking change is Elo. The same frozen bot went from ~1867 before the policy change to spending much of the post-change period around 1500–1700.

However, GXE and Glicko remain substantially higher than the displayed Elo.

The attached plots also show a broad drop in win rate across several opponent-Elo bands after the opt-in change, not just one isolated rating range.

I am not claiming the Elo implementation is broken. The obvious explanation is that registered bots now draw from a much smaller and different population: opted-in humans plus other registered bots.

My question is whether, under this restricted matchmaking pool, a registered bot's displayed Elo is still expected to be directly comparable to ordinary ladder Elo.

It may be worth investigating aggregate post-change data such as:

• size/rating distribution of the /allowbots population;
• opponent availability for registered bots at higher Elo;
• why Elo appears to shift much more than GXE/Glicko.

If this restricted population is structurally different, one possible solution could be a separate bot/opt-in ladder or separate bot-facing rating, rather than displaying the result on exactly the same Elo scale.

Happy to provide the game-level data and methodology if useful.

 

[fiish]

what is this suggesting

 

[DerpySuX]

bots shouldn’t be comparable to ordinary players, because they aren’t. According to this post, bots cannot affect the ladder rating of real players, and this is exactly how it should be. I personally am not a fan that bots can climb the ladder at all but I understand that a compromise must be made as to not encourage evasion of the registry.

Additionally, I’ve seen you call this a “research bot” in pretty much all of your posts now and I have to ask what exactly is the bot researching? I mean no disrespect by this, I’m genuinely curious. PS has stats for usage, team comps, etc built into it that anyone can access. Are you researching player behavior or what? Again I mean no disrespect by this I genuinely would love to know what this bot is aiming to get data for.

 

[littlestar]

Thanks — and no disrespect taken at all. That's actually a fair question.

Just to clarify the first point, I'm not suggesting that games against registered bots should affect the ratings of human players. I think the current protection there makes sense.

What I'm asking about is slightly different: now that registered bots only match against the opt-in population, does the bot's own displayed Elo still represent the same scale/population as ordinary ladder Elo or its own pre-change Elo? If the opponent population is substantially different, then a separate bot/opt-in rating may actually make the distinction you're describing clearer.

As for “research bot”: I'm not using it to research player behavior, usage statistics, or team frequencies.

The project uses Pokémon as a testbed for autonomous decision-making research. We've been experimenting with methods such as reinforcement-learning self-play, deliberately constructed opponent pools, lookahead/search, and distilling stronger search behavior back into a learned policy. A major question for us is whether improvements seen in controlled offline evaluation actually generalize when the agent encounters the much broader and less predictable strategies found on the live ladder.

Most of the scientific comparisons are done in controlled environments. Showdown is useful as an external evaluation environment — essentially a robustness/generalization test against strategies that weren't constructed by us.

That's also why I posted this particular thread. We have the same frozen agent playing immediately before and after the matchmaking change, so when its Elo distribution changes substantially while metrics such as GXE/Glicko move much less, it raises an evaluation question for us: did the agent get worse, or did the population represented by that rating change?

So PS's existing usage/team statistics are definitely useful, but they're answering a different question from the one we're studying.

Hope that explains why I keep calling it a research bot