• Follow our Instagram!
  • Curious about what we're currently doing about bots? Check out the latest updates from staff in this thread.

Battle Bot Jaxcalibur: A Gen 9 Randbats bot that reached #1 on the ladder

Imagine pulling up to VGC Worlds using a bot to memorize the all the best possible opening moves based on open team sheets and whatever potential 2 mons ur opponent throws out. Bots that are good enough to beat humans consistently are genuinely an existential threat to mons.
 
A battling bot with the capabilities to play at the level of a rank 1 ladder player released to the masses will harm competitive tournament singles. It's not going to be like chess, Singles has few real decision points, hidden information, and RNG, so an engine's picks are indistinguishable from a strong human's on most turns. A bot doesn't need to be better than the person using it, it can still be a massive advantage for top players since it can give them paths they otherwise did not see, even if you're better than the AI, you're not going to make a better move every time.
 
Do you have a next step in mind with the source code you've cooked? For example, tweaking it a bit and using that ability to get positive/negative feedback from replays for research purposes on regular tiers. (eg. to find out what makes a good team, which would make for a very desirable helper)
 
Crazy to think what technology is capable of, though honestly i'm surprised this hasn't happened earlier... though perhaps it isn't so suprising considering PS! and specifically randbats is quite niche in the grand scheme of things when comparing it to e.g. chess.
While certainly impressive work, I too agree with the (slight) majority of people saying to not make this open-source and would still be critical of it if there was better detection, as, to quote mien, "it can still be a massive advantage for top players since it can give them paths they otherwise did not see" and could be run on the side and not directly, making possible bot prevention futile in limited bot use case scenarios.
I see you have responded very sensibly by saying you wouldn't release jax without a consensus (thx for that) so these are just my 2 cents.
 
Please don't release this bro. Releasing this would not benefit the community AT ALL. The point of competitive pokemon is that you play against other humans, and releasing a bunch of high quality bots on ladder would fundamentally change the game people are playing. This would be even worse for tour. To the people saying releasing it would force smogon to create policy that combats this new form of cheating, be so fr. Give me 1 solution smogon could implement that could possibly prevent ai cheating. We already have no real way to police ghosting, but at least with human ghosting one of the players has to actually be good for it to make a difference. With this anyone could get high quality ghosting with zero threat of the other person exposing them or even the need to convince them in the first place.
 
Last edited:
Realistically:

This stuff will get easier to make and more powerful as time goes on. It’ll absolutely destroy the integrity of the tour scene of the entire site if we don’t come up with ways to prevent its use in tours

For that reason, holding off on making this available to the public would be best for the tour scene, since it might give us time to adapt and come up with countermeasures (if those are even achievable), without immediately throwing ongoing and upcoming tours into disarray

It’s very helpful that you posted this, since it lets everyone know that we need to work on anti-cheating solutions ASAP. The longer we have to work on those, the better. I hope you avoid making this open source, and I hope you also update us on significant improvements to the bot (I hope anyone who makes a similar bot will do the same)
 
Hello, fellow open source developer here.

I absolutely think you should open source this!
...Later. Smogon needs to come up with some kind of policy response first. But I think FOSS is the morally correct thing to do.

I am also 100% certain that other people will make engines like this regardless of what you do, so you may as well. Many people in this thread are deluding themselves into thinking that they can peer pressure people out of working on AI tools, or better, peer pressure people out of cheating at Pokémon. The truth is that, probably within the next year or two, there will probably be several people with local engines that are roughly this good. At least one will probably open source. Probably some will be kept secret, for any number of reasons.

Finally, I would encourage you to announce your release timeline well in advance, and be in touch with site leadership about it. This would be good for the site, and also good for you/your project. Open-sourcing plans and Smogon policy both frequently suffer from ambiguous goals, scope creep, and the sensation of "I'm so busy, and this is just a hobby project, but I'm not going to release until everything is perfect." This eventually turns projects into vaporware. Having a deadline you're working towards keeps you accountable to another person. It doesn't need to be a mutually assured destruction thing, just a soft target.
 
I don't believe in this community's ability to figure out a solution to the problem without a strong bot actually roaming free. This bot is not an immediate threat as it's not good at non-random formats and if you super deeply care about the competitive integrity of random battle tournaments then you should be playing a nonrandom format instead. Releasing it now allows us to prepare for it. If you are anti-releasing the bot you are anti-vaccine.

In all seriousness I do recognize the argument that the bot's code or w/e could be used to help someone else build a stronger bot that would ruin competitive integrity much sooner, but I feel like many people are against releasing it not because they are thinking about this future consequence, but rather because they are having a knee jerk reaction because they think a bot that can hit #1 on the ladder (in ranbats!) is bad for the integrity of the game and then maybe they will later justify it by talking about the future. The bot is an incredible accomplishment of the author, but its not a threat to serious pocketmonstering.

I'm not of the belief that releasing this bot now will lead to the quicker development of a truly crisis level bot at least not by more than a few months on average. I do think forcing the problem to actually be managed now will set us a year or two ahead. I believe the former because the technology behind the development of bot will grow exponentially making former progress less important. I don't know shit about computers though so if an expert wants to correct me I'd appreciate it.

I'd like to reiterate that SPL should have 1/2 VGC slots and if 1 then another doubles slot.
 
ArcticBreeze said:
I don't believe in this community's ability to figure out a solution to the problem without a strong bot actually roaming free. This bot is not an immediate threat as it's not good at non-random formats and if you super deeply care about the competitive integrity of random battle tournaments then you should be playing a nonrandom format instead. Releasing it now allows us to prepare for it. If you are anti-releasing the bot you are anti-vaccine.

In all seriousness I do recognize the argument that the bot's code or w/e could be used to help someone else build a stronger bot that would ruin competitive integrity much sooner, but I feel like many people are against releasing it not because they are thinking about this future consequence, but rather because they are having a knee jerk reaction because they think a bot that can hit #1 on the ladder (in ranbats!) is bad for the integrity of the game and then maybe they will later justify it by talking about the future. The bot is an incredible accomplishment of the author, but its not a threat to serious pocketmonstering.

I'm not of the belief that releasing this bot now will lead to the quicker development of a truly crisis level bot at least not by more than a few months on average. I do think forcing the problem to actually be managed now will set us a year or two ahead. I believe the former because the technology behind the development of bot will grow exponentially making former progress less important. I don't know shit about computers though so if an expert wants to correct me I'd appreciate it.

I'd like to reiterate that SPL should have 1/2 VGC slots and if 1 then another doubles slot.
Click to expand...
Click to collapse...
For adapting to another format, competent people that want to use their $10000 GPU or rent one for $500 for a week can already use the technical writeup to get most of the way there (even if you had the full source code / weights it would still require a huge rewrite). There are probably a few other nontrivial ideas that might be tricky to figure out on your own, I don't know. In any case, I think even a few months might be an overstatement for how much it would matter. It's also not obvious that these methods necessarily will be as effective or cheap in other formats (I suspect they might be roughly same order of magnitude but slightly worse)

For those without those resources, they won't be able to use these methods directly beyond putting the bot onto randbats ladder/tournaments. Maybe make a few improvements outside of the nets. But it's unlikely to result in a massive improvement and won't transfer to other formats.
 
To a certain extent, stuff like this is like Leeroy Jenkins. When it's out, it's out, and no amount of careful preparation is going to prevent it from happening because it takes one person (even if it isn't you) releasing something like this to trigger all the irreversible consequences. I agree with Blizihguh that this should absolutely be released after a delay to allow the site to get its act together policy wise. Because that will have to happen, site staff are deluding themselves if they think they can kick the can down the road any further on this issue. Once those policy issues are resolved, it is far better for an entire community to have access to the tool than for a select few who are able to make their own or have connections to have an advantage over the rest.

In addition, projects like this of course are of significant academic interest in machine learning, game theory, and the theory of pokemon battling itself. In games where engines exist that can beat top players, those top players learn from engine play and the communities surrounding those games learn more about the game itself. I respect pokemon as much as any game which has received significant academic interest and want it to be studied and mastered with the same rigor. I want to live in a world with better bots for more formats and where humans have better play informed by those bots. I want to see how the techniques used to optimize these bots can be employed for similar games with hidden information, risk management, and a teambuilding/deckbuilding component. And I want to see what legitimate top level competition looks like after those lessons have been learned. As a mathematician who has followed this kind of research for years I appreciate what you've done and will be reading your materials. Thanks for your contribution!
 
jaxcalibur_author said:
Definitely! It also made writing the engine easier because there are fewer pokemon/moves/abilities implement. For the same reason, I think bots would be especially strongs in open-team sheet formats because they could search ahead more easily with fewer unknowns.
Click to expand...
Click to collapse...
May I ask what degree are you studying/have studied in university? Or what field you're working in etc

edit: I ask because I want to learn and try to get into this space aswell, so im trying to gauge what kind of level of fluency do i need to reach this level
 
Last edited:
this is awesome and your work is awesome. i think you should open source it because every program should be open source, but I think it'd be nice to try and get anyone to do anything about the bot problem by holding this over their heads lol. maybe a few months of holding back

if people keep kicking the can on the issue though you might as well release it, no fundamental difference between you doing so and someone else deciding to make one and releasing it without contacting smogon first because they dont care about the integrity of unnofficial pokemon competition. If smogon can only act when things are catatrosphic, then just rip the bandaid and let it happen now than in one year where there'll be 10 bots instead of 1
 
Really impressive work — I read through the technical write-up and had a few questions about the implementation and how well the approach might generalize.

One thing I was especially curious about is how you validated the correctness of the JAX battle engine against the actual Showdown engine. Since the self-play training happens inside your reimplementation, what kind of validation did you use to make sure the two environments were sufficiently consistent? Did you use differential testing, replay-based comparisons, or some other approach? Given the long tail of unusual move/ability/item interactions in Pokémon, I imagine even fairly rare discrepancies could potentially create some interesting simulator mismatch at this scale.

I'm also curious about the auxiliary opponent-action prediction head. You mentioned that it made the bot substantially stronger — did you ever run an ablation to get a rough idea of how much strength that head added on its own?

Finally, how well do you think this overall approach would generalize to other formats, especially teambuilder formats such as OU? Randbats still has substantial hidden information, but the distribution of possible sets and team compositions is relatively constrained and learnable compared with formats where players build their own teams. Do you think the main difficulty in extending Jaxcalibur would be modeling that much larger hidden-information space, or are there other parts of the training/search setup that would become the bigger bottleneck?

Amazing project either way, and thanks for writing up the technical details.
 
On the open-source question, I think I would personally be cautious about releasing the trained weights, at least for now.

The distinction between releasing the research/code and releasing the trained model seems pretty important here. If the weights let a ~2400 Elo version run on an ordinary laptop without search, then releasing them would make practical ladder or tournament assistance much easier, and that seems difficult to undo once the files are public. At the same time, I think there is a lot of research value in the engine, training setup, hidden-information modeling, and search work, so I would be reluctant to see the entire project remain closed indefinitely. A staged release — sharing more of the methodology, evaluations, and perhaps parts of the code while withholding the strongest trained weights until the community has clearer norms — seems like a reasonable middle ground to me.

I was also really interested in your comment that smaller networks trained for longer outperformed larger networks trained for fewer steps. Do you have any sense of how close the final ~100M-game run was to a plateau?

For example, was the base policy still improving noticeably between something like 60M, 80M, and 100M games, or were the gains already becoming quite small near the end? And how were you measuring improvement between training checkpoints — head-to-head games against previous checkpoints, a fixed pool of opponents, an internal Elo, or something else?

More broadly, do you have an intuition for whether the current 8.5M model is still mostly compute-limited, so that simply training it for substantially more self-play would give meaningful gains, or whether you think model capacity would become the more important bottleneck?

I'd be really interested to see a training/strength curve if you have one, since it would give some sense of how much headroom might still be left beyond the current model.

I also had two smaller questions about the architecture/search details.

For the 24 matchup tokens, you mention matching each of your six Pokémon against four of the opponent's rather than using the full 6×6 matrix. How are those four opponent Pokémon selected when more than four are revealed and still relevant? And did you ever compare the 24-token setup against using all 36 matchups to see how much playing strength, if any, was being traded for the efficiency gain?

Similarly, how did you arrive at 32 possible worlds for the hidden-information search? Did you experiment with values such as 8, 16, 64, etc.? I'm especially curious whether, under a roughly fixed search budget, you saw a meaningful trade-off between sampling more possible worlds and spending more rollouts searching each individual world.

Thanks again for the detailed write-up — there are a lot of really interesting design choices here.
 
Last edited:
NotVeryCake said:
Awesome tech as a showpiece. Please never release this. We don't need a cheating epidemic across all of singles. I'd go so far as to say there should be serious punishment if this gets released since it compromises the entire site.
Click to expand...
Click to collapse...
What are U gonna do?
IP ban him?
Like seriously.
Instead of getting someone to the table and talking about measurments on how to police bot usage your first thought is to threaten them?
Y'all are just a bunch of regular people on a nieche internet forum.
You got 0 power

They coded a bot strong enough to climb the randbats ladder. Teenagers are able to evade smogon bans.
Go figure why that wouldn't stop them.

It is only a matter of time until somebody will cheat using a bot.
The only reason nobody has done it so far is that there is no real money to be earned playing on the site & that there are ~ 100k ppl using the site and even fewer playing tournaments.
Or hell far all we know somebody already won a tour using a bot and we will never know.

Try talking to the expert on what countermessures U might be able to take as they are clearly willing to enter discourse.
 
ArcticBreeze said:
I don't believe in this community's ability to figure out a solution to the problem without a strong bot actually roaming free. This bot is not an immediate threat as it's not good at non-random formats and if you super deeply care about the competitive integrity of random battle tournaments then you should be playing a nonrandom format instead. Releasing it now allows us to prepare for it. If you are anti-releasing the bot you are anti-vaccine.

In all seriousness I do recognize the argument that the bot's code or w/e could be used to help someone else build a stronger bot that would ruin competitive integrity much sooner, but I feel like many people are against releasing it not because they are thinking about this future consequence, but rather because they are having a knee jerk reaction because they think a bot that can hit #1 on the ladder (in ranbats!) is bad for the integrity of the game and then maybe they will later justify it by talking about the future. The bot is an incredible accomplishment of the author, but its not a threat to serious pocketmonstering.

I'm not of the belief that releasing this bot now will lead to the quicker development of a truly crisis level bot at least not by more than a few months on average. I do think forcing the problem to actually be managed now will set us a year or two ahead. I believe the former because the technology behind the development of bot will grow exponentially making former progress less important. I don't know shit about computers though so if an expert wants to correct me I'd appreciate it.

I'd like to reiterate that SPL should have 1/2 VGC slots and if 1 then another doubles slot.
Click to expand...
Click to collapse...
I don't entirely disagree with the sentiment but i feel just releasing it, no holds barred, is definitely still a dengerous idea. Pokémon is not new to people barging into its communities and absolutely ruining the hobby for people just chilling on their own time. Just look at what scalpers(of more recent note people like Jake Paul) have done with the TCG. Releasing this to the masses would attract a possibly even worse crowd (AI Bros, "Tech Startup" guys looking to have fun fucking with people) into the place, and it cannot be understated how those types of people will be a complete net negative to the environment, both figuratively and literally.

I'd suggest releasing a build of it to a dedicated counsil so they can analyze specific behaviours between the bot and a random regular player, so they can study how it works and create proper countermeasures, if it's something that should be released at all.

Personally? We need to lock the code for this down and throw the safe into the ocean. I do not want to experience a fucking TF2-level Bot Crisis on Showdown of all places, it's not gonna do anybody any real good.

Also, holy False Equivalence Fallacy, Batman. If you really wanna compare it to something more palpable, it's like a gun(though that only works if where you're from is smart enough to have gun control laws in place). It can "maybe" be used for "good", but if it's made readily and pubilcly available, a lot of very bad people are gonna use it for very bad things.
 
Fantastic questions Xuwu, pretty much had the same thoughts and questions after poring over the writeup. Would love to see a full paper if that's in consideration. Other than more methodology specifics and possibly code, I'd like to see the game engine be open sourced; it's probably the most impactful innovation in isolation and would open up a lot of design space. Others would be able to actually target scalability as an improvement vector without being gated by throughput on the current CPU bound simulators (roughly 14x more throughput than what I can get currently using Foul Play's engine for reference).

I think closed weights is a no-brainer for now and until reliable bot detection gets developed. But I think if prepared for properly, bots can definitely revolutionize the game at the highest level akin to what it has done for Chess and Go, which personally would be incredibly exciting.
 
You must log in or register to reply here.
Back
Top