Curious about what we're currently doing about bots? Check out the latest updates from staff in this thread.
Welcome to Smogon! Take a moment to read the Introduction to Smogon for a run-down on everything Smogon, and make sure you take some time to read the global rules.
A lot of people are taking this far too lightly, a bot that has even just a 35% winrate against a top player of any given tier is very considerably ahead of a good majority of players on here, this sort of thing being accessible or public would seriously damage the integrity of all tournaments in a completely irrepairable way.
if you all care about the future of this site you should start having a serious think about this because I think a lot of more extreme sounding options are still preferred to going into r1 of a tournament and having a real chance of being paired into a bot that has a 50% to end your tournament run no matter how good you are and that is nigh undetectable without human stupidity
Just a small thing for people to keep in mind: we were discussing on Discord what the hypothetical ceiling for the best Pokémon player/bot might be, and the truth is—and I think this opinion was widely shared—that the ceiling might not be that high.
Due to the nature of Pokémon battles—hidden information, RNG, simultaneous action selection—it might be the case that even a "perfect player" would not be at a typical so-called "superhuman" level. In games like chess, bots can keep improving until eventually they reach perfect play. In Pokémon, it's very possible that the ceiling for a bot is only slightly higher than the level of the current best players, while also being very prone to circular strategies: A beats B, B beats C, C beats A.
Just keep this in mind as you continue the conversation.
Who the bots can beat and how perfect they are or are not, or if they are still beatable or not doesn't matter, they are - or will be - good enough. Even an imperfect bot that can prevent you from doing simple misplays and guide you reasonably well through endgames is a fantastic cheating tool in the hands of a decent player. Smogon (and ladders) are full of mid players who have a *fine* understanding of the game, enough to identify why the bot says this move has that percentage and that move has this percentage, players who can act accordingly and be elevated a lot- perhaps to top level - by such a tool. Regular ghosting doesn't *guarantee* you wins either. It does give you an unfair advantage though!
And, like, I don't want battles to be "guy with AI vs guy with AI" either, but every solution that's been floated in this thread is way too draconian for a casual community like this. We're NOT competitive. Nowhere near enough for webcam nonsense. It would kill tournament participation. Or we could completely kill PS! ladders instead? No, as shite as things are going to be, I don't see a way forward that is not some sort of CalcDex-like widely avaitable tool, and better to encourage it being widely avaitable than it to be something only a few technically adept people & their friends have access to... and i guess get more gimmick heavy at team selection.
No, as shite as things are going to be, I don't see a way forward that is not some sort of CalcDex-like widely avaitable tool, and better to encourage it being widely avaitable than it to be something only a few technically adept people & their friends have access to... and i guess get more gimmick heavy at team selection.
or we could just like... not do that? people keep proposing this as though it's a solution and not a scale model of the death of civilization. if a handful of people found a way to bypass the validator and run flutter mane in ou, would we just unban flutter mane and go "well it's not a problem if everyone is using it"? no, we would find a way to either fix the validator or implement some sort of other system. the fact that people are proposing this sort of thing with a straight face is baffling to me
x-posting from a different venue since highly relevant.
As the person likely to end up implementing any bot measures, few thoughts. Not speaking as US/SS/whatever, just as someone who has handled building defenses against malicious users and bots for years now.
someone in other thread said:
3. Handling Bots in Tours
I think the only way forward for tours, (outside of making all tours be centaur which I find horrific), is to make a significantly stronger anti-cheat function on SmogTours. My ideal for this would be a function that does three things:
- Locks the screen to SmogTours when a game is opened. The only way to move to another tab would be to forfeit the game. This is a pretty common strategy in a lot of testing software.
- Disables plugins. Yes, we will have to get along without showdex. I'm not entirely sure how practically implementable this is, but if it can happen, it would be great.
- Check if instances of the same IP are open on main PS or SmogTours. This, combined with the public registry of bots, could allow us to catch a lot of cheating attempts.
There are still ways around this, as there always are, but this would cover a shit ton of the "easy" ways to cheat. It also doesn't require much more on the part of players and doesn't violate privacy in any new and horrifying ways.
This is completely infeasible, I'm afraid, on both an implementation level and in a macro sense. In the technical sense, the list is long. In the broad scheme of things, Smogon/PS works for competitive play because it’s so easily accessible, and making that more difficult is going to drain the number of new people coming into the community incredibly quickly.
More broadly:
The registry idea has come up a lot, so it bears addressing, I think, to serve as a general bellwether for the ideas against bots. It would be cool, it might help for those who are well intentioned, but it's not going to solve the problem of malice (I include tours/ladder tour cheating in this). Software devs are of a sort that tend to not consider consequences - so you'll have a fair few well intentioned devs who join the registry, but it only takes one who cares a bit less about consequences than about making cool shit, and this is completely screwed. If someone makes a browser extension, for example, suddenly everyone is able to cheat maliciously and we have absolutely no way to block against it. In the more short term sense, there are a fair few malicious users out there with technical expertise, who can and have botted to damage the outcome of tests and tours.
- IP detection - already sucks for proxies, entirely manual.
- captchas - easily solvable by bots nowadays, can just bypass by browser extension or merging connections.
- behavioral analysis - way too broad and anyone can add some timers to bypass it, see also browser extension. we can catch obvious bots doing obvious things, but that's still ultimately a manual and reactive measure.
- TLS cipher fingerprints - easily gimmicked around.
- screen lock - requires control a web app can't get without a custom browser.
- lockdown browser - see above.
- "well other games do it" - other games also have kernel level anticheat, like Easy Anti-Cheat and Riot Vanguard, and they both suck at their job and everyone hates them.
LIST NOT ALL INCLUSIVE, these are just the solutions that pop up on google so people don't miss the forest for the trees here.
it needs to be stressed: There is very little lasting effect we can make against these sorts of people. We can only ever handle the problem reactively, not proactively. As much as we can do with well-intentioned users, it's not going to fix the structural problems that any malicious user will be able to exploit. Reactive solutions will only take us so far, and will end up burning out the few users who have to moderate it largely by hand and eye. Thusly, we're going to get more out of looking at policy-level changes in how we do things, rather than hoping there's a technical solution to the problem. As the person who would be doing it, and who has done it for a long time now, I'm afraid to say: that solution is not coming.
The bot problem has been blue- and red-teamed into oblivion across the internet. It is a losing game. I can make many clever things, win small battles in the process, and ultimately still lose the war. What it boils down to is a game of cat and mouse, but the mouse is usually one step ahead. I can lay traps and some people will fall into them, but many will avoid them and many more will just start again knowing where the trap is.
you know what? maybe that's the answer: not to play. if there's no technical solution to this problem that can be implemented on showdown, then we simply stop using showdown and build something new
that might sound insane, but it wouldn't be the first time we've switched our primary battling venue, and i really do believe that in the absence of a technical solution the site's days are numbered anyway
you know what? maybe that's the answer: not to play. if there's no technical solution to this problem that can be implemented on showdown, then we simply stop using showdown and build something new
that might sound insane, but it wouldn't be the first time we've switched our primary battling venue, and i really do believe that in the absence of a technical solution the site's days are numbered anyway
Respectfully, how exactly would a complete ground up rewrite solve the problem? Bots are not a showdown specific issue, its an issue pretty much everything faces on the internet (different kinds and goals depending on the website/app/game/whatever). In fact, showdown has been lucky in the past that Pokemon has been difficult topic for bots to play well, we didn't really have to deal with them (in that regard anyways).
If you want to suggest something radical, fine. But I want to hear how it solves the problem and why its worth the cost of throwing out over a decade of work and requiring us to redo it.
This is true, but doing nothing is not an acceptable option, either. It is not safe to assume that bots are going to stay below human capability when designing policy.
I am less concerned with the ladder as I am for official tournaments. My assumption here is people have and will continue to try using bots no matter what to increasing levels of success. Even a small ounce of prevention will go a long way even if there is no cure.
Any suggestion of giving people access to some AI bullhonky that tells them the most optimal move should be shot down with extreme prejudice.
I play Pokemon for self-fulfillment. If I should lose to someone using a bot so be it. It's their consciousness they must live with. I'd rather lose with honor than win by cheating.
A loss to a robot may just be the greatest thing. A reminder that I am human
To preface, I'm not a tour player, nor a developer, nor do I have any frame of reference for how possible it is to implement any of what I'm about to suggest. I'm an artist and animator who's hobby has been "threatened" by AI for years rather than days, so I hope I can lend a bit of my perspective to this without being useless or reactionary or preachy.
What should actually be done right now:
- Fully support bots/bot-ghosting extensions on showdown. Perhaps make an "official" AI model that people can download and use as they wish. Advertise this on social media. Perhaps a certain AI bro poketuber could help advertise it too. People will whine about this, but it seems like it needs to be done this way if we want any means of detecting when people are using ai. If other models surpass showdown's, give them support and hope they want to "register" with showdown officially.
- Give people an option to not ladder against people who are using the bot extension(s). Right under "allow spectators". If not just to relieve the headache of "oh my god I don't want to lose elo to a fucking bot", then do it to give people the option to revoke consent for their battles being used as training data.
- Overhaul the ladder top 500 lists to indicate which accounts have used the AI extensions, and add an option to hide accounts that have used AI in games while viewing the top 500. On smogtours, perhaps have each turn cross-referenced against the supported model(s) to check for cheating.
I'm of the opinion that bad actors using an unauthorized ghosting bot on ladder are kind of a nothingburger, and I don't want to give it thought because that's exactly what's happening Currently on showdown and I would hope that it improves if what I'm suggesting is implemented. I'm also of the opinion that, in tours, where cheating DOES matter, we can only hope that the use of AI is seen as something serious that's actually worth reporting. If you're friends with tournament players or are one yourself, you know that Normal ghosting is generally Not reported. The number of people who are infracted for ghosting is simply the minority of whom were caught. I have no ghosthorse in that race, but to my point, the real "flaw" in my Very Beautiful And Otherwise Perfect Roadmap is that jerkcords can collude and use AI to cheat together and not get caught, but that's already a ubiquitous issue that everyone has gotten comfortable with.
This is obviously a monumental amount of work, but if you take a look at Any Industry right now you will see that rolling over and doing nothing will make things exponentially worse. Tough love, I guess, but I like this site and I'd hate to see it decline because we're too lazy to do anything about it.
I'm of the opinion that bad actors using an unauthorized ghosting bot on ladder are kind of a nothingburger, and I don't want to give it thought because that's exactly what's happening Currently on showdown and I would hope that it improves if what I'm suggesting is implemented. I'm also of the opinion that, in tours, where cheating DOES matter, we can only hope that the use of AI is seen as something serious that's actually worth reporting. If you're friends with tournament players or are one yourself, you know that Normal ghosting is generally Not reported. The number of people who are infracted for ghosting is simply the minority of whom were caught. I have no ghosthorse in that race, but to my point, the real "flaw" in my Very Beautiful And Otherwise Perfect Roadmap is that jerkcords can collude and use AI to cheat together and not get caught, but that's already a ubiquitous issue that everyone has gotten comfortable with.
congratulations on ragebaiting me into writing a gigantic wall of text (-1h30 of my life) however i deleted all of it because i decided the problems with this section are self evident
i do need to ask who are your friends who engage in "Normal ghosting" because we urgently need to get them off site. like sorry to be the bearer of bad news but we decided about 10 years ago that serial ghosting is actually extremely lame and worth year+ long bans, whatever your friend circle is up to is not normal and everybody else hates them for doing it and hates you for not reporting it
if your answer to bot cheating is "no big deal we're all already cheating anyways right" 1. no the fuck we aren't 2. if we actually are and i'm the weird one then yeah goodbye forever smogon it's been a good ride
To preface, I'm not a tour player, nor a developer, nor do I have any frame of reference for how possible it is to implement any of what I'm about to suggest. I'm an artist and animator who's hobby has been "threatened" by AI for years rather than days, so I hope I can lend a bit of my perspective to this without being useless or reactionary or preachy.
What should actually be done right now:
- Fully support bots/bot-ghosting extensions on showdown. Perhaps make an "official" AI model that people can download and use as they wish. Advertise this on social media. Perhaps a certain AI bro poketuber could help advertise it too. People will whine about this, but it seems like it needs to be done this way if we want any means of detecting when people are using ai. If other models surpass showdown's, give them support and hope they want to "register" with showdown officially.
- Give people an option to not ladder against people who are using the bot extension(s). Right under "allow spectators". If not just to relieve the headache of "oh my god I don't want to lose elo to a fucking bot", then do it to give people the option to revoke consent for their battles being used as training data.
- Overhaul the ladder top 500 lists to indicate which accounts have used the AI extensions, and add an option to hide accounts that have used AI in games while viewing the top 500. On smogtours, perhaps have each turn cross-referenced against the supported model(s) to check for cheating.
I'm of the opinion that bad actors using an unauthorized ghosting bot on ladder are kind of a nothingburger, and I don't want to give it thought because that's exactly what's happening Currently on showdown and I would hope that it improves if what I'm suggesting is implemented. I'm also of the opinion that, in tours, where cheating DOES matter, we can only hope that the use of AI is seen as something serious that's actually worth reporting. If you're friends with tournament players or are one yourself, you know that Normal ghosting is generally Not reported. The number of people who are infracted for ghosting is simply the minority of whom were caught. I have no ghosthorse in that race, but to my point, the real "flaw" in my Very Beautiful And Otherwise Perfect Roadmap is that jerkcords can collude and use AI to cheat together and not get caught, but that's already a ubiquitous issue that everyone has gotten comfortable with.
This is obviously a monumental amount of work, but if you take a look at Any Industry right now you will see that rolling over and doing nothing will make things exponentially worse. Tough love, I guess, but I like this site and I'd hate to see it decline because we're too lazy to do anything about it.
Advertising an AI client in order to be able to detect when they're using that exact software does not take care of bad actors. We can embrace and incorporate bot developers to best know how to target this problem, which is another reason why I think the bot registry on ladder is a great idea to at least discern who is willing to work within the frame of what is allowed on the site and help design measures that keep it safe and competitive. Don't advertise this shit while we can barely attempt to quantity it's impact. I'm surprised I even have to say this from an artist to an artist. Not all industries are being changed by AI in the same way and at least in the case of ours (art) we still have a choice and thankfully many choose with heart even if the effects have been rippling. But even when it comes to how we can approach policy on Smogon, the issue is way less black and white than this post makes it seem.
As for tours, we just need to treat AI assistance as ghosting. Talking to a lot of people this has become the clearest and easiest source of action, though not easily dealt with long term. The technology isn't there for people to take advantage of this consistently (players were already lining up to farm the most recent bot situation that happened in OLT), so for now utilizing this technology is like being ghosted by Django.
I don't think most people who are here to play competitive mons are exactly thrilled to have half of the game (the playing part) solved for you. We can work with bot devs and look at policy to ensure our games are competitive without embracing all of the shitty unregulated baggage that comes with what would constitute losing our hobby.
Reminder that the peak ladder bot peaked randbats ladder, which has a lot more games to work off AND also has all the sets public (at least to my knowledge). The lc ladder bot did also do decently well but 77 gxe isn't exactly great (for reference my chopped ass main is at 78.9 gxe, and I use a lot of meme shit like cetitan and yanmega on there.
I want to make something clear for people who share this sentiment. It’s understandable but it comes from a lack of information.
The accomplishment shared by freezai in the first post is actually fairly unimpressive, though it asks legitimate questions.
I would like to inform you that foul play is already good enough to get reqs in the OU ladder, and in fact managed to get reqs for the Roaring Moon suspect test a few hours after the deadline. The reqs were 1750 ladder and 80% GXE. So bots have been good enough to get suspect reqs for over a year now.
The fact that it had many failed runs and only succeeded after the test had ended should be taken into account, but we should think with this information in mind. Make of it what you will
We should probably just treat anyone using ai assistance in tournament as if they're ghosting, shit's kinda lame as fuck, idk how feasible developing a bot detector ala chess.com's would be but if it was feasible it'd likely go a long way to help with this
Recently bots have topped the VGC Bo1 and Bo3 ladders, top VGC players have challenged these bots and are already struggling to win. I have seen some discussion that I feel is myopic and needlessly hateful towards the enthusiasts working on these projects.
Firstly, it is not very helpful to use the blanket term 'AI' to refer to all Machine Learning. These projects are very different to LLMs like ChatGPT, Claude, and Gemini. Machine Learning has been around since the 1950s and has been ubiquitous for decades already, I am no expert but to my knowledge most of these bots will be trained with self-play in a simulator similarly to AlphaZero for Go/Chess.
There are many people who have attempted to create 'Pokemon engines' already, it is clear that they are not only getting stronger but more accessible. These projects are now feasible for someone working alone with a moderate level of technical proficiency and access to compute to complete. Given this, it is likely that there are strong Pokemon engines that have not been released publicly. A malicious user could soon be cheating with this technology, it is nearly impossible to detect or prevent this happening. We should be thankful that the most successful projects so far have been transparent and not used maliciously by their creators.
Many people have suggested banning use of Pokemon engines on ladder or in tournaments. But, besides for offline tournaments, how is this possible? Chess sites (chess.com and lichess.org) have dealt with a similar problem quite effectively, but many of the heuristics that can be used to detect cheating in chess such as suspicious time usage or 'non-human' moves, are not in my view likely to be possible in a Pokemon context. I'd be happy to be proven wrong about this, but as we have many talented players who already consistently lock in their moves extremely quickly and there are too few options for any move to be really suspicious, it seems like the main way to detect bots will be suspicious levels of activity on ladder, which leaves practically nothing to prevent use of engines in tournaments as they exist now.
With that said, I still don't think catastrophising is warranted. Chess engines have reigned supreme for almost 30 years, but chess is as popular as it has ever been albeit having changed in many respects to the pre-computer age. Competitive Pokemon may also need to adapt to the new reality, but this is not impossible. I don't have the answers about what should be done, but I hope we can use this moment productively instead of just doom-posting.
Chess engines have reigned supreme for almost 30 years, but chess is as popular as it has ever been albeit having changed in many respects to the pre-computer age. Competitive Pokemon may also need to adapt to the new reality, but this is not impossible.
Poor analogy. Computer assisted preparation in chess would be the equivalent of having a bot scout an opponent, prepare precise counter teams, and test them on ladder (or against other engines). Over the board play has historically lagged with catching cheaters and has had several famous scandals of strong players accusing other strong players of being cheaters without this ever being proven. This is worse online.
Online chess is currently going through something of a crisis of legitimacy that is managed in part through the use of live cameras used during games, with players repeatedly looking away from the screen being used as evidence of cheating. People have also used tools in other competitive gaming communities to the point where it is required that video recording of screens, hands, and keyboards are necessary to complement reported results. This is worse when there is a monetary prize involved. There is very little money in mons, but that's not why people cheat. That is, however, why some people participate.
In the current state of the game, it's very feasible for a player to use a bot to both prepare and play the games or even to simulate on the side without pressing buttons for the player. There's very much a cost of privacy to investigating or monitoring these behaviors. I'm no expert on these matters, so I'm not trying to offer an absolute perspective, but I don't think mons is equipped to actually prevent a motivated player from cheating.
