Other Trick Room website: a matchup analyzer and more to come

[Braindean]

Hey guys,

I'm sure we've all seen the bots around here recently. It may be kind of annoying to see bots on the ladder, but that's not all they can do. Pokémon bots, or machine learning models, can be used for a variety of tools now. The first and most obvious next step is a matchup analyzer. A bot can play games hundreds, thousands, tens of thousands of times faster than a human can. Thanks to that, a bot can test a matchup between two teams by simply playing the game out thousands of times to get a more precise evaluation of how good or bad a matchup is and how good each Pokémon is.

To that end, I have created trickroom.dev, a website where you can go and run matchup checks right now. The server will take any two teams you paste in and calculate the matchup. It will show you:

• Who's favoured, with a confidence range
• Which four to bring and who to lead, for each side
• Which Pokémon actually matter in the matchup, and how much each is worth
• How each Pokémon does against each of the opponent's likely fours

Here's an example. Feel free to check it out. I'm currently hosting this on one of my personal computers, which is why it may take some time to finish.

Uh, in case you're wondering why there's a bunch of grayed out navigation options at the top... I designed this site a while ago and was quietly working on it by myself for months with my own ML model. But then a lot of other people went public with their bots which forced my hand early. I have other features that I plan on releasing soon in the future on the site. The next one is the natural progression of the matchup analyzer: the meta analyzer, as I call it. Basically instead of just analyzing the matchup between just two teams, how about analyzing how well your team performs into the "meta"? I'll probably make another post when I finish with that one. There's even more than just that too.

And to other bot devs, if you're reading this: a matchup analyzer's accuracy is derived from the strength of the bot playing the matchup. If the bot cannot properly play the teams, then it is failing to represent the matchup faithfully. I've refrained from testing my model on the ladder, so I don't have an objective evaluation of its strength. It's decent, at least. But I have no doubt it is not the strongest model out there. If you have a very strong model and want to work together, send me a message.

If you have any feedback or questions, let me know. You can send me feedback through the box at the bottom of the site.

 

[Posho]

Hello! I have one concern about the creation of these bots. What is your involvement in the competitive scene of Pokemon? Have you played any high-level tournaments before? Your registration date on this website seems to be a little more than two weeks ago, so it's hard to discern what your intentions with displaying a tool of this caliber are.

You see... many tournament players and I are quite concerned about the fact that programmers with little involvement in the competitive scene seem to be churning out tools and bots like this en masse. I fail to see an angle here other than the desire to show off your tools. I would be less surprised if it was a renowned tournament player producing a tool like this through their input, but that does not seem to be the case here.

So, why should I trust an automating tool from programmers (as you aren't the only case on this forum) who have no experience on the competitive field? What's in it for you? I have read claims of people seeking to have players further understand this game, but once again, I refer back to the point most of you have little involvement on this community.

 

[Calmmind75]

I would have a genuine question, wouldn't this be considered as a way of ghosting?
As stated in the Tournament Rules and General Guidelines, the first thing they talked about in the cheating section is about ghosting, defining it as giving any piece of advice to someone playing a game, and that a player should be able to make their own decisions based on what they observe by themselves. Given the nature of this bot, which gives you data and leads/possible 4s that you/opp can bring seems like a form of external advice, and not a tool, especially since on the screenshot you put in your post there's a section called "plan", which given by how it sounds could just straight up give you the best plays to make.
Would there be any way of regulating this?

 

[DerpySuX]

Using something like this during a game would absolutely constitute as cheating tmk.

 

[ima]

The tools you're providing are skills a tournament player should already know/be applying before even stepping into a Turn 1 scenario, so while this may suit as "training wheels" for any player trying to understand preview analyzing I can't deem your bot is in good faith for our community as this is literally your only post on a website designed for Pokemon tournaments-- the concerns above me are more than valid to express themselves. Arguably, this is worse than the bots as the cheating factor adds up even more than fighting some scrub named "Freaki Nikki" on the ladder.

Yeah sure, I'm aware that AI is AI and the whole idea is that you can't really "stop" development (as a whole, not just Smogon which I believe can at least try to regulate these insane resources that can tamper very hard with tournament play) but the more these tools keep releasing on this site without any policy behind them (and i'm aware there are policies being made, but the first step only involved ladder bots, a tool like this released in unknown faith is obviously much more concerning for Smogon) you'll begin to notice that tournaments with no real regulation for these tools will make it a lot less fair and unique for others (a conversation that should be had in the near future), not to menton you also wipe out the character on Smogon in some shape/form.

my own rant about "character"

There used to be certain matchups such as ABR vs Empo, or Blunder vs Empo where you could theorize who would win in a BO3 due to their very unique playstyles -- obviously if we bring more AI into our tournaments the "character" will deteriorate and there will be more games where we wont know who's using an AI bot or not to help pick their moves, or win the preview matchup. I mean my whole point is basically the greatest players dont play like zombies, they are unpredictable and know how to gague their risk/reward very well. New players training under these bots will have a much harder time figuring out what this means in the future and its quite obivous.

I'm happy that Showdown is implementing these anti-bot features on ladder though on the other hand, that's a step in the right direction however you must do something about these bots for the Smogon community as well in the near future or tools like this that are created by someone who just joined the website will keep popping up and at a certain point it's just going to be genuinely cooked for the scene.

 

[Finabas]

Hi, very interesting project! I am not gonna comment on the morality of creating these tools but more on the technical side / usefulness. I do think with the publication of very strong bots the development of an evaluation tool is inevitable and I'd rather have it be public than in the hand of a few selected prep groups that would have a competitive advantage in preparing for IRL tournaments.

However I'm questioning if the metric and approach of your tool is actually useful for this since it (unless I am misunderstanding the quiet short documentation) just evaluates the winrate against random combinations of Pokemon from the other player. In reality not every combination of Pokemon, not even every combination of lead Pokemon is viable in VGC, so using this as a metric to evaluate how good a given gameplan is doesn't really work at all. You would want your gameplan to be the strongest against say the top 5 or top 10 possible gameplans from the opponent not against random combinations that are not good to begin with.

This image for example shows the suggested gameplans for the MU of the Venuzard sylveon team against Perish trap. I'm aware that most current bots are pretty bad at playing perish trap in general but as someone who has played and prepared perish trap for the Frankfurt regional championship the suggested gameplans for the perish side straight up don't work, and that is likely due to the fact that the metric of winrate vs random leads is just useless for teams with very defined modes (or in general).

Curious if I am missing something or if there are any ideas to fix this but in the current state the tool seems pretty pointless.

 

[vtrouter]

this is literally your only post on a website designed for Pokemon tournaments

Click to collapse...

I've been playing VGC since 2009 and Smogon is just simply not a useful forum for it. Maybe WAY back before skarmbliss or nuggetbridge were even things, but it's just not the environment for it. If I posted something it would look like I'm a random noob.

What's actually happening here, though, is that people are reluctant to tie these projects to their real accounts given the backlash. Dismissing these people as outsiders is probably inaccurate, and at best irrelevant.

I think it's most productive to genuinely think through these issues, as wishing them away is not going to be effective. I think negative feedback on tools like this is valid and I share a lot of the thoughts... I can understand that people have opinions on both the utility of a tool like this, as well as competitive integrity concerns (although those are kind of diametrically opposed--it can't be useless and cheating). However, some community members are just thinly veiling personal attacks and frustrated lash-outs as legitimate commentary.

 

[Braindean]

Hi, very interesting project! I am not gonna comment on the morality of creating these tools but more on the technical side / usefulness. I do think with the publication of very strong bots the development of an evaluation tool is inevitable and I'd rather have it be public than in the hand of a few selected prep groups that would have a competitive advantage in preparing for IRL tournaments.

However I'm questioning if the metric and approach of your tool is actually useful for this since it (unless I am misunderstanding the quiet short documentation) just evaluates the winrate against random combinations of Pokemon from the other player. In reality not every combination of Pokemon, not even every combination of lead Pokemon is viable in VGC, so using this as a metric to evaluate how good a given gameplan is doesn't really work at all. You would want your gameplan to be the strongest against say the top 5 or top 10 possible gameplans from the opponent not against random combinations that are not good to begin with.

View attachment 869849

This image for example shows the suggested gameplans for the MU of the Venuzard sylveon team against Perish trap. I'm aware that most current bots are pretty bad at playing perish trap in general but as someone who has played and prepared perish trap for the Frankfurt regional championship the suggested gameplans for the perish side straight up don't work, and that is likely due to the fact that the metric of winrate vs random leads is just useless for teams with very defined modes (or in general).

Curious if I am missing something or if there are any ideas to fix this but in the current state the tool seems pretty pointless.

Click to collapse...

Small clarification about some things, I should've been more clear. These suggested plans aren't ranked by win rate against random opposing combinations. Team preview is solved as blind simultaneous choice. Each side's plan weights ("Picked") come from how they do against the other side's best mix of plans, and that mix concentrates on the opponent's strong gameplans. Bad ones get close to no weight. So it is doing what you're proposing, roughly, just weighted continuously rather than over a fixed top 10. "Wins vs random" is only there for a quick sense of raw strength.

As for perish trap, yeah. I mentioned it in my post above, but a tool like this is reliant on the strength of the model it is using. The same model plays both teams, so the analysis reflects how the model plays each team. My model's decent, but it's not amazing. It's probably pretty weak at playing perish trap. It may be better to treat the suggested plans as "how an AI that doesn't fully understand this team would play it". Simply improving the model would likely remediate some of those issues.

This is actually the tool I feel most comfortable just releasing to the public as things stand. It's interesting to look at, and maybe there's some insight to glean about particular lead combinations, but that's it. I was aware that it mostly doesn't do anything for decent players who are familiar with their teams.

 

[ima]

I've been playing VGC since 2009 and Smogon is just simply not a useful forum for it. Maybe WAY back before skarmbliss or nuggetbridge were even things, but it's just not the environment for it. If I posted something it would look like I'm a random noob.

What's actually happening here, though, is that people are reluctant to tie these projects to their real accounts given the backlash. Dismissing these people as outsiders is probably inaccurate, and at best irrelevant.

I think it's most productive to genuinely think through these issues, as wishing them away is not going to be effective. I think negative feedback on tools like this is valid and I share a lot of the thoughts... I can understand that people have opinions on both the utility of a tool like this, as well as competitive integrity concerns (although those are kind of diametrically opposed--it can't be useless and cheating). However, some community members are just thinly veiling personal attacks and frustrated lash-outs as legitimate commentary.

Click to collapse...

Quite the contrary, and you've missed the point. Yes, I play a bit of VGC as well (anyone coming to Vegas in December?) and Smogon is obviously not The Forum for it-- but yeah if this tool exists right now for VGC then someone making this for Smogon tiers/singles will happen sooner than you think. The whole idea behind this project is by all means against tournament policy should it ever translate to anything smogon tournament related, which is more than valid reasoning to complain about the morals of this project. I don't see where anyone is attacking the poster though, and nobody is saying to "ignore it", so these are kinda just weird strawmans that kinda construe the entire subject

We should definitely do the opposite and .. not ignore what's happening. That should be a given. You and everyone else also have to understand that with AI, there will never be a perfect solution, as there's already been recent controversial takes being proposed to limit AI in some shape or form. If you want my opinion, I don't think we're going to have a real "solution" until things probably get a bit more out of hand. Every proposal has a hole in them right now and absolutely no one is gonna be on the same boat. We're very early into this and kinda seeing it all play out live so I dont really think anyone has a proper answer, and its likely nobody ever will have a perfect solution. After all, Smogon was never meant to be as serious as Chess where you are forced to put your cam on and play. It doesn't make sense for a website like PS, unfortunately. At the end of the day this is a Pokemon website and we have to solve things our own way.

What I think could happen right now? Obviously this subforum needs a bit more moderation (not to say they're slacking rn, but the fact you can post something like This with no approval is kinda funny) mostly because you never know what the next tool is going to be released next, at some point its going to be a situation where some of these tools just shouldnt be public and we're probably getting to that territory soon enough

I'm also not trying to encourage users Not to post, especially if they're actually new, but in a case like this where the poster hasn't posted once and then comes out of nowhere with a tool like this that is basically against tournament rulings (compared to a ladder bot where the worst thing it can do is probably similar to what we've seen from Mazar in 2017, so the case isn't entirely new in that aspect) you just have to question morals. And I don't think it was meant out of ill intent but in the future the moderation team here should be questioning if certain posts should remain public here or not.

Lastly it's definitely not a case of creators hiding their real alts, the whole point is to work with these creators anyway good or bad (and admins will know if you're alting anyway). It's important if you're newer to Smogon that you should be reading this thread about Pokemon Showdown and how the bots are being treated first before coming up with any argument. I mostly just wanted to point out that nobody is actively ignoring this or berating anybody out of "frustration", when a tool like this comes out of nowhere it's going to receive backlash like this, especially when the intent is unclear or can actively harm a tournament game in the future.

 

[vtrouter]

I mostly just wanted to point out that nobody is actively ignoring this or berating anybody out of "frustration", when a tool like this comes out of nowhere it's going to receive backlash like this, especially when the intent is unclear or can actively harm a tournament game in the future.

Click to collapse...

Yea, to be fair your post was pretty harmless, and not rude. Just using it to make a point against the weird "shun the outsiders" mindset that some community members have. It is present in other messages in this forum (and even in this thread) that are not so level-headed, and I think it comes from a weird place of elitism, which I view as its own problem. And in the case where these alts are actual players, may not even be justified.

What I think could happen right now? Obviously this subforum needs a bit more moderation (not to say they're slacking rn, but the fact you can post something like This with no approval is kinda funny) mostly because you never know what the next tool is going to be released next, at some point its going to be a situation where some of these tools just shouldnt be public and we're probably getting to that territory soon enough

Click to collapse...

I think this is the core question of "immediate action" that can be taken, but I'm not sure there's a clear answer. These tools are easy nowadays whether someone posts about them on Smogon, Twitter, Discord, or keeps it to themselves. I understand the case for limiting access as much as possible and probably would personally support an approval process for posts like these, but I'm not sure we're in the majority.

A single, popular tool like this, which can work with Smogon to maintain tour integrity would be much preferred to thousands of fragmented private tools that can't be policed, and discouraging sharing these with the public makes the path towards the first scenario unclear.

 

[Finabas]

Small clarification about some things, I should've been more clear. These suggested plans aren't ranked by win rate against random opposing combinations. Team preview is solved as blind simultaneous choice. Each side's plan weights ("Picked") come from how they do against the other side's best mix of plans, and that mix concentrates on the opponent's strong gameplans. Bad ones get close to no weight. So it is doing what you're proposing, roughly, just weighted continuously rather than over a fixed top 10. "Wins vs random" is only there for a quick sense of raw strength.

As for perish trap, yeah. I mentioned it in my post above, but a tool like this is reliant on the strength of the model it is using. The same model plays both teams, so the analysis reflects how the model plays each team. My model's decent, but it's not amazing. It's probably pretty weak at playing perish trap. It may be better to treat the suggested plans as "how an AI that doesn't fully understand this team would play it". Simply improving the model would likely remediate some of those issues.

This is actually the tool I feel most comfortable just releasing to the public as things stand. It's interesting to look at, and maybe there's some insight to glean about particular lead combinations, but that's it. I was aware that it mostly doesn't do anything for decent players who are familiar with their teams.

Click to collapse...

That makes a lot more sense yes, thanks for the clarification. I think tools like these can also be interesting for decent players once the model behind it is stronger.

 

[Drifting]

I'm not going to act like there aren't obvious benefits to this. In terms of helping prep specific problematic matchups this is almost everything I could have asked for. I'm Australian, which means people at my local events love using Kiran Singh's sunrain team, a very similar version of which just placed second at Louisville piloted by Andrew Figueras. If I could use a tool like this to help make a game plan for specific matchups like that before getting on the train to go play my League Cup it would genuinely be a lifesaver. In fact, often when I'm at an event I might go up to one of my friends between rounds, pass them my team sheet and ask a question like "if you were in my position, how would you lead against Ttar-Drill?", but I have the benefit of being an extroverted person who has connections from a long time playing Pokemon. A tool like this lets anyone have access to an advisor, and will hopefully spare my friends from being harangued with endless questions about someone else's hypothetical Hard Trick Room matchup, which they have very generously put up with :P.

But I mean like obviously this could be used to cheat online, it's so obvious. Just put in your opponents team during Team Preview and let the bot ghost you on ladder, in tournament, whatever. Especially in a Bo3 format where it doesn't matter if you lose Game 1 while the bot figures out how to play the game for you. It doesn't exactly take a Hans Niemann level scheme to figure out how to use this maliciously.

Maybe you could add an artificial time it takes for the matchup report to complete? Like if it took 10 minutes to show you the data or something, thats fine if I'm prepping for an event I'll just keep it open while I make a sandwich or watch a YouTube video or something, but it would stop you from being able to easily cheat in online games.

Again, this is really cool and I think from an in-person Pokemon perspective I'm excited that it exists. But I would really recommend you add some safety rails stopping this from being an easy tool for cheating in online tournaments.

 

[Braindean]

I'm not going to act like there aren't obvious benefits to this. In terms of helping prep specific problematic matchups this is almost everything I could have asked for. I'm Australian, which means people at my local events love using Kiran Singh's sunrain team, a very similar version of which just placed second at Louisville piloted by Andrew Figueras. If I could use a tool like this to help make a game plan for specific matchups like that before getting on the train to go play my League Cup it would genuinely be a lifesaver. In fact, often when I'm at an event I might go up to one of my friends between rounds, pass them my team sheet and ask a question like "if you were in my position, how would you lead against Ttar-Drill?", but I have the benefit of being an extroverted person who has connections from a long time playing Pokemon. A tool like this lets anyone have access to an advisor, and will hopefully spare my friends from being harangued with endless questions about someone else's hypothetical Hard Trick Room matchup, which they have very generously put up with :P.

But I mean like obviously this could be used to cheat online, it's so obvious. Just put in your opponents team during Team Preview and let the bot ghost you on ladder, in tournament, whatever. Especially in a Bo3 format where it doesn't matter if you lose Game 1 while the bot figures out how to play the game for you. It doesn't exactly take a Hans Niemann level scheme to figure out how to use this maliciously.

Maybe you could add an artificial time it takes for the matchup report to complete? Like if it took 10 minutes to show you the data or something, thats fine if I'm prepping for an event I'll just keep it open while I make a sandwich or watch a YouTube video or something, but it would stop you from being able to easily cheat in online games.

Again, this is really cool and I think from an in-person Pokemon perspective I'm excited that it exists. But I would really recommend you add some safety rails stopping this from being an easy tool for cheating in online tournaments.

Click to collapse...

Hmmm, I don't think it really provides necessarily an unfair advantage. I mean first and foremost, the tool is (partly intentionally and partly due to restrictions) inconvenient to use in a real time setting. You gotta input your team and then your opponent's team into the tool, and then wait like 2-4 minutes for the calculations to finish. Longer if there are other people using it. And then it only gives you numbers about how good the matchup is and which leads/reserves it thinks are good. It can't actually tell you how to play the game. Even if you were to spend game 1 in a Bo3 setting it all up, I actually think the tool is worse than a human at selecting the best leads for game 2 because the tool is incapable of spotting habits or playstyles in individual players. A human can adapt to their opponent, the matchup analyzer can't.

 

[idkup]

think we should be aware that just because some genuinely impressive models have appeared in recent weeks does not necessarily mean all of these models are going to be impressive.

i don't think a policy that can't cleanly put away random play is really cause for concern.

i would hope the community reads claims being made and assesses them on the merits instead of seeing the letters `AI` and going haywire.

 

[DerpySuX]

I think drifting was commenting more on the merit of something like this rather than the actual competency of the tool itself. Given an analyzer worth a shit it could very easily be used to find/form lines much faster than a human ever could. It’s better to encourage the implementation of anti-cheat failsafes now rather than waiting until we get something that truly is cause for concern.

 

[Drifting]

I think drifting was commenting more on the merit of something like this rather than the actual competency of the tool itself. Given an analyzer worth a shit it could very easily be used to find/form lines much faster than a human ever could. It’s better to encourage the implementation of anti-cheat failsafes now rather than waiting until we get something that truly is cause for concern.

Click to collapse...

Yeah, considering that the OP literally ends with a call to action targeted at strong bot developers to collaborate, I don't think it's a stretch to imagine a tool like this being able to run a search algorithm comparable to a Miku/Nessie in the not-so-distant future, as well as give more information on why a particular lead duo is deemed most optimal (e.g what buttons you could click Turn 1 based on what your opponent lead in response).

This tool is in its infancy now, which is exactly why I think we should be trying to anticipate what comes next, ask ourselves how something like this could be used to cheat in an online tournament, and brainstorm some safety handrails. I'm sure someone could develop a version of the tool without said handrails, but most people are just gonna use the thing that already exists and is convenient, so a few countermeasures put in place against malicious behaviour by a well-intentioned developer would probably go a long way to mitigate risk.

 

[Braindean]

I think drifting was commenting more on the merit of something like this rather than the actual competency of the tool itself. Given an analyzer worth a shit it could very easily be used to find/form lines much faster than a human ever could. It’s better to encourage the implementation of anti-cheat failsafes now rather than waiting until we get something that truly is cause for concern.

Click to collapse...

I believe anti-cheat methods are currently in development by the Smogon admins. Not sure where they are with that, but I think it's being worked on already.

 

[idkup]

Yeah, considering that the OP literally ends with a call to action targeted at strong bot developers to collaborate, I don't think it's a stretch to imagine a tool like this being able to run a search algorithm comparable to a Miku/Nessie in the not-so-distant future, as well as give more information on why a particular lead duo is deemed most optimal (e.g what buttons you could click Turn 1 based on what your opponent lead in response).

This tool is in its infancy now, which is exactly why I think we should be trying to anticipate what comes next, ask ourselves how something like this could be used to cheat in an online tournament, and brainstorm some safety handrails. I'm sure someone could develop a version of the tool without said handrails, but most people are just gonna use the thing that already exists and is convenient, so a few countermeasures put in place against malicious behaviour by a well-intentioned developer would probably go a long way to mitigate risk.

Click to collapse...

Honestly I'm very curious how this would go. Based on the information I have, a strong gameplay policy does not necessarily understand the game well enough to understand teambuilding decisions whatsoever (at least in singles). The path of least resistance to hitting #1 on ladder is to learn the common matchups, which doesn't really require an understanding of the game, just what actions win most often in a specific scenario. I suspect we're a bit further from this being an actual problem than people think... but that's not saying much.

I'd go into more specific detail about why I believe this but that's likely irresponsible.

 

[Jesuuuus]

While I appreciate that the model probably isn’t strong enough to be considered ghosting at the moment, would it be possible for the public to be able to see all the matchups being analysed, so that (in future if it did become really strong) if someone was using it in online tournaments it would be traceable?

 

[iforgetwhyimhere]

the public to be able to see all the matchups being analysed, so that (in future if it did become really strong) if someone was using it in online tournaments it would be traceable?

Click to collapse...

Depending on how this would be done, it could easily leak into free scouting material for users. If anyone doing tour prep used it and input their team, could their opponent not look them up? There are ways we could try to avoid this, but a basic search factor would not work without interfering with prep more than assisting it. There are also ways to avoid this situation, but it's something to keep in mind.

 

[Carrotman]

Depending on how this would be done, it could easily leak into free scouting material for users. If anyone doing tour prep used it and input their team, could their opponent not look them up? There are ways we could try to avoid this, but a basic search factor would not work without interfering with prep more than assisting it. There are also ways to avoid this situation, but it's something to keep in mind.

Click to collapse...

As far as i' aware, from the limited playing around i've done with the website, there is no sign-in button or username. So this would essentially be the same as all replays on ps being public, but without any usernames; you would have no information on who the replay is from. Is it marcofiero? Is it your tournament opponent? Is it an 1100 elo rando?

 

[PaperIrori]

You can spot the difference tho between a rando and a pro pretty easily. Even if you don't know it's specifically your opp, now you have a recent usage example of a team that you can account for. Worse if the team has a tech or meta development in general

 

[Braindean]

As far as i' aware, from the limited playing around i've done with the website, there is no sign-in button or username. So this would essentially be the same as all replays on ps being public, but without any usernames; you would have no information on who the replay is from. Is it marcofiero? Is it your tournament opponent? Is it an 1100 elo rando?

Click to collapse...

Yeah I didn't feel like making someone sign in. So all of the analyses are anonymously created. If I made all the analyses public, I guess it would be theoretically possible to infer some things about a user if you see five analyses in a row between one team and a bunch of popular teams, but that feels like a long shot.

 

[Jesuuuus]

Just for the record I wasn't thinking about a signup options where you could see all the teams specific people had been matchup testing, I was just meaning a big list with each item including a timestamp, and the two teams being analyse. That way if you suspect someone of ghosting this way in an online tournament, it could be checked this way.