Stockkarp - a VGC Game Review Tool
Public beta available now (sign-in required) - https://stockkarp.com
Stockkarp is a replay review tool for VGC games played on Pokemon Showdown!. Stockkarp takes a replay and gives each board state and player decision a valuation, telling you who was winning each turn, which plays were strong, which plays cost you, and how alternative lines might have looked. It's heavily inspired by chess.com's game review tool.
How to Use
Brower Extension
Chrome: Stockkarp for ShowdownFirefox: Coming Soon
- Add the extension.
- Click its toolbar icon and connect your Stockkarp account (sign in with Discord or Google).
- Play on PS!. Your battles will appear in your game history at stockkarp.com/reviews, and reviews can be requested with one click.
Manual Reviews
- Log in to Stockkarp using your Google or Discord account. Accounts are required to enforce review limits and prevent platform abuse.
- Click "Review a game", and paste your Showdown replay.
Pick which player you are, and paste your team: it’s required for closed team sheet games and recommended for open ones, since replays don’t show spreads - Wait a few minutes for your processed review.
- When it's done, you can flip turn by turn, review recommended moves, and visualize alternative outcomes!
Reliability and Limitations
While the model that powers Stockkarp plays at a high level, the product focus is making sure the user never has to blindly trust the model's decisions. Each grade has some evidence behind it, so you can judge the analysis yourself (and report any issues/model quirks you find).Each turn lets you see:
1. A "grade" for your move. A scale from "Best" to "Blunder" with special labels for gambles, playing to outs, and reads from both sides.
2. An eval-bar with each side's predicted win probability.
3. Your choice, how it played out, and how it would have played out if the opponent acted as the model expects.
4. The model's recommended option, how it would have played out, and how it would have played out if the opponent acted as the model expects.
5. A look-ahead table that displays your top options and how they fare against your opponent's most likely options.
Limitations
Pokemon is a game of hidden information. The model fills in what it can't see as best as possible, but this often means rare sets, unusual spreads, and anti-meta teams can be misjudged. Stockkarp at least requires your team sheet, and performs best in open team sheet formats where your opponent's options are also enumerated. This is part of what motivates the focus on interpretability and evidence.Replays do not include all decision information. For example, you can decide to double-target a slot, but after the mon faints from the first attack, it will look like you split your targeting.
Only VGC Regulation M-C is supported currently. I currently do not plan to support any formats other than Champions VGC, as I'm frankly not knowledgeable enough in other formats to know if this tool would be desired or if the model would be any good.
Accounts are limited to 5 reviews per day right now. Limits may be adjusted in the future, and I'm considering a paid subscription for more usage. Reviews have a non-zero compute cost and I'd like to not burn a ton of money on this.
Ethical Considerations
Competitive Integrity
The most important thing in developing a tool like this is preventing any possibility of cheating. As a result, Stockkarp is limited. Stockkarp can only analyze completed games, hosted on Showdown servers. There is no custom-position analysis or live tooling. Of course, a review tool can help with matchup preparation, but I believe a tool like this doesn't augment the preparation beyond what replay reviewing already does.Training
Training was done with self-play games. The base model was not trained on any human gameplay, and makes no attempt to mimic human strategy. The base model's team preview and opponent prediction heads were fine tuned on a small number of public human replays (~3000) in order to improve the quality of the analysis in human games. No bot games or private replays were used in the fine-tune.Validation
While reviews are built to stand on their own evidence, I still wanted to know if the model’s judgement holds up against human players. In order to validate that the model's analysis is useful, the model was used to pilot a bot on the PS! ladder. It played one game at a time on the Regulation M-C Bo1 ladder, and was supervised. Over ~5 days, it played ~600 games. The bot reached a performance rating of ~1900. The bot won't be run again until the bot needs to be validated again on a new format, at which point Showdown will have hopefully implemented its bot opt-out. The bot's account is already registered on Showdown as a bot.Data Use
Reviewed games and teams will never be used for training. Administrators (just me lol) have to jump through moderate technical hoops to view other users' data, and can not view it in the app without explicit invitation (when sending feedback/reporting an issue, there's a "share with admins" checkbox). For peace of mind, I also do not plan on playing in any premier events in the near future. I care deeply about users' trust in the platform.Human Coaching
One other ethical concern I had was a tool like this potentially threatening coaches who offer services like VOD/replay review. I've consulted a few coaches on the implications of tooling like this being available, and most are in agreement with me that these tools are likely inevitable and not an existential threat. They (and I) believe human coaching will always be a necessary element in VGC, just like how chess coaches still exist in their post-engine landscape. Humans, at least for the foreseeable future, are more attentive, focused, and give much more interpretable and actionable insights. Tools like these supplement coaching, they don't replace it.About me
I've played VGC since 2009, taking a short break from 2016-2020. I had significant success in the Junior and Senior division (Regional Championships + Cuts, Nationals cuts, Worlds qualification), and am a pretty average masters player. Still, my point is I care about the game. I made this tool because it's what I wanted to use, not to show off. I hope it helps people.If at any point I realize that the tool is doing more harm than good, I have no qualms about shutting it down.
Last edited:
