To make this back to Pokemon... I don't think what the series really need if it wanted to be difficult is "harder bosses" or "better AI" necessarly.
What it would need is some sort of constraint: something that doesn't allow you to freely faceplant into a fight to see what they have then come back and stomp it. Without some sort of constraint, the games can't be made hard, just grindy or tedious.
My personal stance is that Pokémon doesn't need to be hard, per se. I don't think it's a problem that the games lack options for really hard difficulties that put veteran players to the test. But I think there still is a problem with difficulty, that they are way too easy, to the point of becoming a real issue.
We have had many discussions of how difficult it would be to make Pokémon
actually hard. You've raised many good points on how most difficulty can be overcome with grinding, and/or be ruined by RNG ("ruined" both ways - either becoming too easy or too hard without the player's decisions having any influence). Within the current constraints of Pokémon, making the game difficult and fair is fairly difficult.
But consider the inverse: How could the games be made
easier? That is not a particularly hard question to answer. Lower the levels of opponents, have bosses use fewer Pokémon, give them fewer coverage moves, don't let them hold items, build their teams to share really obvious weaknesses, give the player's Pokémon more XP, maybe give the player bonuses that circumvent the rules ... hey, wait a second ...
... and that's the point where it dawns on people that most of these difficulty-lowering measures have actually been implemented in many Pokémon games, with no options to play without them. That's what I think is the actual problem. That the difficulty of the game is set very very low, even within the constraints of their mechanics. It appears to be purposely designed not to put up a fight, which makes your victories feel a lot less meaningful.
And that, I think is a distinction we need to make in the "the games are too easy" debate. The games are too easy, but it doesn't mean they should incorporate some sort of "ultra hard mode" - that'd be too far and too pointless. Rather, it means that the games don't give the impression of honest resistance. There isn't really need for an AI that plays like a human would. That would be way overkill and cater to too few. What frustrates the players is rather that the in-game opponents feel too gimped for your victory to matter. The game is pitting you against trainers using too few, too low-level, and too badly-trained Pokémon with deliberately horrible movesets, no Item strategy, no team building, and none of the bonuses you get as a player. They appear to be set up to be automatically waltzed over with no effort. It's like a racing game where you ride a Ferrari and the opponent rides a VW Beetle, or like a boxing match against a five-year-old. You will probably win, but it doesn't feel like an achievement because you were granted a massive advantage.
Granted, this is a sort of "veterans foremost" problem. A casual first-time player wouldn't notice if a boss suddenly brings out a Special moveset on a Physical-oriented Pokémon, or that their sweeper doesn't have coverage moves, or that the Item slots are barely used at all. They would just be happy to win against the scary-looking dude. Figuratively, cruising along in their Ferrari laughing at all the VW Beetles disappearing in their rearview mirror. It takes some familiarity and attention to recognize the difficulty-lowering measures. But once you notice them, they become sort of jarring. Because you realize how much more the games
could have done to give you some resistance, but is instead set up to fall at your slightest touch like bowling pins.
As an example, look to how BDSP was praised for its difficult end-game bosses. Fans looked at the datamined information and rejoiced. Finally a challenge! But how? The battles didn't feature a massively improved AI, or brutally overleveled Pokémon, or restrictions placed on the player. Everything was played exactly by the book, the same rules that had existed for years. The designers just used all the tools available, including a few competently put-together movesets and held items. You could even argue that these bosses aren't that difficult for a veteran player. You could beat them in a few attempts.
But that's not the important part. These trainers feel like they're trying, and that makes them more rewarding to beat. Just contrast XY Lysandre's Pyroar @No item Hyper Voice/Dark Pulse/Fire Blast with BDSP Cynthia's Roserade @ Expert Belt Dazzling Gleam/Shadow Ball/Sludge Bomb/Energy Ball. One is a quite mediocre mid-game 'mon with three moves and no held item, the other is a heavy-hitter with wide coverage and an item to put even more pain into its moves. Overcoming the former is routine, overcoming the latter feels like a true victory. Or look to BW2's Challenge Mode for more of the same. It's not necessarily more difficult, but it
appears to try, and that makes all the difference. It doesn't treat you as a toddler who needs to face a softened challenge, but attempts to face you on even footing. That makes it a lot sweeter to win.
TL;DR - the games don't need their difficulty raised to 11. But they have a very annoying tendency to be stuck on 2, and it would be nice to have the option to go to 5 or 6 or so.
