At this point it might just be easier to kill duplicate-type moves unless one is hidden power? The double stab thing has been a persistent problem with a lot of attempted fixes but if it's still going on, just e.g. removing fire blast from torkoal may be best.
The thing with mons like Gardevoir and Dragonite is that pretty much any combination of four of those moves works fine. If you have a gardevoir generated with psychic, fire punch, and thunderbolt, any one of dbond, hypnosis, memento, reflect, taunt, sub, cm is totally fine. There are a few exceptions, e.g. CM needing special attacks, Protect needing wish/toxic, focus punch (for dragonite) needing sub, etc. If they are still getting bad sets, non-core moves reliant on other moves (focus punch on dnite comes to mind) would be the first to go probably, but I haven't seen bad sets with mons like this in my experience. I'll go generate some random sets and see what I find.
Things I've seen that probably aren't great:
double STAB
heal bell + rest altaria
fake out + bulk up hariyama
subpunch on dragonite can just go i think because it makes sub get everywhere else and i mean dnite can do other things
ninjask that had protect but not bp, that could be a bit awkward to fix
Long post ahead, but I'm passionate about randbats giving intelligent, yet still "random" seeming sets that are closer to Battle Factory than CC, but still not quite there. I hope these thoughts help. I'd be happy to help create this logic, but I'm not a javascript coder and don't have time to learn currently.
I'm not sure Focus Punch really requires sub. There are a number of Pokemon in ADV that sometimes run unprotected Focus Punch in their respective tiers, especially (but not exclusively) on CB sets. Just some examples that come to mind are: Tyranitar, Heracross, Snorlax, Swampert, Medicham, Aggron, Muk, Kangaskhan, Sudowoodo, and Kecleon. A lot of these Pokemon either don't have another fighting option or their other fighting option doesn't hit the target nearly hard enough to be worth it without Focus Punch's power. Brick Break is literally half as strong.
I also don't think Protect BP-less Ninjask is egregious. It still can be useful for toxic stall, to outrun something that has used Dragon Dance or Agility, scout CB pokemon, etc. The only issues here that I'd work on would be double stab and maaaaaybe fake out + bulk up or similar sets on other mons with fake out. But this is a reasonably small set of pokemon in the first place, so I wouldn't prioritize it.
But I would definitely prioritize the double STAB. That Torkoal example with just Flamethrower and Fire Blast should definitely be avoided if possible, but I would not wholly prohibit any situation where 2 attacks of the same type can be given together. True you can just remove the other STAB from the movepool, but I don't think this is a good idea. Some amount of unpredictability about the strength of a STAB attack is a good thing. If the opponent always knows Torkoal has Fire Blast or Flamethrower, they can act accordingly. That's not ideal.
I think it would be best if it allowed a second attack to be given of the same attacking type as a previous attack IF the Pokemon would be receiving its 4th attacking move, so it still has 3 distinct attacking types in that circumstance. There are lots of real examples of Pokemon using this type of movepool on pokemon with 4 attacks, so I don't think it should be a strict enforcement of
never having 2 attacks of the same type.
There also needs to be a few other restrictions around combinations of moves. I'm sure you already have logic for these requirements, but if Sub Punch Dragonite is causing issues, for example, then I think it may be worth changing up how that's coded to integrate it into move selection more generally.
I don't have all the answers on what pairings should be at this time (and I think some of this is coded already?), but I think the general structure of how to work the required pairings in should work with the set of rules for creating possible randbats movesets below. I also think this is pretty generalizable to any generation, with the only major change needed for each generation (aside from the initial movepool options) being which move hard pairings are required, since effects change generation to generation. Some edge cases like Ditto, Unown, and Wobbuffet in all generations or like Magikarp in RBY randbats would also have to be coded in, since they don't really fit the typical mold.
Proposed movepool selection rules:
Move 1:
- If the Pokemon has a STAB attack >= 60 BP (70?) in its movepool, then a STAB attack.
- If the Pokemon does not have a STAB attack >= 60 BP in their movepool, such as how Gengar and Blissey do not, then it still has to be an "attack" >= 60 BP, of some sort
- Attacks lower than 60 BP, are not considered "attacks" for the purposes of moveslot 1 (or any other moveslot). This includes things like Rapid Spin, but also low power priority attacks like Fake Out, Mach Punch, or Shadow Sneak or other attacks that are primarily used for another purpose, such as Clear Smog, Flame Charge, Acid Spray, or Icy Wind.
- Variable power attacks like Low Kick, Foul Play, or Flail are counted as "attacks" and can be called in moveslot 1.
- Fixed Power attacks like Seismic Toss or Super Fang are counted as "attacks" and can be called in moveslot 1, however they are considered "typeless" attacks for later checks (i.e. theoretically Dusknoir could get both Night Shade in moveslot 1 and Shadow Sneak later on without violating any same-type attack rules. Night Shade is not considered ghost-type, and Shadow Sneak isn't even considered an attack!).
- Retaliatory moves like Counter are not "attacks" and cannot be called in moveslot 1.
- Multi-hit attacks are counted as "attacks" because I'm pretty sure they'd only even be in a movepool if they were strong enough. i.e. Bullet Seed is not in ADV movepools, but it is in BW movepools.
- Some esoteric moves:
- Pursuit is counted as 40 BP (could be persuaded to say 80, but 40 will decrease its prevalence slightly which is probably preferred for matchup issues)
- Payback is counted as 100 BP, since it's most often used on slow Pokemon
- Weather Ball counts as 100 BP (yay Castform)
- Acrobatics counts as 110 BP
- Draining Kiss counts as an "attack" even though it's low because of the combined effects of insane 75% healing and being Comfey's only good STAB, which also has priority. All Comfey should get Kiss.
Move 2:
- If move 1 was a move that requires a hard pair (and a possible paired move is also in the movepool, which it should be), then a hard paired move
- If move 1 was not a move that requires a hard pair or the paired move is not also in the movepool, then any other move from the movepool, except for another attack of the same type as move 1
Move 3:
- If move 2 was a move that requires a hard pair (and a possible paired move is also in the movepool, which it should be), then a hard paired move
- If move 2 was not a move that requires a hard pair or the paired move is not also in the movepool, then any other move from the movepool, except for another attack of the same type as move 1 or move 2
Move 4:
- If move 3 was a move that requires a hard pair (and a possible paired move is also in the movepool, which it should be), then a hard paired move
- If move 3 was not a move that requires a hard pair or the paired move is not also in the movepool, then:
- if moves 2 and 3 are both attacks, then any other move from the movepool that does not require a hard pair, except for another attack of the same type as moves 2 or 3
- if moves 2 and 3 are not both attacks, then any other move from the movepool that does not require a hard pair, except for another attack of the same type as moves 1, 2, or 3
Sets these rules would allow or not allow for some example Pokemon. The most complicated by far is the last one, since it has a boosting moves, a possible one-way required hard pairing, and multiple attacks of the same STAB
Torkoal:
Flamethrower, HP Grass, Explosion, Fire Blast - allowed
Flamethrower, HP Grass, Yawn, Fire Blast - not allowed
Flamethrower, HP Grass, Explosion, Yawn - allowed
Flamethrower, Rest, Explosion, HP Grass - not allowed
Flamethrower, Rest, Sleep Talk, HP Grass - allowed
Flamethrower, Rest, Sleep Talk, Fire Blast - not allowed
Gardevoir:
Psychic, almost any other 3 moves
Almost any move set from its movepool is currently legal by my proposed rules, so long as it has Psychic, since its movepool does not contain any examples of duplicated attacking types. The only requirement is if it pulls Wish or Protect it must get the other one.
Dragonite:
HP Flying, almost any other 3 moves
HP Flying is required, since it's the only STAB. There's two other restrictions as well. The only no-go is both Fire Blast and Flamethrower, since Dragonite is not a Fire type and this isn't specifically duplication of a STAB. In addition, if Dragon Dance is picked, it will require a second physical attack to be selected, since it's agnostic to the fact that HP Flying was selected in moveslot 1. So Dragon Dance has to followed by a physical attack like Earthquake, for example
Houndoom:
Crunch, Fire Blast, HP Grass, Pursuit - allowed
Crunch, Fire Blast, HP Grass, Flamethrower - not allowed... however...
Fire Blast, Crunch, HP Grass, Flamethrower - allowed so long as they get selected in this order. Since Crunch vs Fire Blast getting picked first is an even chance, this shouldn't bias anything for dual typed pokemon that have 2+ options for both STABs in their movepool.
Crunch, Fire Blast, Will-O-Wisp, Pursuit - allowed. Even though the first 3 moves weren't all attacks, Houndoom can still get Pursuit as the 4th move because Pursuit is not an "attack" since it's considered to be only 40 BP.
Crunch, Fire Blast, Will-O-Wisp, HP Grass - allowed.
Kyogre:
Surf, Ice Beam, Thunder, Water Spout - allowed
Surf, Ice Beam, Calm Mind, Water Spout - not allowed
Surf, Ice Beam, Calm Mind, Thunder - allowed
Surf, Ice Beam, Water Spout, Hydro Pump - not allowed. Triple STAB will never be possible with these rules.
Salamence:
HP Flying, Dragon Dance, Earthquake, Fire Blast - allowed
HP Flying, Dragon Dance, HP Flying, Earthquake - "allowed" but obviously not OK to duplicate HP Flying. So there has to be a check as well that HP Flying isn't "allowed" as a move even though it's in the legal Dragon Dance -> hard pair list.
Dragon Claw, Dragon Dance, HP Flying, Earthquake - allowed
Dragon Claw, Fire Blast, Hydro Pump, Dragon Dance - not allowed
Victreebel (assuming Solarbeam requires Sunny Day, but Sunny Day does not require Solarbeam):
Solarbeam, Sunny Day, HP Fire, Sludge Bomb - allowed
Solarbeam, Sunny Day, Sleep Powder, HP Fire - allowed
Solarbeam, Sunny Day, HP Fire, Giga Drain - not allowed. Giga Drain conflicts with the earlier selection of Solarbeam. As a result, no set will ever have both Giga Drain and Solar Beam, since Solarbeam requires a non-attacking move in Sunny Day and Giga Drain can only get pulled in as a second stab if all 4 moves are attacks. The order does not matter... see another possible ordered attempt at the same set:
Giga Drain, Sunny Day, Solarbeam, HP Fire - not allowed. Sunny Day does not require Solarbeam and Solarbeam conflicts with the earlier selection of Giga Drain. For Solarbeam to get pulled after Giga Drain has been pulled, it would have to get pulled 4th, but it can't get pulled 4th, since moves that require a hard pair like Solarbeam requiring Sunny Day can't get pulled 4th.
Giga Drain, HP Fire, Sludge Bomb, Sunny Day - allowed (Sunny Day does not require Solarbeam, so it's perfectly fine to get selected 2-4 without Solarbeam following)
Giga Drain, Sleep Powder, Sludge Bomb, Swords Dance - not allowed. Swords Dance can't get pulled 4th, since it requires a hard paired physical attack to get pulled after it. However...
Giga Drain, Synthesis, Swords Dance, Sludge Bomb - allowed
Giga Drain, HP Fire, Swords Dance, Sludge Bomb - allowed
Giga Drain, HP Fire, Swords Dance, HP Ground - "allowed" except that HP Fire and HP Ground together is impossible. Need to consider all HP the same move for movepool selection, which I'm sure randbats already does!
Victreebel has only 5 attacks, 2 of which are Hidden Power, meaning it really only has 4 distinct attacks. And in this scenario, Victreebel can't ever have both Giga Drain and Solarbeam. That means you know that Victreebel will always have at least one supportive move, which could be Sleep Powder, Sunny Day, Swords Dance, or Synthesis. I don't think this is a bad thing, though because a Pokemon with as limited attacking options and good supporting options as Vic would likely never run 4 attacks in real matches anyway.