I don't think that the move Swagger or any move should be banned, because that will facilitate more bans on other moves, like Stealth Rock. I think the best way to go with this is to ban Swagger and Foul Play in conjunction with the ability Prankster. Swagger and Foul Play by itself is easy to deal with, but when you throw in Prankster, it becomes pretty annoying. Give it a Substitute to sit behind and paralysis, and it becomes extremely frustrating to deal with. I understand that running multi-hit moves will break through this, but how many multi-hit moves can really damage something like Klefki or Thundurus? Plus, all the multi-hit moves I can think of either have a small pool of Pokemon that can learn them or aren't actually that effective in terms of type advantage against the two most common SwagPlay Pokemon. Priority moves won't even do much to them, especially if they're hiding behind a Substitute, or worse, you hit yourself or paralyze.
Also, when talking about Infiltrator or sound moves and Magic Bounce, I don't want to run an Infiltrator, sound move, or Magic Bounce on every single team. There are better abilities that might fit my team better. Plus, the sound moves or Infiltrators that penetrate Substitutes are still affected by paralysis and Swagger, still leaving them a low chance of attacking on that turn. I'd rather not leave everything up to chance when I want to attack, and I think a what I think is 37.5% chance of attacking is a bit low for me to control. With Magic Bounce users, the only three Pokemon that fall under that category are Xatu, Espeon, and Mega Absol. Barring Mega Absol, the other two have a weakness to Foul Play, which will do tons of damage to both of them even without any attack boosts at all. With Mega Absol, it already has abysmal defenses and no recovery moves, leaving it to be a bad candidate to counter SwagPlay.
Last, I think SwagPlay almost defines an uncompetitive scene for Pokemon. Granted, Double Team and Minimize were much more infuriating than SwagPlay, but I still think SwagPlay isn't competitive. I see many posters discussing about how weaker players can beat more skilled players through this, and while this is true to an extent, I don't think that's the main reason it's not competitive. I think the main reason it's uncompetitive is because it makes you rely on luck instead of skill to make a move.
Overall, SwagPlay is far from a win-all and a strategy without counters (Numel anyone?), but I certainly think it's a brainless, uncompetitive strategy that I believe isn't healthy for the metagame.