That, my friend, is where you are wrong. Maybe no-one really prepares for going against Infernape. And maybe they don't expect anything but U-turn from it. But I know from experience that while a good portion of the meta can easily take him, (or her I guess) Ape can destroy so many other parts of the meta with his versatility and mixed offensive capabilities, that as long as you take out his threats, he can wipe the floor with them, or manually switch if they are still there. If you use U-Turn to switch, however, you risk Talonflame hitting you with priority BB, Aqua Jet from Azu, or really any 2x priority move. Or just being flat out outsped. U-Turn isn't quite that great, and neither is it "one of the the only reasons Infernape has even a slight amount of relevance in OU" For example, I don't recall even seeing anyone mention U-Turn on this thread until you did, and this is in fact the Infernape Gen VI OU thread.
Well, if you're letting a Talonflame or Azumarrill switch in on Ape, you're doing something wrong. U-Turn should be used to gain offensive momentum. For instance, let's say you're up against a team with a Hippowdon that you can't really touch with any of Infernape's moves. Rather than close combatting or Flare Blitzing it as it comes in and then U-Turning out after, the correct play would be to U-Turn out as it switches in, and then proceed to switch to something that *can* threaten it, to prevent it from getting stealth rock set up or Slacking off on the switch. The important part of U-Turn here, though, is that if you just switched, you wouldn't be able to see whether or not they actually brought in Hippowdon after all; U-Turn's relevance is in the "slow" switch it provides- being able to switch yourself *after* you see your opponent's switch.
In fact, by maintaining momentum like this, you're wearing down Ape's counters by not allowing them to recover. Once you've worn them down enough through smart play, Ape's capabilities as a wall-breaker can then be utilized. However, before this point, as you put it, "a good portion of the meta can easily take him." The point of U-Turn is that it can allow Ape and his/her teammates to break down the parts of this Meta that do wall Ape.
Also, why the hell would you U-Turn against Talonflame? You can just OHKO it on the switch with Thunderpunch. If it comes in to revenge ape, then you obviously switch; if you're running Ape without anything that can safely switch into Talonflame, then you definitely need to revise your teambuilding strategy.
Azu is a little more tricky; I'd probably opt to U-Turn out on the predicted and into something that resists Aqua Jet and can threaten Azu back; however, once it's in KO range, a Thunderpunch on the switch can finish Azu off easily. Plus, if it doesn't quite manage to, Ape can outspeed the aqua jet with a Mach Punch to nab the KO. Remember that Azu doesn't have reliable recovery, so it's really not that hard to wear it down to the point where it can't safely switch in on Ape- if that's their only Ape counter, then wearing them down to this point effectively allows Ape to nab a KO every time it enters the battle.
If Ape can sweep their team straight up without switching out once, that's great. Good job. They're a horrible teambuilder. The point is, this won't happen higher up on the ladder.