My issue with infernape is that it's not always powerful enough to sufficiently deal with certain things. True you can solve that with Swords Dance/Nasty Plot, but you can also use things like Terrakion who have way higher offensive stats off the bat that also have access to Swords Dance. On the Special side you have lucario who, while it might not have the speed (or great offensive typing) that infernape has, it has a (slightly) higher special attack stat, as well as a much better STAB special fighting move. It also has better typing defensively, although it still takes resisted hits like tissue paper. Finally, it has a much better Swords Dance set due to access to Extremespeed. Finally, as a Fighting type SD sweeper, Breloom does a much better job than it due to its much higher-powered STAB moves and higher attack, not to mention access to Spore. Sure, Infernape has some incredibly powerful attacks in the sun, but it's pretty difficult to keep Sun active in the current metagame, and unlike other SD attackers, one of its STAB moves is rendered completely useless when Rain is up.
tl;dr, I think that as a whole, Infernape is pretty outclassed unless you're running it on a sun team, and even then it's still outclassed in other scenarios.
EDIT: @Arctic
Sure, Infernape resists Bullet Punch/Ice Shard, but it's MUCH frailer, and it takes neutral damage from SR, which can add up, especially since ScarfApe commonly carries U-Turn. Terrakion on the otherhand is a complete powerhouse from the bat, and it resists SR. As a plus it's also immune to sand, which is another thing that puts a timer on Infernape's already short life. ScarfApe typically carries Flare Blitz as one of its STAB moves, which combined with passive damage can quickly cause it to fall.