From what I've observed, even with power and followers, or the illusion of such, more people get pissed off at you for doing that than the person you're trying to get rid of.
Of course, if he has no followers, he has no way of swaying anyone's opinion. I wouldn't consider him relevant.