I'm in school so my Pokemon thoughts are a bit muddled, but physical Contrary users are horrible. None of them (except... Conkeldurr? Terrakion?) get STAB on their Attack boosting move, while there are a multitude of Special Attack lowering moves (Draco Meteor, Leaf Storm, Overheat, Psycho Boost, might be missing one) and your Contrary user is incredibly likely to get STAB on one of them. Physical Contrary is too slow and doesn't have enough power.