Chansey's physical walling ability is the scary part--not its special walling ability. To ban Chansey because of its ability to wall special attackers would be silly. The metagame already contains threats such as Snorlax and Porygon2 that, while not as specially bulky as Chansey, are still enormously bulky on the special side and have actual offense. My first thought when Porygon2 comes in on a special attacker that it can wall is usually, "OK, gotta not take too much from Tri Attack/Tbolt/Ice Beam," and my second thought is usually, "uh oh, what if Tri Attack haxes me? Gotta prepare for that" before I make my move/switch something in. I rarely find myself just trying to muscle through it, so in effect it acts in the same manner as Chansey. In this way the metagame is already pretty heavily centralized around physical attackers.
Dedicated special walls are not only much easier to have because special attacks tend to have lower base power, but there exist more Pokemon suitable for the task. Umbreon is similar to P2, and guys like Hitmontop have a bunch of SDefense, whereas all the guys with high Def like Rhyperior and Steels bar Bronzong (who has equally good SDef anyway) are weak to common physical attacking types like Fighting/Ground/Fire.
So yeah, if you were to ban Chansey just because it stops special attackers, you'd have to ban Togekiss because it stops defensive Pokemon. They both have very general things that they're very good at and impact battles and team-building considerably for certain types of teams.
The problem is that when something is that good at its primary role, it shouldn't have a secondary role (of soft-walling physical attackers that don't have STAB SE) that's equally scary. That's just too much. So now you not only have to switch in a physical attacker to kill Chansey--you also have to make sure it has a Choice Band, or can boost, or has oodles of base Atk. Too much to expect. Then you add in Chansey's supporting movepool--while you're busy switching in that physical attacker, Chansey can use Aromatherapy, Wish to pass infinite bulk to something actually scary, or use Toxic/TWave on you and cripple whatever you switched in.
So Chansey's problem definitely isn't that it walls special attackers. Other things already do that. Chansey's problem is that it has Umbreon's support movepool with better-than-Snorlax special defense and beyond-frailty levels of physical defense.