Well actually you can still come in on most physical threats you're supposed to continuously regardless of the investment, the issue w/ investing everything in physical defense is if any of those threats you're supposed to stop cold carry a special attack(i.e Ttar, Garchomp), you can no longer switch into them, as now you're crippled or dead, which kinda means you can't do your job very well at all.
And actually there are special attackers that have difficulty or very well even can't take you if you're investing in special defense. Any boosting special variant of MLuc not carrying flash cannon can't actually kill you at +2, and those that do get a 12.6% chance to OHKO while you threaten to OHKO w/hammer arm at a 62% chance, guaranteed OHKO after a layer of spikes, so he can serve as an emergency check at the very least.
Aegislash variants carrying lefties/spooky plate can't beat you at all as HP Ice/Flash cannon are guaranteed 3HKO's, the Autotomize WP set can only 2hko you at best with both HP Ice and Flash cannon, and that's after the +2, w/Leech seed+leftovers recovery+spiky shield you might be able to turn that into a 3HKO. Humorously enough, Autotomize WP sets can't hope to kill you w/o the WP boost, due to synthesis/leech seed/spiky shield(not all three of course), so you can sit in its face and set-up spikes(if you have them) all day, then proceed to phase, or just stall w/leech seed+spiky shield. Aegislash variants carrying LO have an easier time getting through, but you can pull off a win vs them too.
Thunderus-I and Thunderus-T running Timid LO(non boosting sets) can only get 3HKO's at best w/Hp Ice. As I showed above non LO variants of Gengar have severe difficultly doing anything to you w/o sludge wave, LO sludge bomb variants that don't opt for dazzling gleam and instead go for something like HP fire/ice only get 3HKO's at best. Alakazam dominates you, but if you absolutely need to you can survive any one hit from above ~85% health and break it's sash.
Granted, of course there are better options for special defensive walls, that wasn't my point. My point was that as it stands you get your ass kicked by anybody with a special move you're not immune to, even things like specs Keldeo(granted it's keldeo, but you resist half it's moveset, and you still can't stay in when physically invested) and some things you're supposed to come in on can carry special moves, which kind of hinders your walling potential as a whole. Sure you don't have to stay in on those special threats, but this also means that certain threats that would usually force you out on the switch can't do so as safely anymore(given you have the right coverage move) as they risk dying before you do.
