Simple, ground types. Being forced out by a very common type because you can't hit it is bad. Even if you have weather ball, which is restricted to sun or rain teams, ground types still mean you can't spam your electric type moves, which is the biggest selling point of specs bolt. Even thunderclap, which may look nice, is bad on specs sets since they can easily use a status move, such as boosting moves or status inflicting moves until you either have to switch out or run out of pp, which is extremely unideal. Having to predict the ground type switching in on draco is alright, but other than that its not too good.
Simply put, raging bolt wants to switch up moves and with only one calm mind boost and booster sp.Attack boost, you are already outdamaging specs. Maybe it could work on specific teams, but its simply not too good.