Well, let's look at the components of Baton Pass in order of importance:
1a & 1b. Scolipede and Espeon: Baton Pass chains absolutely need these two to be even remotely viable. However, both lack bulk, so most powerful attackers can get by just these two with brute force. A limit of 2 Baton Passers would make Baton Pass chains unviable and is overkill.
3. Vaporeon: Vaporeon provides the physical bulk that Espeon and Scolipede lack. Vaporeon can be annoying because of its ability to phaze and boost its defenses. Just these three can be hard to deal with if you do not have proper counters. However, strong special attackers like Aegislash can still muscle through a core of only Scolipede, Espeon, and Vaporeon. Dark types also don't fear this core very much. Talonflame, Mega Pinsir, Manaphy, Mega Heracross, Mega Manectric, Thundurus-I, Azumarill and Landorus-I are all examples which do not have much trouble with this core. A team with 3 Baton Passers is borderline viable because of all of these common threats which can easily destroy them.
4. Sylveon: Sylveon provides the special bulk and Dark type counter which has made Baton Pass teams prominent. Still, even with Sylveon, Baton Pass still has trouble with Crumbler Aegislash, Talonflame, Mega Pinsir, Manaphy, Thundurus-I, Azumarill, and Landorus-I with Sludge Wave. It is more manageable than 5 or more Baton Passers because they do not have Zapdos to handle things like Mega Pinsir and Talonflame, and they do not have Smeargle to shut down a threat with Spore. Because they do not have Smeargle for Ingrain, they have to rely on prediction to avoid phazing. A team with 4 Baton Passers can be very good, but manageable for other players who are prepared.
Thus, I think that 4 is the magic number for those who want to nerf Baton Pass without making it unviable. I think it is appropriate for players to meet any bans halfway in the middle. What I mean is that we can help people by nerfing Baton Pass, but the playerbase also bears some responsibility in better preparing themselves for Baton Pass teams. We should not cater to players who do not like Baton Pass by making it completely unviable. I think we should be conservative with any ban, and set the limit to 4 Baton Passers.