If you read Takeshi Shudo's blogs, he said he was deciding to choose between Misty or Team Rocket to drop after Johto, and he chose Misty because he favored TR but also because he felt Misty was the least justified character on the show at the time. Also as said they had just brought Brock back when Johto started, so him leaving again so soon would have felt odd. Misty just filled the token female role, which as seen now with 8+ female leads later, they could replace Misty with any other kind of female and it would make no difference.Brock definitely stayed on far too long but I think a lot of that was just because of the sheer backlash from Orange Islands. It probably took a LOT of back & forth in the planning room on coming to the decision to drop Misty and I bet part of that was "okay but we have to at least keep brock".
Overall when you watch through Johto especially now it's fairly obvious the writers seemed really bored with Misty's character, and probably would have replaced her after Orange Islands if there was some female to use when Gold/Silver came out. For the whole first two seasons of Johto she only has like 4 episodes that she stars in (the crybaby Marill ep, the battle over Totodile, the Seaking catching Contest, and one more I think), and that was over 100+ episodes. Then in late Johto they tried to do a bit more with her with the Whirl Cup and capturing Corsola...but by then it just felt too late.
Also keep in mind the writers obviously knew at least a year ahead of time R/S was having a new playable female lead from the start and a new sidequest with Contests, so the writers probably decided somewhere around late 2001 or early 2002 that the would be dropping her. Episodes are scripted/planned out months before they air on TV, so obviously Misty leaving wasn't a last minute decision, it had to have been decided almost a year before Johto even ended.
Then just the fact that they started promoting the hell out of every female lead who came after her, beginning with May and then continuing with Dawn, etc afterward, really just makes me think they just have more fun writing all the other female leads. They give them more storylines, battles, pokemon, rivals, put them in different outfits, etc. Even the SM girls which had to share limited screentime with each other still got treated similarly with their own subplots and roles.
I suppose the problem was just poor planning in the OS especially the Johto portion. The writers had no idea how long the show would go on back then, and Misty/Brock were generally never given their own storylines/subplots separate from Ash back then, and only really got focus in the odd filler episode here or there. And then whatever they learned was forgotten about as soon as the next episode started. And now here we are nearly 18 years after the OS ended, and it holds true to this day.