At the highest viability Rankings, the borders of the two Terms are vague... many People would argue that all S/S+ Mons we have in OU and Ubers right now can be considered "broken" in their own way. In General though, if a mon is considered "viable" instead of "niche" or "unviable", it means that the mon in question has a distinct role, or can even choose between multiple roles depending on the Teams, checks, walls or threatens many prominent Mons, etc.
While in comparison, a "niche" mon, only has one, really small and minor role and use in a Tier, and is an answer to very few or even just one relevant mon.
There's some more Details, but this is the core difference