Perhaps, when attacking the nest, the leader of the nest (with better stats than a mon normally would and an actual item with a pretty good AI) and a minimal of 2 other weaker mons attack.
When befriending a nest, a small battle occurs and you have to use "special actions" that take up turns in battle and leave you vunarable to attacks. If you succed with these actions, the leader begins to like you and, if you win, the leader may invite you to the nest.
If the above occurs, you cannot have the leader join you. Just weak members of the nest.
Random encounters should be strong mons with powerful items and bloated stats. This makes the player act carefully in order to prevent himself from being wiped out.
Each nest should have a trust gauge where, if a mon from that nest dies, it goes down. There is no way to increase the gauge, so if it gets Down to about 30%, you are barred from the nest for the rest of the RP and going near the nest will evoke attacks.
I have more ideas I can send as well
Notice: will be inactive for a day or two. Shouldn't be more than 2 days.. Sorry, please forgive me.