On simulators, it simply fails. If you Spored the enemy earlier and it's still asleep, Relic Song won't put your opponent to sleep, Effect Spore won't, and PShift won't work. In WiFi battles, I assume the occurence is an automatic loss.
I thought that might be the case. I thought Psycho-Shifting sleep sounded fun, but difficult and inefficient to do, and that led me to the chance of sleep moves. Thanks!