When you select a format, then choose a team, and click on
Battle! Find a random opponent
, if the team is invalid, you'll receive a popup, but the selected format will remain in the dropdown. However, when you challenge someone directly, the format resets if the team is invalid.