It's not really pejorative. It's about power balance. That balance is almost always tipped in favour of the older person, partially because of their legal status, but also because of how we're basically expected to look up to our elders. That sort of thing is especially noticeable with children/teenagers because, well, most of them wish they had the sort of "freedom" that adults have. It's the same reason why the age of consent tends to rise for people in positions of authority over a minor (i.e. a teacher can't have sex with their student unless said student is over 18, even in countries where the AoC is 16), it's why any teacher having sex with a student (regardless of age) isn't considered kosher, and it's why there tend to be exceptions for younger people dating/fucking under the age of consent, but never for people much older.
Maturity comes into it, of course... but it isn't the sole reason for this shit being considered nasty. Of course, every teenager believes they're mature for their age, despite them always looking back years later and realising that they were immature idiots, but shit like the age of consent exists not just because of children being too immature to really understand the implications of sex, but also because of how easy it is for adults to pressure them into it (whether the adults realise they're doing it or not).
Basically what Jorgen said. On a related note, this sort of shit really belongs in the sex and relationships thread, heh.