I partly agree. When I play Pokémon, I mainly want some fun and entertainment, not a super hard challenge. But I don’t fully agree that the earlier generations were more challenging. My experience is that as long as you know the basic gameplay mechanics, all Pokémon games are generally easy. If we take Gen 1 as an example, I had some serious issues the first time I played Blue as a kid, but if I were to go back today, I would have a much easier time since I now know how to play it properly. I think the earlier generations just required more grinding and had worse options compared to the newer generations, and from personal experience, it can be hard to go back to them when you are used the newer generations. Thus, they can feel “more difficult”, but they aren’t really if you take a closer look.
Regarding tough challenges in Pokémon, that’s not something I look for – at least not in the main game (but I am a big fan of post-game Battle Facilities). Difficulty in Pokémon games is not important to me, and to the extent that it is, I prefer having a fair and balanced difficulty over something that is hard in an unfair way. For instance, I am a big fan of the Indigo Disk, it was difficult but in a balanced way since the opponents don’t cheat, they just use good strategies against you. In comparison, I’m not quite as fond of the Totem battles or Ultra Necrozma in Gen 7. Because in those battles, the opponents have an unfair advantage over you. These types of challenges are still okay, but they are not the kind of difficulty I’m looking for in Pokémon.
I don’t mind a bit of a challenge in video games in general, but at the same time, I don’t like when games are 
too difficult. If I struggle to beat a hard challenge/boss/dungeon/whatever in a game, then I tend to lose motivation and quit playing. Or maybe lower the difficulty level, or use a guide for help, depending on what the options are. I have rarely experienced this in Pokémon during the main story, but it can happen in the post-game. For instance, I have never beaten the Battle Factory in Platinum despite using guides and trying many times, it is just too hard for me (though I might go back and make more tries in the future).
I agree with you that the Pokémon games have been getting better with most things from Gen 5 and on. I can still go back to the older games, but I generally prefer the newer generations. If we look at Gen 4, I thought it felt modern for its time, but it has certainly aged since then. I can still go back and play Platinum without major issues (which I did last year), but going back to D/P or HG/SS is a lot harder.
I completely agree with you regarding nostalgia. When it comes to Pokémon, my enjoyment of the games is partly based on how much fun I had with them when they were new, but also on how much fun I (think I would) have with them today. Regarding Crystal, I replayed it last year and I found that it was mostly carried by nostalgia. The game has a lot of old and outdated mechanics, and if that wasn’t enough, it also features all of the infamous Johto gameplay issues. For more in-depth thoughts, see my review 
here.
I won’t deny that I had tons of fun with Gen 1-2 as a kid, they and Gen 3 were a big part of my childhood and early teenage years. But nowadays, they don’t hold up all that well. I prefer the newer generations, and I honestly feel that I have had more fun with Pokémon as an adult than as a child. I consider Gen 5 to be my top favorite and that’s not because of nostalgia… or at least that’s what I want to think, but I’m not sure anymore. It has been 14 years since I first played B/W, and 13 years since B2/W2. If I were to replay them today, would they still feel as great as they did back when they were new? I’m not sure, and I’m a little afraid to find out. I replayed the main game of both Black and Black 2 a few years ago, and I still enjoyed them, but I found myself missing many game mechanics and QOL updates that were introduced in subsequent generations.
But even if Gen 5 is my current favorite (and have been for 14 years... man, time really flies), I hope there will be a future Pokémon game that becomes my new favorite. As much as I love Gen 5, I don’t want it to stay my favorite forever. I want there to be a new generation which takes its place, just like Gen 5 dethroned Gen 4 for me back in 2011.
Yeah, after having been a part of the online Pokémon fandom for 20 years, I have seen this as well. Gen 3 was hated when it was new, but several years later, people wanted remakes (and eventually got them). The same thing happened with Gen 4 and 5, though the latter has yet to get any remakes (and I hope it never happens). I think which generation(s) people prefer is often related to which ones they grew up with, but not always. For instance, I grew up with Gen 1 and 2, but they are my least favorites, and I would prefer to never revisit them again. Meanwhile, Gen 5 is my favourite, which was released when I was in my early 20s.
Regarding Gen 6 and 7, my opinion is the opposite of yours. I loved Gen 6 and X/Y, while Gen 7 and S/M weren’t nearly as enjoyable for me. I agree that Gen 7 had a better story, but story isn’t that important to me in Pokémon. S/M had some pretty big issues with the gameplay, which made them less enjoyable to play. US/UM fixed a lot of the issues with S/M, which came at the cost of making the story worse. But since I don’t value story that much, I consider US/UM to be the superior Alola games. X/Y on the other hand had a very average story, but the gameplay was just plain fantastic all around, so I find them superior to all games from Gen 7. Regarding Gen 8, I would rank it below both Gen 6 and 7, but I still enjoyed it a lot. I also liked the story in S/S.
Overall, I rank the generations like this: 5 > 6 > 9 > 7 > 8 > 4 > 3 > 1 > 2
As I said in my last post, Gen 5 is on the top, followed by Gen 6-9, then Gen 1-4. My ranking is purely subjective, based on nothing but my own personal experiences with the games. I don’t care what the fandom or “people in general” think, my own opinion is the only thing that matters.