ORAS are definitely better than RS, but what does it for me with Emerald is the way that the Aqua and Magma stories are tweaked so that you fight against both. Part of it may be that I played Emerald before any of the other Hoenn games, but trying to go back and play the others just feels weird when the secondary team (Magma in Sapphire, Aqua in Ruby) is just kinda hangin' out.I have played through both the Pyramid and the Palace in Emerald, and I disagree with you.
Not sure what you find difficult about the Pyramid. From what I remember, I found it fairly easy for the most part. You just had to adapt your team depending on the theme for the wild Pokémon in every round, while also having a well-built team that could handle trainers. I also found Blissey incredibly useful in the Pyramid since it can heal your team outside of battle and cure status during battle. I did find Gold Brandon to be really challenging though, I remember that I had serious problems in my battle against him. Probably the hardest Gold Battle against a Frontier Brain out of those that I have won. But the rest of the Pyramid wasn't super hard from what I can remember.
As for the Palace, it has a huge luck factor for sure, but it isn't only about luck. There are ways to raise your chance of winning. Since Natures are determining what attacks your Pokémon will use, picking a good Nature is one of the keys to victory. One strategy I have seen is using a team of Pokémon with Hasty Natures and 3 offensive attacks + Protect. Another strategy which have I thought about but never actually tried in practice was to make sure every Pokémon has at least one move from every category, making sure no turns are "wasted". And some Natures that are usually useful can still be good here, one example being Adamant which gets a huge chance of attacking when the Pokémon is below 50% HP. That said, I might just have been really lucky while battling at the Palace, but I recall that my team of Salamence/Starmie/Regice worked very well there.
If anything, my experiences with the Emerald Frontier are that the Pyramid and Palace are two of the easier facilities since I only lost once in each of them on my way to the Gold Symbol battle. The only one I had an easier time at was the Pike where I made it to Gold Lucy on my first try. I found the Arena, Dome and Tower considerably harder as I had to make a lot more tries at each of them. I haven't gotten the Gold Symbol from the Factory (yet), but I'd still say it is the hardest facility.
I do also not really agree that the Emerald Frontier is the hardest in-game challenge. As far as Battle Facilites go, I have always found the Subway in Gen 5 and the Tree in Gen 7 to be very tough and hard to beat. The Battle Institute wasn't all easy either but I think it was easier than the Factory since I managed to beat it in US/UM, but I have yet to beat the Factory in either Emerald or Platinum. I also found the Battle Tower in Crystal extremely hard and unforgiving, but that might be because I was very young and had less experience when I battled at it. Regarding the Gen 4 Frontier, I found it easier than the Emerald Frontier on the whole. But I think this might have been because I used better Pokémon/teams and had more experience when I played the Platinum Frontier compared to the Emerald Frontier, which was basically my starting point for serious semi-competitive battling. Platinum only had 5 facilities and only required 49 wins for each facility (170 wins for the Hall), while Emerald required more variying amounts. Some were lower, others were higher. The most notable is that the Tower required 70 wins in Emerald but only 49 in Platinum. The Factory required less wins in Emerald though. And winning 100 battles in a row at the Platinum Tower to get the TC upgrade was another thing I found rather difficult and challenging from the Frontier there. On the whole, I'm not sure what I'd say the hardest in-game challenge in Pokémon is. The Emerald Frontier is a candidate for sure, but it faces competition from other facilities such as the Tree, Subway, Institute, Crystal Tower and Gen 4 Frontier (especially the Factory).
I also disagree about Emerald being the best game. While it was definitely the best for its time, it is no longer the best if you ask me. I find it to be mostly obsolete nowadays because of OR/AS. One possibly unpopular opinion of mine is that I think both Emerald and its Battle Frontier are overrated, which feels a bit sad to say since I really like them, but that's how it is. People often prefer Emerald over OR/AS and complain about OR/AS not having the Battle Frontier, but I do personally not care. I find OR/AS to be superior on the whole, even without the Frontier or anything else from Emerald. Playing through the Maison twice more after X/Y was fun enough for me, I liked trying new teams and strategies there. I also get the impression that many of those who complain aren't into Battle Facilities overall, they have never tried the Emerald Frontier and wouldn't have tried the Frontier if it had returned in OR/AS. I could be wrong, but this is the impression I get from many of those who complain.
I agree with you when you said Gen 4 Frontier is easier, mainly just because there's less hax that get pulled (in my experience). The guy who has the longest streak in the Gen 3 Tower lost to literally the perfect combo of an opposing team and unluckiness