On this, I'm not totally sure why customs got axe'd other then they were confusing/gimmicky/overpowered, which is fair, but i think that's because we took the wrong path with customs.
We put far too much emphasis on matching lore/dex entries in ZA, which I would say is a big mistake. Reading over the X/Y, ORAS and USUM Mega dex entries, hardly any of them even suggest what their abilities are. Some of which could be misleading!
(source (1)):
https://www.deviantart.com/celestcsilvari/journal/All-Mega-Pokemon-Pokedex-Entries-736403897
(yes im citing a fuckin deviantart page, please bear with me. If you want to find specific dex entries go look for them yourself)
I get that this whole metagame is speculative and based mostly off assumptions, but given how the tier has progressed, its fair to say that there is a sembalence of balanced to be maintained.
:sableye-mega: (Magic Bounce) "Ultra Moon: It blocks any and all attacks with its giant-sized gemstone. However, the stone's a heavy burden, and it limits Mega Sableye's movements." This dex entry may suggest Wonder Guard, as absurd as it may sound. Its only mildly in the direction of Magic Bounce, but not close enough to be a for-sure answer.
:mawile-mega: (Huge Power) "Ultra Sun: It has an extremely vicious disposition. It grips prey in its two sets of jaws and tears them apart with raw power." As far as I know, this is the closest a dex entry is somewhat tied to a Mega's ability. Even still, very few would of guessed Huge Power and may have thought of other offensive abilities like Sheer Force.
:banette-mega: (Prankster)
"Ultra Sun: Extraordinary energy amplifies its cursing power to such an extent that it can't help but curse its own Trainer."
"Ultra Moon: Mega Evolution increases its vindictiveness, and the cursing power that was held back by its zipper comes spilling out."
Okay sure, this dex is close to Prankster, but its only approaching the ballpark.
My point is, many dex entries don't reflect a Mega Pokemon's actual ability. Therefore, i believe we should at the very least give customs
one more chance, on the basis that:
1) The effect would be something already found in the series
2) The effect could easily be read in battle.
I also want to emphasize that I'm not trying to suggest that every new Mega, or even every Mega, can be viable in OU. Some Megas will remain UU and RU, and thats okay. But I also think that its unfair to let some megas, like Malamar and Glimmora (arguably) rot for the rest of the tier's existence because we cant give them anything that would balance them, especially considering that we're probably not gonna get ZA UU. To put simply: I think that we had a fairly poor vision for a lot of these Megas due to new toy syndrome and thus we should give customs another shot.
Every ZA Mega doesn't need an absurdly powerful custom, or even an ability with a "new" effect. Most megas could get away with copying another Pokemon's ability, but just rebranded. Examples include:
:scrafty: (mega) Due to Mega Scrafty's x4 weakness to Fairy, it would love Tera Shell. Unfortunately Tera Shell only works on Terapagos because John Gamefreak and Barry Council said so. Therefore, i dont see why we cant just remake Tera Shell, then give it some quirky name like "Shielded Skin" with the activation text "Its skin is reducing the effectiveness of attacks!" or something.
:malamar: (mega) We don't and shouldn't bring back Contrarian. Although i do think Pranskter would actually be pretty great on Mega Malamar since it gets priority screens, Topsy Turvy, Octolock and various other useful tools, I am curious on where Mega Malamar would be if it set Trick Room on switch in.
:falinks: (mega) Falinks is actually quite close to being good. But it needs 2 tools: Something to help it click No Retreat, and something to block Intimidate. An idea was a version of Download that also blocks Intimidate, but instead it grants a defensive boost based on the opponents preferred attacking stat. We also know that Game Freak gave various abilities an Intimidate immunity and/or an alternate Intimidate interaction, such as Scrappy and Inner Focus, so an Intimidate immunity isn't that absurd of a trait to tack on Given the council gave it Dauntless Shield to help it click No Retreat, I don't think this would be too out of place.
These are only examples I gave, because I want to drive home the point that we basically pulled ideas from Reddit and prayed it worked, so when it didnt we went back to what already exists, but now we realise that the existing list of abilities dont actually help many of the existing ZA Megas. So I think we should have some sort of middleground where
Customs are allowed, but only to the extent that we know exists in the franchise already. These ideas pull from what we know is already viable on whoever their ability holders are (Terapagos, Porygon) but shake them up just enough to give the Mega some much needed viability without giving it some wildly unbalanced effect. "Erm but werent most customs already based on existing abilities?" I hear you say and to that I rebut with: Yes. But the abilities we gave were more like Calyrex's As One, an already conceptually broken ability, or did some crazy unorthodox effects, like Brass Bond, and not true "Pokemon-like" abilities.
TLDR: We should consider retrying customs but only basing those customs off existing abilities and adjusted to fit the purpose of the ability holder.