Arikado if you liked Vesperia you'll really enjoy Graces, the banter and fighting system is the most similar (though sadly lacking a sarcastic character), but ya I'd say you can skip the first Xillia.
Also lol, I'm a bit surprised you got stuck anywhere in Abyss. I'll admit that the secret stuff has been pretty hard but I don't recall there being any significantly difficult dungeons thus far.
Also also, I'm now 50 hours into Abyss doing pre-end-game story and item hunting now that I finally found the final Mieu powerups.
Overall, I think there was more traveling back and forth and backtracking in this Tales game than any other I've played before. Thank god they give you a quick jump power later in the game because otherwise this would have been hell.