When Wrap is legal, Vic's main RBY set is Wrap/Sleep Powder/Razor Leaf/Stun Spore. You probably don't want to replace Stun with Hyper Beam/Swords Dance, especially as Stun Spore lets you A) Wrap things you otherwise wouldn't be able to, and B) punish Gengar even after you use Sleep. Victreebel is much scarier than Exeggutor because of its Wrapping ability, but it's also a lot squishier. Victreebel and Egg thus play very differently, but will be compared because of their similar roles as Grass-types and Sleep inducers. Egg is just way more reliable, hence why Victreebel isn't used that often.
When Wrap isn't legal, Victreebel goes from good gimmick to just gimmick. It takes on the role of Swords Dance (SD) sweeper, with a set like Hyper Beam/Swords Dance/Razor Leaf/Sleep Powder. The idea is to bring it in late-game, use that late Sleep Powder if they sac'd their original sleeping mon to get a turn of free setup, then Hyper Beam everything except the Rocks, which get Razor Leaf'd. Even if everything doesn't go according to plan, you're likely to severely cripple their Tauros with Razor Leaf. The problem is that as far as late-game mons are concerned, you already have and want to be using Tauros, and probably don't want another mon like it to make the middle game effectively 4-on-5. This is why the Wrap set is preferred when it's legal: it lets Vic do something in the mid-game.
Victreebel is often compared to Venusaur in the SD role. It's popular to consider Victreebel the strictly better Grass SD user, given its higher Attack stat, but Venusaur might actually be useful in the mid-game with its superior Speed (useful for guaranteeing outspeeding Cloysters) and physical bulk (makes fighting Snorlax much easier). For this reason, I'd suggest Hyper Beam for Victreebel, but Body Slam for Venusaur (or perhaps Double-Edge for the +2 2HKO on Chansey, although the recoil and lack of paralysis chance probably aren't worth it), as the former is a cleaner, whereas the latter is better for midgame pressure.
In GSC, it's basically garbage in OU. It's kinda like a slow, super-frail Tentacruel that can can only hit Skarmory on the switch and is weak to un-dodgeable Hidden Powers from Electrics, but on the flip side, can put things to sleep for free setup. Recommended set is Sludge Bomb/HP Fire/Swords Dance/Sleep Powder. Its main problem is that it's basically made of glass. If it could tank any sort of hit, it might be useful, but it can't, so it's not. It can be pretty effective in UU, although it's still mega-frail and pretty slow for what it wants to do. Nidoqueen is also a top 3 mon and really effective at shutting it down. If you want a glass cannon in UU, Dodrio is a better bet. If you want a Swords Dancer, Pinsir is better with its superior Speed and near-OHKO on Slowbro at +2.