Omastar all the way.
Both Pokemon have very cool designs in their own way. Omastar, however, is a cut above the competition (oh, the irony) for several reasons.
1. Realism. Varieties of ammonites that sort of resembled Omastar did exist, whereas Kabutops is a fanciful takeoff on marine crustaceans. Moreover, as Rodan mentioned, scythes would be impractical and unwieldy on land, let alone in the sea (the denser medium would make it more difficult for Kabutops to move its scythes fast enough to use them effectively for hunting).
2. Overall design. While I do love the warrior vibe I get from Kabutops and its sleek appearance, Omastar just has more going for it in this area. Omastar's eyes and beak look alien enough to be truly menacing, and the presence of tentacles helps there (although the four big ones look more cute than deadly). The spikes on Omastar's shell are understated enough that they don't immediately grab your attention like Kabutops' scythes, but they're strange enough to accentuate the "terror of the ancient sea" image.
3. Nostalgia. Omastar actually introduced me to Pokemon. When I was a kid, I found an Omastar card from the Fossil expansion lying on the ground somewhere. Fascinated by this bizarre creature, I looked at the card and saw it was "Pokemon". I asked one of the neighbors' kids what this was (funnily enough, this sick individual wasn't into Pokemon at ALL), and he explained the basic premise of it. I was hooked immediately, and started watching the anime and collecting the cards as soon as I could (I was unable to get a game until Ruby's release, though, since my parents wouldn't buy me a Game Boy and I had to save up for it on my own). Omastar is singlehandedly (single-tentacledly?) responsible for introducing me to what became my biggest obsession, and he's consistently remained in my top ten favorite Pokemon since the beginning.
