Well, I think peole sort of think he's supposed to be a spellcasting class (tinker seems to be the "rogue" archetype, and crusader a "fighter" or "cleric" archetype) when in fact he's a hybrid/melle boss. I've herd it before that elves are really good because larger mana pool = better spirit sword. It's probably true, I should try it sometimes. However humans get their CP bonus damage applied without having the SS active. Also after the first spirit sword in a fight, you've allready used up all your mana and you'll definitely go for more than one at the boss fight (and if you spend mana on spirit sword you don't spend it on fireballs so it's a bit ho-hum on paper. And if you spend it on spells it might be harder to time your spirit swords right. Eh.).
I'm curious what you mean about the super high damage and pretty high damage? You'll have about the same damage whatever you do if you go human, but not forcing spells more than you need and timing conversions pays off in both the short term (burst damage, yay) and the long term (omg, my damage burst is getting progressively higher!). Spirit sword doesn't stack, so you'll always have "pretty high" damage for the strike you need (empty mana first), and every spirit sword will get +10% more for every spirit sword used during a run from being human.
VGT still takes building up resists and all that jazz, it's not that easy, but hey, it's fun
And I'm not sure if you can get so much CP over the cause of a run than there.
EDIT: Also something to try out is the Binlor Halfling Transmuter. If you're using stoneskins, and converting for spirit sword, you'll end up with lots of potions. The damage increases from binlor are % based so they sinergize with the SS, and the potions give you more fuel to keep hitting stuff. It might be a thought worth pursuing somewhere. Damage's nowehre nearly as crazy but your staying power comes from CP which you're focusing anyway (instead of say, drac or GG healing). Might not work on endurance runs like NC and VGT but I'm sure there's places where it would.
EDIT2: Sorry about the long posts and stuff - I could talk about transmuters all day <3
EDIT3: Does anyone know if SS refreshes you mana when you hit it so you always get full benefits even if you've spent all your mana? Cuz then elves are a bit better than I thought and I could shorten this post significantly