You can hit him with weakenings or stone hearts, you can fireball him without softening him up, death protections work, and not burning your dodge potions if you have them makes him popcorn... Also, you can use Taurog's DP string, or TT's potions string to take him out with little problems. CYDSTEPP on priests also works in some variants, and so on and so forth... Even big health and Stonesking can get you places (I've binlored him to death occasionaly)
He does make you pay attention, that's true, but unless you go all in before you find him he's all right. Or the easiest boss by far with dodge and reflex (with the possible exception of the warlock).
I almost got pwned by Shifty Brickwork!