How about adding a custom class in the game itself in the first place?
The player could choose 3 abilities from a pool of already existing/unlocked ones and tweak their numbers.
The class wouldn't count in order to unlock anything /quests.
You could experiment all you want and give yourself challenges without removing the difficulty of the overall game, even without being a modder yourself!