I have never been very happy with the madness rules for roleplaying games, be they the "gotta catch them all" of the Cthulhu games, or White Wolf's terrible derangement system. They are neither horrible nor interesting, or are social penalty window dressing --always a poor game mechanic choice. Insanity ought to be macabre as fuck in most games.
Below is a short in the dark at what I'm more interested in. I would have these be caused by particular monsters or encounters if a saving throw is failed, or possibly arrange for a table of them. Though I hate the idea of insanities that are completely disconnected from their cause.
I also considered having a Willpower save version based on the disease rules from d&d 4e, but am essentially too lazy to stat that, though the mechanic would better suit insanity than disease -- the latter is too boring to be the focus of much of a game session, and therefore there is no need for such a sophisticated mechanic. The former, however, is interesting enough to have a sliding scale and so forth.
These result in some constant penalty or semi-permanent effect.
Formication: You sense what feels like a swarm of writhing insects underneath your skin. You start each encounter at a maximum of half your maximum hit points because of self-inflicted wounds from scratching and attempts to dig them out.
Maliced Hand: Your hand does not obey your commands, but rather does its own thing. It cannot be used helpfully and, left unrestrained will use weapons or spells (as appropriate) to attempt to harm your friends. The hand can do anything you can do, and uses your skills and abilities (including make escape artist checks).
Otherworldly Nightmares: You suffer from torturous nightmares that subject you to alien locales and mindsets, and prevent you from effectively resting. You do not heal naturally, and if applicable, memorize one less spell of the highest level that you can cast.
Anhedonia: You cannot experience pleasure. This prevents you from benefiting from spells or effects that impart morale bonuses. Alas, you are all too capable of experiencing suffering, and penalties of that sort affect you normally.
Each day that you do not indulge in these, you suffer -1 to attack rolls and spell DCs, until your bonuses reach 0.
Dark Hunger: You must feast upon your own kind. One average human-sized body will sustain a character with this madness for three days.
Morbid Fascination: You want to make art out of fresh human body parts, bathe in gallons of human blood, or otherwise indulge yourself at the expense of freshly dead enemies. One human will sustain a character with this madness for a day.
Alien Limb: You become convinced that one of your major limbs or body parts is not a part of you, and must be removed. Left to your own devices, you will take all appropriate measures to sever the limb or organ. Roll on the following chart (d6): 1-3: Leg, 4: Secondary Arm, 5: Eyes, 6: Primary Arm.
You follow some terrible rule, like never entering a sunlit area. A Willpower save may be attempted to temporarily suppress these for an encounter, with the DC of the effect being the same as the thing that caused the effect.
Clinical Lycanthropy: You believe that you are afflicted by lycanthropy. On nights when the moon is visible and not overly occluded, you will must make a Will save or enter a form of sleepwalking in which you will attempt to kill a member your own species. While in this state you will not respond to language and will especially brutally. If damaged while at less than half your maximum hit points, you may make a Willpower save to shake off the effect for the remainder of the night. In any case, you will remember nothing about your time spent sleepwalking.
Solar Horror: You cannot bear the light of the sun, and will not willingly enter a sunlit area. If you are forced to (or force yourself to), you suffer a -5 penalty to your attack rolls and spell DC. As a variant, one might have a lunar horror.
Dread of Magic: You cannot stand to have spells cast upon you, even beneficial spells. You automatically attempt saving throws against all spells, including things like cure spells.
Piquerism: You refuse to use a weapon that does not deal piercing damage because you crave causing piercing injuries. A successful willpower saves will only suppress this for a round.