Man Cave? No... No... No... That term is bullshit. It implies that you're a crude, dumb son of a bitch, when it's the rest of the world that has gone fucking retarded. Eventually they'll shut the fuck up, and come to realize that things were fine the way they were, because that's how evolution ended up. Animals in the wild take the easy way when doing anything, so why the hell does humanity like to make shit complicated for the rest of humanity, and then worship every turd a deer craps out?
Man Cave? Fuck no. You own the house, you have a MANsion.
You want more space? Build an addition the size of the current house. Clear out the clutter somewhere where there's too many shoes. The wife wants a zen space? Buy one of the turtle sand box things and fill it with kitty litter so she can rake out the cat crap like the Japanese do(this is a joke, but seriously, domestic cats ruin many good Japanese sand gardens thinking that they are litter boxes, please spay and neuter your cats, as some of them are simply retarded enough to do this).
Just remember that once you have a combination movie theater/reloading room/indoor grille and BBQ pit, that you're on the right track. Nothing is too much. Not at all. Every king has his castle, and the only people who really hate on a guy who has one, are ones that don't have their own along with the moats, drawbridges, and murder-holes to go with it. Why do you need a 36 foot fucking tower with a goddamn keep? Because you want a 36 fucking foot tower with a goddamn keep. Fuck the zoning boards!
Do it...Because you've earned it.