I'm sure if a ghost tried hard enough even I'd lose it, but there's really little they can do without you letting them. Besides, they aren't even in a physical body, so they're not quite as threatening to me as someone breaking into my house would be.

That, and I've always had a "mess with me if you dare" attitude in regard to anyone or anything trying to come at me.