Downvotes on a comment have already affected your total karma score. Deleting the comment doesn't undo the karma loss. However, you have a chance to avoid further karma loss if you decide to delete the comment shortly after it starts getting negative reactions. (I have done this when I realized that my comment was getting downvoted for snark.)

I have deleted in the past, and saw my score immediately jump, but perhaps I was being upvoted elsewhere.

Or maybe it was the tooth fairy.

No idea.