I think disagreeing with a comment is not enough to merit a downvote. I'd rather see downvoting as a signal to the author about the quality of their post - while flagging is a signal to the system itself in cases where the post is unacceptable and should be removed from the site.

When I disagree with a comment, I'd rather write a dissenting one or a reply stating my counter points. Another method of dealing with disagreements is to simply upvote other comments you agree with.