You can't say with a straight face that the founders thought Constitutional protections should apply to everyone, even people abroad, when they declined to extend any protection to black men living in America. Or Indians. Or women.
Except that you can say it with a straight face. The fact that they had to explicitly call out exceptions for Indians and Blacks means that they felt that they would have otherwise fallen under the general classification of "men".