If you are really worried, use layer 3 or layer 2 security (say with IPSec) to secure NTP communications.

Yes, there is a bit of a bootstrapping problem, but you can address that with a bootstrapped handshake that sets a clock baseline.

Alternatively, you could just hardwire a radio receiver (like say... a GPS receiver).