You're right that that is a theoretical possibility, but I think it's very unlikely in practice. In the VC world, term sheets are only technically non-binding; breaking a signed term sheet is considered a Big Deal.
It's a big deal, but it happens and not always for the right reasons. Imnsho an investor should only walk away from agreed upon terms if something nasty and previously undisclosed turns up during DD.
This may have changed, but a few years ago multiple friends with YC companies told me there were consequences for VCs, w/r/t YC, who ruthlessly withdrew term sheets.
Absolutely, word gets around and before you know it the stream of decks dries up and other people no longer want you as co-investors in rounds, so not just YC where there are consequences for such trickery. This is just bad for the industry as a whole, a lot of this hinges on trust and if parties start breaking that trust capriciously then that is a problem.
What I have seen - and just once - is that a DD did in fact turn up some major issues, the investors backed out and the startup then made a stink pretending they were stiffed. But the truth on that one will likely never see the light. So not 100% of these can be laid at the door of the VCs either.