But they predict based on context. If you prompt it with "You are a creative writing bot that excels in generating unpredictable, yet coherent text", does it predict what is unpredictable?
Yes, that's all it can ever do, predict a continuation. It's not a person though, so it'd not be predicting what you might find unpredictable, but rather predicting training samples that it associated with unpredictability (e.g. with some following feedback of "well, that was unpredictable!"). IOW it'll be predicting the most predictable unpredictability that it can.
