If you look at dedicated writing AI tools like NovelAI, you can see that this is exactly how they work. They have a concept of a "lore book" with "keywords" that you can use to remind the AI about how to properly generate the content.
For example, if I were writing a novel about a fantasy kingdom and there is an evil king, then I can write a description of the king ("The king is evil and wears a red crown") with the keyword to "include" it being "crown", "king", etc.
As the number of tokens increases with these AIs I think this problem will decrease decrease (1k was NovelAI originally, and now GPT-3 Codex is 8k). I do wonder what the "ideal" solution is in the future though. Or do you have to create a "lore book" for ChatGPT too?
(A thought I've had is that maybe we can automatically "create" the "lore book" and then re-inject that seamlessly? For example, in the fantasy kingdom example, maybe ChatGPT can generate a list of characters for each "chunk" of text and then automatically inject that into the "last" batch of text.)
But if we are looking at a thinking AGI that is “superior” to human abilities we should see Tolkien or Martin. If the argument is “just wait” then there needs to be sone explanation of why a Tolkien is really possible with this type of AI.
Funny you mention those two. They also suffer from the problem: “starts to break down after a long running thread before eventually becoming almost useless.”
Tolkien kept changing his world as he went along, so the Lord of The Rings required him to make changes to the Hobbit, and his Silmarillion needed extensive edits to bring it back into synch with LoTR canon.
As for Martin (assuming you mean G.R.R.)… well, he’s already mimicking that other failure mode of ChatGPT - that response times can get long when under load, and eventually the whole session might just time out.
For example, if I were writing a novel about a fantasy kingdom and there is an evil king, then I can write a description of the king ("The king is evil and wears a red crown") with the keyword to "include" it being "crown", "king", etc.
As the number of tokens increases with these AIs I think this problem will decrease decrease (1k was NovelAI originally, and now GPT-3 Codex is 8k). I do wonder what the "ideal" solution is in the future though. Or do you have to create a "lore book" for ChatGPT too?
(A thought I've had is that maybe we can automatically "create" the "lore book" and then re-inject that seamlessly? For example, in the fantasy kingdom example, maybe ChatGPT can generate a list of characters for each "chunk" of text and then automatically inject that into the "last" batch of text.)