I dislike the term bullshit because it's use regarding ChatGPT does not match the dictionary version "stupid or untrue talk or writing; nonsense".
If your sentence "LLMs output is bullshit" is wrong you may be better off changing the sentence than rewriting the dictionary to fit your sentence.
I mean you can redefine words if you like, like how young people use sick and bad to mean much the opposite of what they did, which is fine as a fashion statement, but in trying to reason about LLMs it muddies the reasoning. Which of course is often why academics do it - see Hobbes, Calvin 1993 https://www.reddit.com/r/calvinandhobbes/comments/1300k80/ac...
I think a hallucinated sentence embedded in a paragraph of truth fits the definition: stupid and nonsense. A bullshitter can be right sometimes or even most of the time. They are still a bullshitter.
I once overheard a parent telling a kid that "whale blubber" was actually whale farts, and that all the people that used to kill whales did it to get their farts and how silly that was. Of course that's not at all true, but that kid believed it. I felt sorry for the kid being told such absurd things by someone they trusted.
I have to wonder if I ask an LLM enough times, if it would give an answer about whale blubber that involved whale farts. That parent may have read it or heard it somewhere else, and the LLMs may also have that disinformation.
"Garbage in, garbage out" is definitely something humans and LLMs share in common.
If your sentence "LLMs output is bullshit" is wrong you may be better off changing the sentence than rewriting the dictionary to fit your sentence.
I mean you can redefine words if you like, like how young people use sick and bad to mean much the opposite of what they did, which is fine as a fashion statement, but in trying to reason about LLMs it muddies the reasoning. Which of course is often why academics do it - see Hobbes, Calvin 1993 https://www.reddit.com/r/calvinandhobbes/comments/1300k80/ac...