Do you feel that you may be being a bit provocative by calling LLM's 'bullshit machines'?
I understand the frustration as I've been bullshitted by these models just as much as the next programmer, but surely with recent advancements in RAG and reasoning, they're not just 'bullshit machines' at this point, are they?
Or are they modern day oracles? The full title is inclusive and invites consideration. Besides, how many different types of models are there? How are they used? For freely available ones made accessible to the general public via interfaces, when were they last updated? Do the interface implementors even care or was it a simple project to make money via ads and microtransactions?
This is a new area of media literacy and requires critical thinking.
Nonetheless, I do acknowledge the value of RAG-based approaches that attempt to qualify their reasoning through provided sources.
It is literally a machine that says what you want to hear. Was anyone in this thread around for Talk to Transformer? Remember the unicorns demo? They may have now figured out how to wrangle the technology so it's more likely to spit facts, but it's still the same thing underneath.
I wrote a post below about how AI hallucinated a whole regulation that didn’t exist which has been flagged for some reason.
I have colleagues who have had arguments with clients who have asked AI questions about planning law and been given bullshit which they then insist is true and they can’t understand why thier architects won’t submit the appeal that they’re asking for.
I think we’re in an era where any text, true or not, is so easy to generate and disseminate that the status of the written word is reduced to the standard of the gossip that used to be our main source of information before the printing press was invented. Now half the internet is AI generated bullshit as well.
Oh right, sorry. Should I have linked to it? It seemed pretty relevant to this discussion and most of it was quoting something so I reused a lot of it. Didn't realise that was taboo.
> In philosophy and psychology of cognition, the term "bullshit" is sometimes used to specifically refer to statements produced without particular concern for truth, clarity, or meaning, distinguishing "bullshit" from a deliberate, manipulative lie intended to subvert the truth.
Waffle means something very different though in the U.S., to “flip-flop” on a position. Not hold it for any fundamental reasons. But I don’t think you can say that an LLM holds a position whatsoever. Also, https://en.m.wikipedia.org/wiki/On_Bullshit the essay originally popularizing the definition of bullshit considered here. Note the references to LLM output at he bottom.
> It’s really an ideal term to describe what LLMs do.
Only when you relax it to the point it also describes what most people do.
To be specific: if by "truth" you mean objective, verifiable truth, and by "caring about truth" you mean caring about objective, empirically verifiable evidence, then by far most people are mostly only ever trying to be persuasive.
Why? I don't have a problem recognizing that "being convincing" to the right people is a very good way of learning and verifying knowledge about objective reality. Just because it's by proxy, doesn't mean it doesn't work. This is how every one of us learns most things, and what's really sad to me is deluding yourself into thinking we're more special than that.
Honestly, it just sounds to me like you’ve come up with some axioms in your mind about the “fundamental nature” of humanity - and then granted yourself the luxury of certainty about them. That, with a dash of misanthropy, is something I’ve been seeing more and more on HN these days.
I have been thinking about this and I have an idea for local LLM I want to try.
It is based on the assumption LLMs make mistakes (they will!) but be confident about it. Use a 1B model and ask it about a city for fun mistakes in an otherwise impressive array of facts. Then you will see how bullshitty LLMs are.
Anyway the idea is to constrain the LLM and a language understanding and to choose a constrained response.
For example give it the capability to read out something from my calendar. Using the logits it gets to choose what that calendar item is. But then regular old code does the lookup and writes a canned sentence saying "At 6pm today you have a meeting titled $title".
This way, my meeting schedule won't make my LLM talk like a pirate.
This massively culls what the LLM can do but what it does do, is like a search and so just like Google (before gen AI!) you get a result but you can judge it as a human.
[quote]BULLSHIT involves language or other forms of communication intended to appear authoritative or persuasive without regard to its actual truth or logical consistency.[/quote]
That is an emotionally manipulative definition and anthropomorphizes the LLMs. They don't "intend" anything, they're not trying to trick you, or sound persuasive.
I've never used Claude, but Perplexity often says that no definitive information about a topic could be found, and then tries to make some generalized inferences. There's a difference between a specific implementation, and the technology in general.
In any case, it's worthwhile for people to understand the limitations of the technology as it exists today. But calling it "bullshit" is a mischaracterization; I believe based on an emotional need for us to feel superior, and to dismiss the capabilities more thoroughly than they deserve.
It's a little like someone saying in the industrial revolution, "the steam shovel is too rigid, it will NEVER have the dexterity of a man with a shovel!". And while true and important to know, it really focuses on the wrong thing, it misses the advantages while amplifying the negatives.
As the technology exists today: imperfect, often prone to mistakes, and unable to relay confidence levels. These problems may be addressed in future implementations.
That's the same message, without any emotional baggage, or overly dismissive tone.
That would be great if those who are selling the technology described it that way. I, and apparently others, feel like maybe "bullshit" is a better counter to the current marketing for LLMs
> Being persuasive (i.e., churn out convincing prose) is how LLMs were designed to be.
No. They were designed to churn out accurate prose that accurately reflects their model of reality. They're just imperfect. You're being cynical and emotional to use the term bullshit. And again, it anthropomorphizes the LLM, it implies agency.
> that accurately reflects their model of reality.
you are also seemingly anthropomorphising the technology by assigning to it some concept of having a “model of reality”.
LLM systems output an inference of the next most likely token, given: the input prompt, the model weights and the previously output token [0].
that is all. no models of reality involved. “it” doesn’t “know” or “model” anything about “reality”. the systems are just a fancy probability maths pipelines.
probably generally best to avoid using the word “they” in these discussions. the english language sucks sometimes. :shrug:
[0]: yes i know it is a bit more complicated than that.
It literally has a mathematical model that maps what would, colloquially at least, be known as reality. What exactly do you think those math pipelines represent? They're not arbitrary numbers; they are generated from actual data that is generated by reality. There's no anthropomorphizing at all.
a corpus of training data from the internet is finite.
any finite number divided by infinity ends up tending towards zero.
so, mathematically at least, the training data is not a sufficient sample of reality because the proportion of reality being sampled is basically always zero!
fun with maths ;)
> What exactly do you think those math pipelines represent?
probability distributions of human language, in the case of text only LLMs.
which is a very small subset of stuff in reality.
-
also, training data scraped from the public internet is a woeful representation of “reality” if you ask me.
that’s why LLMs i think are bullshit machines. the systems are built on other people’s bullshit posted on the public internet. we get bullshit out because we made a bunch of bullshit. it’s just a feedback loop.
(some of the training data is not bullshit. but there is a lot of bullshit in there).
You're really missing the point and getting lost in definitions. The entire point of human language is to model reality. Just because it is limited, inexact, and imperfect does not disqualify it as a model of reality.
Since LLMs are directly based on that language, they are definitely based on and are a model of reality. Are they perfect? No. Are they limited? Yes. Are they "bullshit"? Only to someone who is judging emotionally.
> The entire point of human language is to model reality.
is it? are you absolutely certain of that fact? is language not something that actually has a variety of purposes?
fiction novels usually do not describe our reality, but imagined realities. they use language to convey ideas and concepts that do not necessarily exist in the real world.
ref: Philip k dick.
> Since LLMs are directly based on that language, they are definitely based on and are a model of reality.
so LLMs are an approximation of an approximate model of reality? sounds like the statistical equivalent of taking an average of averages!
i am playing with you a bit here. but hopefully you see what im getting at.
by approximating something that’s approximate to start with, we end up with something that’s even more approximate (less accurate), but easier than doing it ourselves.
which is the whole USP of these things. why think about things when ChatGPT can output some approximation of what you might want?
> so LLMs are an approximation of an approximate model of reality?
Yes, and we as humans have a mental model that is just an approximation of reality. And we read books that are just an approximation of another human's approximation of reality. Does that mean that we are bullshit because we rely on approximations of approximations?
You're being way too pedantic and dismissive. Models are models, regardless of how limited and imperfect they are.
Now we're deeper into it -- I actually agree, somewhat. See above for deeper insight.
These LLM systems output "stuff" within our reality, based on other things in our reality. They are part of our reality, outputting stuff as part of reality about the reality they are in. But that doesn't mean the statistical model at the heart of an LLM is designed to estimate reality -- it estimates of the probability distribution of human language given a set of conditions.
LLMs are modelling reality, in the same way that my animal pictures image classifier is modelling reality. But neither are explicitly designed with that goal in mind. An LLM is designed to output the next most likely word, given conditions. My animal pictures classifier is designed to output a label representative the input image. There's a difference between being designed to have a model of reality, and being a model of reality because the thing being modelled is part of reality anyway. I believe it's an important distinction to make, considering the amount of bullshit marketing hype cycle stuff we've had about these systems.
edit -- my personal software project translating binary data files models reality. Data shown on a screen on some device modelled as yaml files and back again. Most software is an approximation of reality soup stuff. which is why I kind of don't see that as some special property of machine learning models.
> Does that mean that we are bullshit because we rely on approximations of approximations?
The pessimist in me says yes. We are pretty rubbish as a species if you look at it objectively. I am a human being that has different experiences and mental models to you. Doesn't mean I'm right about that! Which is why I said "I think". It's just my opinion they are bullshit machines. It is a strong opinion I hold. But you're totally free to have a different opinion.
Of course, there's nuance involved.
Running with the average of averages thing -- I'm pretty good at writing code. I don't feel like I need to use an LLM because (I would say with no real evidence to back it up) I'm better than average. So, a tool which outputs an average of averages is not useful to me. It outputs what I would call "bullshit" because, relative to my understanding of the domain, it's often outputting something "more average" than what I would write. Sometimes it's wrong, and confident about being wrong.
I'd probably be pretty terrible at writing corporate marketing emails. I am definitely below average. So having a tool which outputs stuff which is closer to average is an improvement for me. The problem is -- I know these models are confidently wrong a lot of the time because I am a relative expert in a domain compared to the average of all humans.
Why would I trust an LLM system, especially with something where I don't feel like I can audit/verify/evaluate the response? i.e. I know it can output bullshit -- so everything it outputs is now suspected, possible bullshit. It is a question of integrity.
On the flip side -- I can actually see an argument for these things to be considered so-called Oracles too. Just, not in the common understanding of the usage of the word. Like, they are a statistical representation of how we as a species use language to communication ideas and concepts. They are reflecting back part of us. They are a mirror. We use mirrors to inspect our appearance and, sometimes, to change our appearance as a result. But we're the ones who have to derive the insights from the mirror. The Oracle is us. These systems are just mirrors.
> You're being way too pedantic and dismissive.
I am definitely pedantic. Apologies that you felt I was being dismissive. I'm not trying to be. The averages of averages thing was meant to be a playful joke, as was the finite/infinite thing. I am very assertive, direct and kind of hardcore on certain specific topics sometimes.
Where in the loss function of LLM training is the relationship between their model of reality and their predicted tokens? Any internal model an LLM has is an emergent property of their underlying training.
(And, given the way instruct/chat models are finetuned, I would say convincing/persuasive is very much the direction they are biased)
No, the lesson or the quote is not anthropomorphizing LLMs. It is not the LLM that "intends", it is the people who design the systems and those who make/provide the training data. In the LLM systems used today the RLHF process especially is used to steer towards plausible, confident and authorative sounding output - with no to little priority for correctness/truth.
I really hope that was intentional and the full effect of that naming choice was known beforehand, because I have already written the whole thing off, and I don't believe I'm the only one.
I understand the frustration as I've been bullshitted by these models just as much as the next programmer, but surely with recent advancements in RAG and reasoning, they're not just 'bullshit machines' at this point, are they?