In our book Calling Bullshit, Jevin West and I provide the following definition.
Bullshit involves language, statistical figures, data graphics, and other forms of presentation intended to persuade by impressing and overwhelming a reader or listener, with a blatant disregard for truth and logical coherence.
Notice that this is what large language models are designed to do. They are designed with a blatant disregard for logical coherence in that they lack any underling logic model whatsoever.
@ct_bergstrom one could also say that LLMs are designed with a blatant disregard for syntax, in the sense that they're not explicitly taught syntax. but syntax turns out to be very useful for predicting the next word, so LLMs learn syntax. and internal coherence is also useful for predicting the next word, so they _do_ learn that to varying degrees (dependent on scale, it seems)—just so far not to the degree humans do.
@talyarkoni This is an interesting comment. Yes, I guess you could say that. I suppose the difference is that vast amounts of text offer a good training set for learning syntax and a poor one for learning logic.
@ct_bergstrom yeah i think that's right. but another way to think about it is that this is also true of humans. many (most?) humans also can't solve these kinds of riddles without multiple rounds of prompting and correction.
I had the same reaction when people made a big deal of GPT-3's failure to do arithmetic. doing "basic" math isn't actually easy for the vast majority of humans; it takes a lot of time to develop both ontogenetically and phylogenetically. why should LLMs be different?
@talyarkoni @ct_bergstrom Yes, but it also fails on some easier logic problems that a child could answer..
That said, I'm startled by the physical intuition here where GPT4 understands that sliding a hat with a card underneath will change the position of the card.
