My daughter, who has had a degree in computer science for 25 years, posted this observation about ChatGPT on Facebook. It's the best description I've seen:
@DrewKadel
That is very brilliant, with one caveat. It's not people wishing it to give actua answers, it's marketed and sold as something that dies just that.
"May occasionally generate incorrect content" is a hoax: LLMs by design are incapable of associating a likelihood value to their output.
Their output is totally randomly, and totally by design, true or false, without any specific interest in either. LLMs are bullshit[1] engines
[1]: Frankfurt, H. On bullshit, 2005
@dataKnightmare On Twitter someone responded that my daughter's description was a better way of saying what he's been saying:"It's responses are like what a college freshman who hasn't done the reading would say while trying to bullshit their way through a seminar."
@DrewKadel @dataKnightmare More like they have done their reading... But don't have time to reason properly and systematically and are forced to BS their way through a seminar.
The quality of answers improves significantly when you give them that time ("lets think step by step" etc.).
Given more compute LLMs would be able to do Fermi Estimates on par with the dude himself, if not better. There's nothing putting a hard limit on that, since even the humans are able to do it when motivated.
And with plugins like Wolfram they could build a proper model and perform exact calculations on it.
