For @emilymbender , but also anyone else who has an opinion on the subject. :)
I've been rereading "Thought experiment in the National Library of Thailand" and thinking about my own early (long pre-LLM) thoughts about Searle's Chinese Room, and wondering where I actually stand on various questions at the moment.
And I realize that it's not clear to me exactly what Bender is saying, in "Thought experiment".
In particular, on the claim that "it is possible to manipulate forms well enough to be indistinguishable from a system that understands the meaning of the forms, reasons about it, and responds appropriately", is Bender denying that that is possible?
Or is it just that, even if a system does that form-manipulating well enough to be externally indistinguishable, the form-manipulating system will still not, in the deeper sense, be understanding and reasoning and responding?
That is, will the person who attempts to learn Thai solely by frowning at lots of Thai, in fact fail to be able to fool a Thai speaker? Or is the point that, even if a Thai speaker is convinced, they will have been fooled?
I would say that all of your take aways are valid from this exercise. The point is that if you spend enough time looking at Thai writing you'll be able to know "oh, after these x Thai characters it should be followed by these other y characters", but you would still not know anything about the meaning of the Thai characters that you are writing. You wouldn't be able to tell me the meaning of the writing nor translate it, for example.
The point is that these llms are
"You wouldn't be able to tell me the meaning of the writing nor translate it, for example."
But if you were to give me some Thai writing that said (in Thai) "What does this mean?" followed by a sentence in Thai, I would be able to give you back some Thai that explained (in Thai) the meaning of the sentence.
And if you were to give me some Thai writing that said "Please translate this into English", I could give you back writing (in Thai) saying "Sorry, I don't speak English!".
(Now no human could actually do this, it's not how our brains are built. But an LLM trained purely on the contents of the redacted National Library of Thailand could! At least I think it could. I'm not sure whether Bender is saying it couldn't, at least not well enough to be indistinguishable from a human Thai speaker.)
Not quite. "I would be able to give you back some Thai that explained (in Thai) the meaning of the sentence."
You would be able to give back some Thai that PROBABLY explained the meaning, maybe.
But the point is that you still don't know the meaning yourself. You wouldn't be able to actually consider the question as a question.
The way I would put it is that there is a system here that is indistinguishable from a Thai speaker, in terms of how it communicates in written Thai.
I have implemented the system in my brain (which is implausible because brains don't work like that, but we stipulate it for the sake of the example), but does that matter?
There is a system here to which I can pass written Thai, and get back what seem to be entirely fluent responses in written Thai.
On what basis do I say that it doesn't really understand, or doesn't "get to meaning" or whatever?
Searle would probably say that since *I* (the English speaker) don't understand, then the system doesn't; but that seems wrong to me. The system insists that it DOES understand, so unless we have some principle like "a human body can't contain two different thinking systems" (which seems weird to me) I don't know how we say that we know it's wrong.
Now maybe you and/or Bender et al are saying that in fact there can be no system which, having experience of nothing but written Thai, can produce Thai responses that are indistinguishable from those that a Thai-speaking and understanding human would produce. That would be fine, and we can determine that empirically.
But perhaps you/they are saying that EVEN IF the system can produce those indistinguishable responses, we still know that it doesn't truly understand. And in that case I'm very curious for the basis of that.
Not sure if that makes my uncertainty any clearer. :)
I guess another take on the Systems Reply, and this is maybe just a less snarky version of Searle's, is that since I am the only person there (in the National Library), and I don't understand Thai, therefore of course no one understands Thai; who else is there?
But for me that's the "there can be only one intelligent system in a person" argument.
It's probably true in fact, but in the counterfactual conditions that we've stipulated, a person acquiring the ability to do purely syntactic transforms on Thai writing without understanding Thai, I am perfectly willing (in fact, forced) to say that there are two people (two understanders) here who happen to share the same body; the English speaker and the Thai speaker.
It's especially weird (and counterfactual) that the English speaker is part of the instantiation of the Thai speaker, but that's just because the example is so factually impossible.
Getting less impossible by allowing the English speaker in the Thai National Library to use various degrees of external aids, starting at slips up paper and then going up through ledgers and index cards, to whole filing systems and then ultimately digital computers, we eventually get a situation where hypothetically a different person could be relatively quickly trained to use the system, and so whether or not that person speaks Thai comes to seem less relevant.
Or if we give the person not just some slips or paper and a copy of MySQL, but a really big neural network, we end up with them creating an LLM trained by feeding it everything in the redacted Thai National Library.
Does that LLM actually understand Thai? Well, that's exactly the question that we came in with :) so it seems like the thought-experiment hasn't really gotten us anywhere, if we think of it this way.
The fact that the person feeding the data to the LLM doesn't speak Thai is, at least, very obviously irrelevant; and that leads me to think that it was irrelevant all along...
