Remember seeing something about GPT-4 doing well on standardized tests? It turns out it may have memorized the answers.
https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks
#gpt4 #AIHype #ThisIsWhyWeDontTestOnTheTrainingData
ehhh memorize isn't even the right word. still anthropormophizing too much, dagnamit
@quinn @janellecshane @KevinMarks talk of storage as computer memory, as in RAM/ROM, is standard
@quinn @janellecshane I might have said it memory mapped a file, but that's a different metaphor.
You could say that the LLM had already read the answers to those questions (again with the existing metaphor of reading data)
@KevinMarks @janellecshane It's just important to remember that these amazing new AIs are still closer to being very elaborate magic 8-balls than they are to being children, or even cats.
@mapto @janellecshane @KevinMarks when is the last time you said that your computer memorized that file?