writing some written qualifier questions today, and including this at the bottom. interested to see how it goes.
"When I grade your answer to this question, I will start by posing the question above, verbatim, to a couple of frontier LLMs. Your response will be graded in terms of how much better I believe it to be than what I get from the LLMs."
@regehr How well would the LLMs do on quals if you were grading without knowing about LLMs?
@xentrac interesting question. until very recently, I don't think very well, but they've gotten better and keep surprising me. even so, I've never seen anything like an insight from an LLM, but they are starting to at least competently summarize things.
@regehr I might just have low standards for insights, but I fairly frequently have Opus 5.5 point out a solution to things that seem like difficult design problems to me. Sometimes they're wrong, but not always, and I'm used to a lot of my own "insights" being wrong when I work through them too.
@regehr Plausibly nothing I'm doing is doctoral level work.
For example, the thing I've been exploring most over the last month is a mutation-free language I'm calling "5emel", which samples the program's call stack to figure out which functions are worth memoizing, with the objective of getting programs that are simpler, more adaptable to memory pressure, and automatically incremental without the large slowdowns reported in work like Acar's "self-adjusting computation". I feel like this might be doable if I can steer around the pathological cases that cause time or space to explode, but it's crucial to keep the overhead of normal, non-repeating computations low enough to stay viable.
But possibly the project is doomed, in which case it would not be viable as a thesis project if I were in a doctoral program.
@regehr But the bot has been very valuable in suggesting things like:
- what if we hash things lazily?
- what if we use Baker's shallow-binding arrays to provide persistent arrays that are efficient for single-threaded use?
- what if we admit functions to be memoized only once we see repeated arguments?
- what if we unwind functions when we sample them stochastically and find that we stored the result they are currently trying to compute during a previous sample, but the function isn’t yet flagged for memoization, so it didn't notice?
Although I think the first three of these are straightforward cases of applying already published results, the fourth doesn't seem to be. I don't know if it will make it into the final system, but it seems like an important attack to try on the problem.
Also, I think that when I first started trying to elicit this implementation from the AI, I had only vague and confused ideas about how to go from flagging an on-stack function for memoization to having it record its in-progress result once it finishes, and Opus (5.5 or 5, not sure) pointed out the in-retrospect-obvious idea of changing the return address in its stack frame, and that this also allows us to avoid walking the same already-flagged frames repeatedly, thus doing a potentially large amount of repeated work.
