writing some written qualifier questions today, and including this at the bottom. interested to see how it goes.

"When I grade your answer to this question, I will start by posing the question above, verbatim, to a couple of frontier LLMs. Your response will be graded in terms of how much better I believe it to be than what I get from the LLMs."

@regehr How well would the LLMs do on quals if you were grading without knowing about LLMs?

@xentrac interesting question. until very recently, I don't think very well, but they've gotten better and keep surprising me. even so, I've never seen anything like an insight from an LLM, but they are starting to at least competently summarize things.

@regehr I might just have low standards for insights, but I fairly frequently have Opus 5.5 point out a solution to things that seem like difficult design problems to me. Sometimes they're wrong, but not always, and I'm used to a lot of my own "insights" being wrong when I work through them too.

@regehr Plausibly nothing I'm doing is doctoral level work.

For example, the thing I've been exploring most over the last month is a mutation-free language I'm calling "5emel", which samples the program's call stack to figure out which functions are worth memoizing, with the objective of getting programs that are simpler, more adaptable to memory pressure, and automatically incremental without the large slowdowns reported in work like Acar's "self-adjusting computation". I feel like this might be doable if I can steer around the pathological cases that cause time or space to explode, but it's crucial to keep the overhead of normal, non-repeating computations low enough to stay viable.

But possibly the project is doomed, in which case it would not be viable as a thesis project if I were in a doctoral program.

@xentrac sounds cool!