What I'm finding more and more over time is that the vast majority of devs I come across don't have a good barometer for performance, e.g. "this load should take about X resources" - they just don't have a point of reference for what good performance looks like and so the thing they're working on doesn't seem slow or problematic.

And that's because most things aren't slow or fast. They're slower or faster than other things you know of or someone's expectations and in this case there is no reference point set for what "good" is.

For example, we get issues all the time filed for timeouts on machines running 25-100 requests/sec yet exhausting hundreds of simultaneous threads on the pool - there's no point of reference they've seen for all those synchronous calls and thread exhaustion being terrible for performance and pile-up issues. They don't know with proper async for I/O calls there would be very few threads concurrently in use and it'd eliminate most exhaustion stalls.

I'm not sure how to improve the learning here, but I feel people need more comparative examples available that they can reference and sanity check against vs. their own workloads to gut check and go "oh, maybe this is slower than it could be and deserves some investigation".