Something I've been trying to find out about fixed- vs variable-costs of LLMs:
I've seen descriptions of the energy cost to TRAIN an LLM, but that's a one-time thing (as currently constituted). But how much energy does a "query" take? (Expert answers only please.)
@shriramk I only found this one
“Martin Bouchard, cofounder of Canadian data center company QScale, believes that, based on his reading of Microsoft and Google’s plans for search, adding generative AI to the process will require “at least four or five times more computing per search” at a minimum.”
here
https://www.wired.com/story/the-generative-ai-search-race-has-a-dirty-secret/
So one would have to know generic search costs to calculate it
@CerstinMahlow This is pretty useful:
https://towardsdatascience.com/the-carbon-footprint-of-chatgpt-66932314627d