@msprout I don't really work in AI things, but according to some twittes, this model use float16 rather than float 8, meaning each parameter use 2 bytes, thus the smallest one, aka the 7B model, will use at least 14GB of VRAM. My 3070 only has 8GB of VRAM. LOL

I got 64GB of RAM, I will try running it on CPU