This is cool, now we just need to locate 1,000,000 A100-80GB equivalent GPU-hours. If we had a SETI@Home type project setup for this, it would be straightforward - only $50K worth of electricity for the 65B model.
Given the immense momentum behind LLaMA, I'm pretty disappointed that Meta won't just open-source it, but I guess reproducing it is better long-term.
I think the time for a folding@home or Berkeley bionic@home style project is now. it will also serve as a backend server farm for all the university based research activities, thereby ensuring the research outcome are not beholden to any one company or benefactor.
I remember setting up my PS3 & home desktop for folding project. Its fair game especially if I can use the box to heat the room instead of the furnance.
Given the immense momentum behind LLaMA, I'm pretty disappointed that Meta won't just open-source it, but I guess reproducing it is better long-term.