Microsoft just open-sourced bitnet.cpp, a 1-bit LLM inference framework. It let's you run 100B parameter models on your local CPU without GPUs. 6.17x faster inference and 82.2% less energy on CPUs.
(github.com)
from yogthos@lemmy.ml to technology@lemmy.ml on 05 Jan 19:51
https://lemmy.ml/post/41280771
from yogthos@lemmy.ml to technology@lemmy.ml on 05 Jan 19:51
https://lemmy.ml/post/41280771
threaded - newest