A SOTA quantization algorithm for high-accuracy low-bit LLM inference, seamlessly optimized for CPU/XPU/CUDA, with multi-datatype support and full compatibility with vLLM, SGLang, and Transformers (github.com)
from yogthos@lemmy.ml to technology@lemmy.ml on 01 May 20:01
https://lemmy.ml/post/46704869

#technology

threaded - newest