skills.homes logoskills.homescapability registry
home/categories/documents/davila7-claude-code-templates-cli-tool-components-skills-ai-research-tokenization-sentencepiece-skill-md
cd ..
documentscontent-media

sentencepiece

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

Voir le code sourcedocuments
davila7
maintainer
davila7
Mis à jour 1/20/2026
Étoiles
17577
Forks
1576
quick start

Installation and usage

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

Installation
$ install --globalskills.sh
Utilisation

Après l'installation, vous pouvez utiliser ce skill en exécutant la commande suivante dans votre terminal :

skills use sentencepiece