home/categories/documents/davila7-claude-code-templates-cli-tool-components-skills-ai-research-tokenization-sentencepiece-skill-md
documentscontent-media
sentencepiece
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
maintainer
davila7
更新於 1/20/2026
星標
17577
分支
1576
quick start
Installation and usage
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
安裝
$ install --globalskills.sh
使用
安裝後，您可以通過在終端運行以下命令來使用此技能：
skills use sentencepiece