home/categories/machine-learning/orchestra-research-ai-research-skills-02-tokenization-sentencepiece-skill-md
machine-learningdata-ai
sentencepiece
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
maintainer
Orchestra-Research
更新于 11/20/2025
星标
6563
分支
515
quick start
Installation and usage
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
安装
$ install --globalskills.sh
使用
安装后，您可以通过在终端运行以下命令来使用此技能：
skills use sentencepiece