skills.homes logoskills.homescapability registry
home/categories/machine-learning/orchestra-research-ai-research-skills-02-tokenization-sentencepiece-skill-md
cd ..
machine-learningdata-ai

sentencepiece

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

Ver código-fontemachine-learning
Orchestra-Research
maintainer
Orchestra-Research
Atualizado 11/20/2025
Estrelas
6563
Forks
515
quick start

Installation and usage

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

Instalação
$ install --globalskills.sh
Uso

Depois de instalar, você pode usar esta skill executando o seguinte comando no terminal:

skills use sentencepiece