home/categories/llm-ai/eyadsibai-ltk-plugins-ltk-data-skills-huggingface-tokenizers-skill-md

llm-aidata-ai

huggingface-tokenizers

Name: huggingface-tokenizers
Author: eyadsibai

Use when "tokenizers", "HuggingFace tokenizer", "BPE", "WordPiece", or asking about "train tokenizer", "custom vocabulary", "tokenization", "subword", "fast tokenizer", "encode text"

查看源碼 llm-ai

maintainer

eyadsibai

更新於 1/15/2026

星標

分支

quick start

Installation and usage

Use when "tokenizers", "HuggingFace tokenizer", "BPE", "WordPiece", or asking about "train tokenizer", "custom vocabulary", "tokenization", "subword", "fast tokenizer", "encode text"

安裝

$ install --globalskills.sh

使用

安裝後，您可以通過在終端運行以下命令來使用此技能：

skills use huggingface-tokenizers