count-dataset-tokens
This skill provides guidance for counting tokens in datasets using specific tokenizers. It should be used when tasks involve tokenizing dataset content, filtering data by domain or category, and aggregating token counts. Common triggers include requests to count tokens in HuggingFace datasets, filter datasets by specific fields, or use particular tokenizers (e.g., Qwen, DeepSeek, GPT).
Installation and usage
This skill provides guidance for counting tokens in datasets using specific tokenizers. It should be used when tasks involve tokenizing dataset content, filtering data by domain or category, and aggregating token counts. Common triggers include requests to count tokens in HuggingFace datasets, filter datasets by specific fields, or use particular tokenizers (e.g., Qwen, DeepSeek, GPT).
ইনস্টল করার পর, টার্মিনালে নিচের কমান্ড চালিয়ে আপনি এই স্কিল ব্যবহার করতে পারবেন:
skills use count-dataset-tokens