Skill

sentencepiece

Text tokenization for LLMs including HuggingFace Tokenizers and SentencePiece. Use when training custom tokenizers or handling multilingual text.

Installation

Add the marketplace

/plugin marketplace add zechenzhangAGI/AI-research-SKILLs

Install plugins

/plugin

Run these commands in Claude Code to add this plugin to your environment. The marketplace must be added before you can install its plugins.

From Plugin

tokenization

View Plugin

From Marketplace

ai-research-skills

View Marketplace

Author

@zechenzhangAGI

View GitHub Profile