A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Gpt-tokenizer is a pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models. Licensed under MIT, it is popular in the JavaScript ecosystem with 444.8K weekly downloads.
Gpt-tokenizer has an good overall health score of 69/100. Its strongest dimension is security (100/100), while quality (50/100) is the area with most room for improvement. This package is in good shape overall.