openai/GPT-3-Encoder

openai/GPT-3-Encoder

Description: Javascript BPE Encoder Decoder for GPT-2 / GPT-3

Language: JavaScript

License: MIT

Stars: 134

Forks: 94

Open issues: 0

Created: 2020-11-23T17:05:55Z

Pushed: 2026-03-25T19:39:31Z

Default branch: master

Fork: yes

Parent repository: latitudegames/GPT-3-Encoder

Archived: yes

README: GPT-3-Encoder Javascript BPE Encoder Decoder for GPT-2 / GPT-3

About

GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a javascript implementation of OpenAI's original python encoder/decoder which can be found here

Install with npm

npm install gpt-3-encoder

Usage

Compatible with Node >= 12

const {encode, decode} = require('gpt-3-encoder')

const str = 'This is an example sentence to try encoding out on!'
const encoded = encode(str)
console.log('Encoded this string looks like: ', encoded)

console.log('We can look at each token and what it represents')
for(let token of encoded){
console.log({token, string: decode([token])})
}

const decoded = decode(encoded)
console.log('We can decode it back into:\n', decoded)