Commits

Branch selector

 main

User selector

All users

All time

Commit History

Commits on Apr 5, 2024

• Set value of TOKENIZERS_CPP_CARGO_TARGET based on ANDROID_ABI (#28)

  yuxuanchiadm

  authored
  07376b9

  Copy full SHA for 07376b9

Commits on Apr 4, 2024

• [HFTokenizer] Allow use of "add_special_tokens" for HFTokenizer.Encode and "skip_special_tokens" for HFTokenizer.Decode methods. (#27)

  Abhishek8394

  authored
  35bad0e

  Copy full SHA for 35bad0e

Commits on Mar 30, 2024

• [Web] Expose getVocabSize and idToToken to web, bump version to 0.1.3 (#25)

  CharlieFRuan

  authored
  7466de5

  Copy full SHA for 7466de5

Commits on Dec 19, 2023

• Add support for querying vocabulary from tokenizer (#22)

  Ubospica

  authored
  27dbe17

  Copy full SHA for 27dbe17

Commits on Nov 18, 2023

• Bump version to 0.1.2

  CharlieFRuan

  committed
  467d5cd

  Copy full SHA for 467d5cd

• Update build.sh to make web tokenizer work with chrome extension (#20)

  Show description for 08316ea
  manuongithub

  authored
  08316ea

  Copy full SHA for 08316ea

Commits on Nov 16, 2023

• fix rwkv world tokenzier (#19)

  Show description for 4456fa8
  BBuf

  authored
  4456fa8

  Copy full SHA for 4456fa8

Commits on Oct 28, 2023

• Update to reduce memory realloc (#18)

  tqchen

  authored
  22b8932

  Copy full SHA for 22b8932

Commits on Sep 6, 2023

• [FIX] Remove boost dependency from submodule msgpack (#16)

  

  Siyuan Feng

  authored
  e47442f

  Copy full SHA for e47442f

• msgpack as 3rdparty library instead of fetch_context (#15)

  Show description for 1eae242
  BBuf

  authored
  1eae242

  Copy full SHA for 1eae242

Commits on Sep 1, 2023

• add rwkv world tokenizer (#14)

  Show description for eec72a6
  BBuf

  authored
  eec72a6

  Copy full SHA for eec72a6

Commits on Aug 15, 2023

• Update rust to support latest tokenizers (#11)

  tqchen

  authored
  4f42c9f

  Copy full SHA for 4f42c9f

Commits on Jul 16, 2023

• Update CMakeLists.txt (#6)

  luiyen

  authored
  6e3a37e

  Copy full SHA for 6e3a37e

Commits on Jun 3, 2023

• Fix windows build by linking against ntdll (#5)

  junrushao

  authored
  c53bc04

  Copy full SHA for c53bc04

Commits on May 26, 2023

• Fix readme typo (#4)

  erjanmx

  authored
  8465f9b

  Copy full SHA for 8465f9b

Commits on May 24, 2023

• Update to parcel2

  tqchen

  committed
  a9f9359

  Copy full SHA for a9f9359

Commits on May 22, 2023

• Update CMakeLists.txt (#2)

  Show description for bd24b8a
  songkq

  authored
  bd24b8a

  Copy full SHA for bd24b8a

• Merge pull request #3 from mlc-ai/npm

  Show description for 19d3a00
  tqchen

  authored
  19d3a00

  Copy full SHA for 19d3a00

• Add npm package

  tqchen

  committed
  75353f5

  Copy full SHA for 75353f5

Commits on May 20, 2023

• Add examples

  tqchen

  committed
  5703f8d

  Copy full SHA for 5703f8d

Commits on May 19, 2023

• Update CMakeLists.txt

  tqchen

  authored
  470bbd4

  Copy full SHA for 470bbd4

Commits on May 17, 2023

• Initial checkin of tokenizer cpp

  Show description for a3fd734
  

  6 peoplecommitted
  a3fd734

  Copy full SHA for a3fd734

Commits on May 16, 2023

• Initial commit

  tqchen

  authored
  1294dd4

  Copy full SHA for 1294dd4