Add BPE tokenization with tokenizeBPE and detokenizeBPE - #119877

Related: #108247
Related: #110366

The OpenAI models bill by token and are bounded by a context window, and there was no way to ask ClickHouse how many tokens a text costs. This is the use case @alexey-milovidov singled out in #108247: "For the purpose of this task, I recommend focusing only on the first use-case as a motivation: Estimating the number of tokens to be processed by LLM."

SELECT length(tokenizeBPE(prompt, 'cl100k_base')) AS tokens FROM prompts;
SELECT detokenizeBPE(tokenizeBPE('hello world', 'cl100k_base'), 'cl100k_base');

Vocabularies are declared in the server configuration, in the .tiktoken format, each with the pre-tokenizer it goes with (r50k for GPT-2, r50k_base and p50k_base; cl100k for cl100k_base; o200k for o200k_base):

<bpe_vocabularies>
    <cl100k_base>
        <path>/var/lib/clickhouse/tokenizers/cl100k_base.tiktoken</path>
        <pretokenizer>cl100k</pretokenizer>
    </cl100k_base>
</bpe_vocabularies>

A vocabulary is loaded when a query first names it and shared from then on, and reloaded when the file changes. vocabulary holds the content inline instead of path, which is how the tests and the documentation examples declare one.

Notes on the implementation:

• The pre-tokenizers are the regular expressions of tiktoken written out by hand: they need lookahead, which RE2 does not support, and the o200k one needs backtracking. Character classes come from ICU, so \s is the Unicode White_Space property.
• Merging keeps its candidates in a heap instead of rescanning the piece. A piece is only bounded by the length of the text — a megabyte of one letter is a single piece — and that one tokenizes in 0.4 s rather than not finishing.
• Text is tokenized as text: a vocabulary has no special tokens, so <|endoftext|> in the input is tokenized like any other text. A string that is not valid UTF-8 is tokenized too and round trips byte for byte.

Checked against tiktoken over 26000 texts per vocabulary (52 hand-written cases, 20000 random ones, 4000 chunks of this repository's docs and source, and the whole of CHANGELOG.md as one string): every id matches and so does every decode. 3000 random byte strings per vocabulary round trip exactly. One thread on CHANGELOG.md does 26 / 25 / 17 MB/s for the three vocabularies, against 28 / 29 / 39 MB/s for tiktoken itself.

Not covered here, and left to the issue: arbitrary dictionaries loaded from tables, the text-index tokenizer interface with a digest to keep an index consistent, and the aiEstimateTokenCount wrapper of #110366, whose shape (a function or a parametric aggregate) is still open. If you would rather have vocabularies come from a DICTIONARY than from the configuration, say so and I will move it.

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Added the functions tokenizeBPE and detokenizeBPE, which tokenize text with a byte pair encoding vocabulary in the .tiktoken format and turn the token ids back into text. The number of tokens a text costs an OpenAI model is length(tokenizeBPE(text, 'cl100k_base')). Vocabularies are declared in the bpe_vocabularies section of the server configuration.