/* This is exactly the same test as the non-unicode cases. However, there are reasons why this
   * should be made a special case and may deviate in the future:
   * 
   * As of now there appears to be no final word as to how characters that consist of more than one 
   * numeric token should correspond to textual tokens, and results vary according to different 
   * models' tokenizers. For example, given a character 'Z' that corresponds to the numeric tokens {1,2,3} 
   * some (llama-2) will pad the length of the total number of tokens by returning spaces as tokens 
   * (i.e. ' ', ' ', 'Z') while others (GPT4Tokenizer) will pad with the character itself (i.e. 'Z','Z','Z').
   * 
   * This is very evident when tokenizing ideograms and emojis, but can arise with various unicode characters 
   * as well. See pull request for more relevant discussion https://github.com/SciSharp/LLamaSharp/pull/862
   *
   * Currently the method will remain consistent with the output of ITextTokenizer.CountTokens, meaning
   * any redundant tokens will not be ommited as long as they are counted by CountTokens.
   * 
   * StreamingTokenDecoder, while sufficiently useful for this task, was not designed with producing
   * output for one numeric token at a time in mind, so ITextTokenizer.GetTokens should not be considered 
   * an example of proper use.
   * 
   * Note: if this message is removed, also remove references to it in LLamaSharpTextEmbeddingGenerator.GetTokens
   * and LLamaSharpTextGenerator.GetTokens
   */