Skip to content

Streaming TextDecoder drops a U+FEFF that follows bytes held over from the previous call #67561

Open

Description

@TomStrepsil

Version

v24.20.0

Platform

Darwin 24.6.0 Darwin Kernel Version 24.6.0: Tue Apr 21 20:18:13 PDT 2026; root:xnu-11417.140.69.710.16~1/RELEASE_ARM64_T8132 arm64

Subsystem

util (TextDecoder)

What steps will reproduce the bug?

const decoder = new TextDecoder();
decoder.decode(new Uint8Array([0xe2, 0x82]), { stream: true });
const text = decoder.decode(new Uint8Array([0xac, 0xef, 0xbb, 0xbf]), { stream: true });
console.log([...text].map((c) => c.codePointAt(0).toString(16)));

The bytes are the UTF-8 for U+20AC (€) followed by U+FEFF, split inside the €.

How often does it reproduce? Is there a required condition?

Always, when all of these hold:

  • the decoder is UTF-8, with ignoreBOM false (the default), and { stream: true } is passed
  • no earlier call has produced any output
  • the call starts with bytes held over from an earlier call, and the bytes after the sequence they start begin with EF BB BF

With ignoreBOM: true, or once an earlier call has produced output, the U+FEFF is kept, as it is when the same bytes are decoded in one call. TextDecoderStream drops it too.

What is the expected behavior? Why is that the expected behavior?

[ '20ac', 'feff' ]

Deno 2.7.10 and Bun 1.4.4 print this.

In the Encoding Standard, only the first scalar value a streaming decoder outputs is checked for a BOM:

  • decode() step 1 resets the decoder and sets BOM seen to false only when do not flush is false. The first call sets do not flush to true, so the second call keeps the decoder's state and BOM seen.
  • The first call outputs nothing: E2 82 leaves the UTF-8 decoder needing one more byte, so serialize I/O queue reads no item and BOM seen stays false.
  • In the second call, AC completes U+20AC and EF BB BF decodes to U+FEFF. Serialize I/O queue reads U+20AC first: ignore BOM and BOM seen are false, so it sets BOM seen to true, and U+20AC isn't U+FEFF, so it is appended. It then reads U+FEFF with BOM seen true, so it appends it too.

What do you see instead?

[ '20ac' ]

Additional information

This appears to come from the streaming UTF-8 fast path added in #61549 (in 25.6.0 and 24.14.0; not in v22). When a call starts with bytes held over from the previous call, lib/internal/encoding.js decodes the sequence they start and the rest separately:

const res = (prefix ? decodeUTF8(prefix, ignoreBom, this[kFatal]) : '') +
            decodeUTF8(u, ignoreBom || prefix, this[kFatal]);

ignoreBom || prefix is meant to stop the BOM check on u once prefix has been decoded. But when ignoreBom is false it evaluates to prefix, a Uint8Array. The binding reads that argument with args[1]->IsTrue(), which is false for an object, so a BOM at the start of u is stripped.

Passing a boolean, such as ignoreBom || prefix !== undefined, may fix it. Since prefix always holds at least the held-over bytes, decoding it outputs at least one scalar value, which sets BOM seen in the spec.

I ran the repro on v24.20.0 only. Both lines above are in the source of every current release line, and the fast path was never backported to v22:

Version ignoreBom || prefix and args[1]->IsTrue() present
v27.0.0-alpha.0.1.1 yes
v26.13.1 yes
v24.22.3 yes
v22.23.5 no
main at 2231574 yes

The open #61559 moves this line into a helper shared with UTF-16 but keeps the same ignoreBom || prefix expression.

Activity

  1. added
    encodingIssues and PRs related to the TextEncoder and TextDecoder APIs.
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    encodingIssues and PRs related to the TextEncoder and TextDecoder APIs.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions