Skip to content

lib,src: reset TextDecoder after a decoding error - #67610

Draft
vedchaudhari wants to merge 1 commit into
nodejs:mainfrom
vedchaudhari:textdecoder-reset-on-error
Draft

vedchaudhari wants to merge 1 commit into
nodejs:mainfrom
vedchaudhari:textdecoder-reset-on-error

Conversation

@vedchaudhari

Copy link
Copy Markdown
Contributor

Draft until whatwg/encoding#369 lands.

The spec is changing so that when TextDecoder#decode() throws in fatal mode, the decoder resets and the next call starts fresh. Node currently keeps the old state, so if you keep streaming after an error:

  • a BOM at the start of the next chunk shows up as U+FEFF
  • UTF-16 keeps throwing on valid input, because half a surrogate pair is still held
  • iso-2022-jp decodes AB as U+758E, and gb18030 leaks a leftover byte

This resets the decoder when an error happens, for UTF-8 (in JS) and the ICU encodings (in C++), and nothing changes when fatal is off. I added tests for it and also checked it against the new WPT tests from the WebKit PR, which all pass now.

cc @ChALkeR since you wrote the UTF-8 fast path and opened whatwg/encoding#359.

Fixes: #67604
Refs: whatwg/encoding#369

Assisted-by: an AI coding agent. Reviewed and tested locally by me.

A fatal decode() error now resets the decoder, as the Encoding
Standard requires, so the next call does not continue with the bytes
or BOM state from before the error.

Signed-off-by: vedchaudhari <vedc2853@gmail.com>
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. encoding Issues and PRs related to the TextEncoder and TextDecoder APIs. i18n-api Issues and PRs related to Node.js internationalization support. needs-ci PRs that need a full CI run. labels Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. encoding Issues and PRs related to the TextEncoder and TextDecoder APIs. i18n-api Issues and PRs related to Node.js internationalization support. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Small TextDecoder decode() change

2 participants