Skip to content

Preserve UTF-8 characters across head read buffers - #1286

Open
FanWu-ai wants to merge 1 commit into
shelljs:mainfrom
FanWu-ai:fix-head-utf8-boundaries-oct13
Open

FanWu-ai wants to merge 1 commit into
shelljs:mainfrom
FanWu-ai:fix-head-utf8-boundaries-oct13

Conversation

@FanWu-ai

Copy link
Copy Markdown

Problem

head() decodes each file-read buffer as a complete UTF-8 string. When a multibyte character crosses the default 64 KiB boundary, it is replaced with invalid-character markers even though the source file contains valid UTF-8. For example, a first line containing 65,535 ASCII characters followed by é is returned with �� at the end.

Change

Use Node's built-in StringDecoder for each file's incremental reads and flush it when reading finishes. This preserves characters split across buffers and retains replacement behavior for incomplete UTF-8 at EOF. The decoder is local to each file, and reading still stops once enough lines have been read. No dependency or API changes.

Verification

Node.js 24.19.0 on Linux x86_64:

  • Pristine full suite: 640 passed, 6 pre-existing explicit skips; ESLint passed.
  • Added 16 tests covering every split point of two-, three- and four-byte characters, with and without a trailing newline, the default buffer size, one-byte buffers, incomplete EOF and separate-file decoder state. Fourteen fail against the unchanged source; the two controls already pass.
  • Final focused head suite: 29 passed.
  • Final npm test, including posttest ESLint: 656 passed, 6 pre-existing explicit skips.
  • Final npm run test-with-coverage: 656 passed, 6 skipped; 95.63% statement and 96.45% line coverage overall; src/head.js has 100% statement, branch, function and line coverage.
  • Node-support consistency, generated documentation and git diff --check passed. Regenerating documentation leaves README unchanged.
  • An additional 168 cases agree with native GNU head, covering eight buffer sizes, UTF-8, combining marks, CRLF, invalid bytes, EOF and line counts. An early-stop control verifies only one buffer is read when the requested first line is already available.

The six reported skips are existing test/cmd.js cases: five asynchronous-command cases and the missing-execPath case. Other operating systems and Node versions have not been run locally.

AI assistance: OpenAI Codex assisted the investigation, implementation and tests.

@FanWu-ai
FanWu-ai marked this pull request as ready for review October 12, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant