feat(html): opt-in HTML backends (nokolexbor + rust) behind env switch by gildesmarais · Pull Request #501 · html2rss/html2rss

gildesmarais · 2026-09-02T19:33:30Z

What changed

Introduces Html::Document / Html::Node facade and Html::Backend registry (HTML2RSS_HTML_BACKEND=nokogiri|nokolexbor|rust, default nokogiri).
Adds Lexbor adapter (Backend::Nokolexbor) as a measured experiment.
Adds optional Rust extension ext/html2rss_parser (rb-sys + magnus + scraper): NativeEngine, direct SST construction, thin CSS/attr DOM ducks — compile via rake compile (no gemspec.extensions).
Wires make perf-baseline / make perf-suite for all three backends; records baselines under spec/perf/.
Experiment memo: lib/html2rss/html/BACKEND_EXPERIMENT.md.

Why

Nokogiri dominates parse/normalize cost on the auto-source path. The goal was a second engine that can beat it on wall time and allocations without breaking admission fidelity. Nokolexbor and Rust both land as opt-in so we can measure; neither is promoted to default.

Benchmarks

Auto-source (`make perf-baseline`, Ruby 4.0.0, median of 5)

Backend	wall_time_s	allocations	wall Δ%	alloc Δ%
nokogiri	0.189	724272	—	—
nokolexbor	0.162	687591	−13.8%	−5.1%
rust	0.188	702936	−0.3%	−2.9%

Suite wall (`make perf-suite`, seed 42, 1887 examples)

Backend	wall_time_s	wall Δ%
nokogiri	12.99	—
nokolexbor	12.45	−4.1%
rust	12.92	−0.5%

Source: spec/perf/baseline-auto-source.md, spec/perf/baseline-suite-wall.md.

Insights

Nokolexbor — no-go for default. Modest wall win, but page_1.html admission drifts (Nokogiri 65/61/13 vs Lexbor 69/71/17). FFI node wrappers reintroduce GC/dispatch cost.
Rust — experiment OK; not ready to default. Suite green after rake compile (1 pending: XPath-only pager). Same page_1 drift as Lexbor. Auto-source ~parity with nokogiri (~0% wall / ~−3% alloc) — thin DOM path still pays Ruby wrapper cost; SST short-circuit exists but does not yet deliver the promotion bar (>30% wall / >50% alloc).
Next leverage: keep SST-first / native batch extraction and close admission parity with nokogiri — not more Nokogiri-shaped node ducks. Do not add gemspec.extensions until packaging (precompiled gems) is ready.
Assure (Conditional): OK as documented opt-in; No as production default. P1s called out in review: Document→SST serialize+reparse on some paths, silent admission drift, dual Normalizer rules, XPath gap, method_missing native leak.

Risk

Opt-in only; default remains nokogiri — low production risk if env unset.
:rust requires local rake compile + Rust toolchain; LoadError if missing.
Fidelity drift under non-nokogiri backends can change feed item sets if someone opts in blindly.
Dual Ruby/Rust SST normalize rules can drift without the golden/constants sync specs.

Review map

lib/html2rss/html/backend.rb + backend/{nokogiri,nokolexbor,rust}.rb — env registry and adapter contracts
ext/html2rss_parser/ — pure parse/sst vs magnus ruby/ boundary; optional compile wiring in rakefile.rb
lib/html2rss/sst/normalizer.rb + spec/.../native_engine_sst_spec.rb — SST short-circuit and golden fidelity
lib/html2rss/html/BACKEND_EXPERIMENT.md + spec/perf/baseline-*.md — verdicts and numbers
Suite adapters (xpath → named helpers, sitemap stays Nokogiri)

Validation

HTML2RSS_HTML_BACKEND=rust full RSpec (seed 42): 1887 examples, 0 failures, 1 pending
bin/heuristic-perf-baseline --backend all → spec/perf/baseline-auto-source.md
bin/perf-suite --write … → spec/perf/baseline-suite-wall.md
Assure: Conditional (opt-in OK; not default)

Test plan

mise exec -- bundle exec rake compile
mise exec -- bundle exec rspec --no-fail-fast
HTML2RSS_HTML_BACKEND=rust mise exec -- bundle exec rspec --no-fail-fast
make perf-baseline and confirm page_1 counts vs nokogiri
Confirm default (unset env) still uses nokogiri

Hide HTML parse construction behind Html::Backend so Response and domain callers no longer construct or type-check raw Nokogiri HTML documents. Sanitize and XML/XPath paths stay on Nokogiri.

Wire Lexbor via nokolexbor (dev dependency), CSS compatibility shims for :first/:not(:first-child), and XPath pager fallback when Lexbor rejects selectors. Full suite green under both backends.

Extend dual-backend perf harness (heuristic + suite wall-clock), capture numbers, and document no-go for production default while facade stays.

Add rb-sys/magnus workspace under ext/html2rss_parser with rake compile only (no gemspec.extensions) so nokogiri default installs stay zero-Rust.

Parse once with scraper, normalize in Rust mirroring Normalizer rules, and map IR to SST via Magnus so the :rust path can skip the Ruby walk.

Expose NativeEngine Document/Node CSS and attr ducks, register Backend::Rust behind HTML2RSS_HTML_BACKEND=rust, and keep sitemap/XML on Nokogiri.

Add NodeSet/#text/#attr ducks, named descendant text helpers, sitemap Nokogiri policy, and NativeNode identity so Microdata and selectors work.

Wire rust into perf harnesses, refresh suite/auto-source numbers, and update the backend experiment memo with promotion gates still unmet.

Document rake compile and sandbox BUNDLE_PATH caveat for the opt-in Rust HTML backend experiment.

Add criterion microbenches for string normalize vs today's serialize+reparse Document path, feature-gate magnus so benches link without libruby, and snapshot Phase 0 before numbers.

Walk scraper Html in normalize_from_html so Document#to_sst skips serialize+reparse; keep string to_sst as parse+reuse; assert Document≡string.

Move Attrs/Node/Index/Document construction into SST::Hydrator and have Rust emit one nested Hash IR, deleting per-node Magnus eval/build.

Detach comments for real, document Path A short-circuit, fix perf-suite empty-array under set -u, and refresh baselines plus BACKEND_EXPERIMENT honesty on the ≥15% wall checkpoint miss.

Path A materializes Attrs/Node/Index/Document via cached Magnus classes, dropping the O(nodes) Ruby Hash nest that dominated wall and allocations vs nokogiri on page_1.

Path A SST now beats nokogiri on page_1 wall and alloc; auto-source alloc flips to a clear drop while fidelity counts remain 69/71/17.

Record page_1 SST wall/alloc ≤ nokogiri and auto-source alloc drop after killing nested Hash IR; fidelity gate still open.

Mend html5ever adoption-agency splits into libxml-like nested li/a chains and blackhole following siblings into the still-open parent anchor so page_1 semantic/html/auto_source counts match 65/61/13.

Both parity gates are met: Path A SST wall/alloc ≤ nokogiri on page_1 and admission counts 65/61/13; suite green under HTML2RSS_HTML_BACKEND=rust.

CI characterization so mend_lists regressions fail the suite instead of only drifting remasure docs.

gildesmarais added 20 commits September 2, 2026 16:57

feat(html): introduce Document/Node facade over Nokogiri backend

09cfebd

Hide HTML parse construction behind Html::Backend so Response and domain callers no longer construct or type-check raw Nokogiri HTML documents. Sanitize and XML/XPath paths stay on Nokogiri.

feat(html): add nokolexbor backend behind HTML2RSS_HTML_BACKEND

7069fcc

Wire Lexbor via nokolexbor (dev dependency), CSS compatibility shims for :first/:not(:first-child), and XPath pager fallback when Lexbor rejects selectors. Full suite green under both backends.

docs(html): record parser backend experiment go/no-go memo

02042ef

Extend dual-backend perf harness (heuristic + suite wall-clock), capture numbers, and document no-go for production default while facade stays.

docs(html): point Html module at backend experiment memo

c5531d7

feat(html): scaffold optional rust html2rss_parser extension

c6cc632

Add rb-sys/magnus workspace under ext/html2rss_parser with rake compile only (no gemspec.extensions) so nokogiri default installs stay zero-Rust.

feat(html): build SST::Document directly from rust parser

0ad2965

Parse once with scraper, normalize in Rust mirroring Normalizer rules, and map IR to SST via Magnus so the :rust path can skip the Ruby walk.

feat(html): register thin rust HTML backend

40d982d

Expose NativeEngine Document/Node CSS and attr ducks, register Backend::Rust behind HTML2RSS_HTML_BACKEND=rust, and keep sitemap/XML on Nokogiri.

fix(html): make rust backend pass full suite

e0d1175

Add NodeSet/#text/#attr ducks, named descendant text helpers, sitemap Nokogiri policy, and NativeNode identity so Microdata and selectors work.

chore(perf): record rust html backend baselines

79c40be

Wire rust into perf harnesses, refresh suite/auto-source numbers, and update the backend experiment memo with promotion gates still unmet.

docs: note optional rust html backend compile

c462fab

Document rake compile and sandbox BUNDLE_PATH caveat for the opt-in Rust HTML backend experiment.

chore(perf): bench rust SST normalize paths

1a52b9d

Add criterion microbenches for string normalize vs today's serialize+reparse Document path, feature-gate magnus so benches link without libruby, and snapshot Phase 0 before numbers.

perf(html): SST normalize from existing rust Html tree

bdf70c6

Walk scraper Html in normalize_from_html so Document#to_sst skips serialize+reparse; keep string to_sst as parse+reuse; assert Document≡string.

perf(html): hydrate SST::Document from one native IR payload

043cc78

Move Attrs/Node/Index/Document construction into SST::Hydrator and have Rust emit one nested Hash IR, deleting per-node Magnus eval/build.

chore(perf): remasure rust after one-parse hydrate

41a3c87

Detach comments for real, document Path A short-circuit, fix perf-suite empty-array under set -u, and refresh baselines plus BACKEND_EXPERIMENT honesty on the ≥15% wall checkpoint miss.

perf(html): build SST from rust IR without nested Hash

a42639b

Path A materializes Attrs/Node/Index/Document via cached Magnus classes, dropping the O(nodes) Ruby Hash nest that dominated wall and allocations vs nokogiri on page_1.

chore(perf): remasure rust after direct SST build

b511307

Path A SST now beats nokogiri on page_1 wall and alloc; auto-source alloc flips to a clear drop while fidelity counts remain 69/71/17.

docs(html): note direct Magnus Path A remasure

4272dff

Record page_1 SST wall/alloc ≤ nokogiri and auto-source alloc drop after killing nested Hash IR; fidelity gate still open.

fix(html): align rust SST admission with nokogiri on page_1

9326356

Mend html5ever adoption-agency splits into libxml-like nested li/a chains and blackhole following siblings into the still-open parent anchor so page_1 semantic/html/auto_source counts match 65/61/13.

docs(html): record rust nokogiri parity status

afa2d0d

Both parity gates are met: Path A SST wall/alloc ≤ nokogiri on page_1 and admission counts 65/61/13; suite green under HTML2RSS_HTML_BACKEND=rust.

test(html): lock rust page_1 admission to nokogiri 65/61/13

7446919

CI characterization so mend_lists regressions fail the suite instead of only drifting remasure docs.

Navigation Menu

Uh oh!

Uh oh!

feat(html): opt-in HTML backends (nokolexbor + rust) behind env switch - #501

feat(html): opt-in HTML backends (nokolexbor + rust) behind env switch#501
gildesmarais wants to merge 20 commits into
masterfrom
experiment/html-backend-nokolexbor

gildesmarais commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

gildesmarais commented Sep 2, 2026

What changed

Why

Benchmarks

Auto-source (make perf-baseline, Ruby 4.0.0, median of 5)

Suite wall (make perf-suite, seed 42, 1887 examples)

Insights

Risk

Review map

Validation

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Auto-source (`make perf-baseline`, Ruby 4.0.0, median of 5)

Suite wall (`make perf-suite`, seed 42, 1887 examples)