Skip to content

Commit 6a65ec4

Browse files
chriszarateclaude
andauthored
Migrate DE-RTC as the third sync engine (#9)
* Vendor automerge-php at includes/lib with lazy loader Imported verbatim from the Gutenberg chriszarate/refreshed-de-rtc branch (lib/compat/wordpress-7.1/distributed-editing/automerge-php, itself a verbatim port of wordpress-develop add/distributed-editing, PR WordPress/wordpress-develop#12334). MIT, PHP 8.2+, namespace WordPress\DistributedEditing\Automerge, no WordPress dependency. The library's conformance suite (php tests/run.php) referenced an upstream/automerge submodule never committed on either source branch; the 11 fixture files it reads (3 storage fixtures + 8 fuzz-crashers) are now vendored under upstream/automerge/, fetched from automerge/automerge@8d7b12f (MIT; recorded in VENDORED_FROM_COMMIT.txt). Suite passes 680/680 locally. includes/lib/automerge-php-loader.php mirrors y-php-loader.php: a lazy PSR-4 shim plus gutenberg_sync_engines_automerge_php_is_supported() (PHP >= 8.2 + mbstring) for callers to gate on. Excluded from phpcs like y-php. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Port the DE-RTC merge core as a frozen engine slice includes/engines/de-rtc/merge-core.php is the exact call-graph closure (113 functions, ~6.4k lines) of the engine-facing entry points in the Gutenberg chriszarate/refreshed-de-rtc branch's de-rtc.php (itself a verbatim port of wordpress-develop add/distributed-editing): update construction/normalization, the serialized-block and block-identity three-way merges, the rich-text merge model, version snapshots, sync-meta format/parse, canonicalization/hashing, and the automerge runtime gate. Extracted token-exactly; the only deltas are the vendored library path and delegating autoloading to automerge-php-loader.php (marked DELTA in place). Excluded from phpcs and frozen like the intent-log cross-language core. The plugin loads it behind a function_exists guard so a Core/Gutenberg build that ships DE-RTC itself wins. tests/phpunit/wpDeRtcMergeCore.php drives the merge seam directly: disjoint block edits merge, non-overlapping rich-text edits within one paragraph merge, overlapping edits escalate (de_rtc_rebase_failed / manual_conflict_required), concurrent differing appends escalate by design, tampered/mismatched updates are rejected, engine-unaware saves get server-generated updates, and version snapshots trim at the limit. Two upstream-faithful behaviors are pinned rather than fixed: the legacy native-automerge-php-v1 lane is dead code (NativePort::merge accepts only document pairs, so operations-array updates are rejected as unverifiable) and same-edge concurrent appends require review. Full PHPUnit: 162 tests / 707 assertions green (baseline 148 + 14 new). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add WP_De_RTC_Engine: the DE-RTC merge core behind the engine SPI The third engine (slug de-rtc): server-governed three-way merges of whole-content proposals. Clients send proposal updates ({proposalId, baseVersion, proposedContent, clientUpdate}); under the per-room ingest lock (three-way merges are order-dependent, so ingest serializes like intent-log, unlike yjs-server's lock-free CRDT path) the engine resolves the named base from its version snapshots, merges with wp_de_rtc_get_automerge_retry_save_result(), and either advances the canonical document (disposition applied + a server-authored content row peers replay) or settles the proposal as escalated (manual-conflict-required, requires-unfiltered-html) or voided (unknown-base-version, invalid-payload, tamper/malformed details). Canonical state ({version, content, sync_meta.version_snapshots}) lives in room meta — not in a post_content sync-meta block, which is what made most of upstream DE-RTC's periphery (revision scan, kses allowance, preservation filters) unnecessary here. Genesis is deterministic from post content and ADOPTS an existing upstream sync-meta block (version lineage continues, content stripped). Server-driven checkpoints reuse the shared retention invariant (trim behind the previous checkpoint, floor-clamp stale cursors). The capability lane escalates whole proposals kses would rewrite (upstream sequesters just the risky blocks for review — documented gap until the review lane exists). Registered in the plugin and on the settings screen. PHPUnit: 174 tests / 763 assertions green (12 new engine tests: SPI identity, genesis+lineage, apply/broadcast, stale-base three-way rebase, conflict escalation, kses escalation, per-row voids, type fencing, persistence across instances, checkpoint trim + floor clamp). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add the de-rtc client engine: proposal wire over a local doc bridge The client half of DE-RTC keeps Distributed Editing's core inversion: the client NEVER merges. A local Y.Doc (shared engines/yjs schema — same undo manager, awareness anchor, and record mapping as the yjs engines) bridges the editor; the session codec speaks the proposal wire: - One coalesced in-flight proposal: local edits mark the doc dirty and a whole-content proposal is built only when none is pending, so a typing burst costs one proposal per poll cycle. The base version is the version last APPLIED to the doc — stale bases are exactly what the server's three-way merge is for. - Canonical rows are deferred while local edits are dirty/in flight (applying them would overwrite edits the server has not seen); the newest deferred row applies when the local state settles. Escalated proposals are abandoned locally once canonical applies — the upstream review/decision UI is a listed follow-up, as is the client-side block-native descriptor builder (proposals send clientUpdate: null and the server derives operations via its engine-unaware writer lane). - Server genesis bootstraps the doc (hydrate is a no-op; pre-bootstrap local changes buffer and replay), mirroring yjs-server. - Collections are inert (the intent-log precedent): awareness flows, rows are ignored, nothing is proposed. The doc bridge serializes/parses through the editor's own block library (wp.blocks at runtime; resolved from the subtree in Jest/tsc, mocked in tests). Registered as the third adapter in src/index.ts. Jest: 347 tests green (8 new: identity, no-seed hydrate + genesis bootstrap, pre-bootstrap buffering, proposal coalescing + settle, remote apply + onRemoteChange + rebased base version, deferred canonical under in-flight, own-row slot clearing, inert collections). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix de-rtc client rebase races; add the de-rtc collaboration e2e spec Three client fixes found by driving the real editor two-up (each verified with the spec + Playwright traces): 1. Own accepted rows advance the base version. A client's successive typing chunks were proposed against the same old base, so an author's continuation conflicted with their own just-accepted chunk (rich-text overlap at the same position). When our own row round-trips unchanged, advance the version WITHOUT touching the doc (newer keystrokes survive); the next chunk proposes against it. 2. Block-level incorporation of own merged rows. When the server merges peers' work into our proposal, adopt canonical blocks we have not edited since proposing, keep locally-edited blocks (reconciled by the next proposal), handle peer tail-insertions, and bail to deferral on any other structure change. Truly concurrent same-block edits resolve block-level last-writer-wins — the same class of silent register policy yjs-server carries, at coarser grain (documented in the engine comparison). 3. Dispositions only settle the proposal they name. A previous proposal's disposition arrives in the response AFTER the one whose rows already settled it; clearing the in-flight slot unconditionally let a peer's canonical row clobber unproposed local edits, which the next incorporation then wrongly pinned as 'local'. The e2e spec (three scenarios: server-side genesis + bidirectional text sync, concurrent different-block edits surviving the three-way merge, and the save flow persisting both users' settled edits) passes repeatedly in-suite and solo. It opens sessions through a local helper that force-closes the second user's welcome guide when the fixture's preference dispatch loses to late preference hydration (an open modal aria-hides the page and fails discovery even though sync is healthy). Deliberately absent scenarios are the v1 gaps: title sync, and the empty-post concurrent-append case (a BY-DESIGN escalation under DE-RTC policy). Known pre-existing full-suite flake, verified NOT caused by these changes: http-only/collaboration-sync-body-size can fail after any engine-flip suite under load (control: yjs suite + body-size fails identically without de-rtc involved); solo it is green, CI retries absorb it. Jest 347 green; de-rtc e2e 3/3; full e2e 31/32 (the flake above). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Document the de-rtc engine and run automerge-php conformance in CI - AGENTS.md: engines list, repo-layout entries for the frozen merge core and the vendored automerge-php (incl. the PORTING_STATUS.json rewrite side-effect and the fact that the shipping merge path never calls the library), new green baselines (Jest 347, PHPUnit 174/763, e2e 32/32 with the pre-existing body-size-after-engine-flip flake called out), and the de-rtc known-gaps entry. - docs/engine-comparison.md: de-rtc added to the one-sentence list, the parity and host-resource tables, known gaps (no benchmark profile, no review UI/title sync, block-level LWW on same-block concurrency, no client-side proof descriptors), and the bottom line. - README.md: de-rtc in the engines list. - CI: run the vendored automerge-php conformance suite (680 mapped upstream tests, plain php runner, no WordPress) alongside y-php's. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix conflict syntax error * Actually fix conflict * Pin PHP 8.4 for the conformance suites; surface the automerge report The automerge-php conformance step failed on CI while passing locally: its grapheme-cluster paths split text with PCRE2's \X, and PCRE2 10.42 (bundled by PHP <= 8.3 builds, including the stock runner's 8.3.6 and Ubuntu 24.04 packages) carries a GB11 bug where two ADJACENT emoji-ZWJ sequences count as ONE extended grapheme cluster. PCRE2 10.44 (bundled from PHP 8.4) counts two. Exactly 2 of the 680 mapped upstream tests fail under the old tables — grapheme cursor tracking through an inserted family emoji, and a UTF-16-boundary splice — reproduced in php:8.2/8.3-cli containers (678/680) vs php:8.4-cli (680/680). CI now pins PHP 8.4 via setup-php for the conformance suites (y-php is 7.4+-compatible and unaffected), and the automerge step keeps its JSON report visible — a pass prints the summary line, a failure dumps the failing entries — instead of discarding stdout, which had swallowed the report entirely (the step 'failed silently' in 1.6 s). AGENTS.md documents the PCRE2 >= 10.43 parity requirement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 021e37f commit 6a65ec4

59 files changed

Lines changed: 52974 additions & 59 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,17 +81,53 @@ jobs:
8181
npm ci --ignore-scripts
8282
npm run build
8383
84+
# The conformance suites run on the runner's PHP directly (no
85+
# WordPress). automerge-php's grapheme-cluster paths need the fixed
86+
# GB11 rules of PCRE2 >= 10.43, which PHP bundles from 8.4 — under
87+
# PHP <= 8.3 (PCRE2 10.42, including the stock runner PHP) two
88+
# adjacent emoji-ZWJ sequences count as ONE \X cluster and exactly
89+
# 2 of its 680 tests fail. Pin 8.4.
90+
- name: Set up PHP for the conformance suites
91+
uses: shivammathur/setup-php@v2
92+
with:
93+
php-version: '8.4'
94+
extensions: mbstring
95+
tools: composer
96+
8497
- name: Install PHP dependencies
8598
run: composer install --no-progress
8699

87100
# The vendored y-php library's own conformance suite: translated
88101
# upstream Yjs tests plus byte-level fixtures generated from the JS
89-
# implementation. Runs on the runner's PHP directly (no WordPress).
102+
# implementation.
90103
- name: y-php conformance
91104
run: |
92105
composer --working-dir=includes/lib/y-php install --no-progress
93106
composer --working-dir=includes/lib/y-php test
94107
108+
# The vendored automerge-php library's own conformance suite:
109+
# ported upstream Automerge JS/Rust tests (680 mapped). Needs no
110+
# Composer install (hand-rolled runner). The runner exits non-zero
111+
# on any failing test; keep its report visible either way.
112+
- name: automerge-php conformance
113+
run: |
114+
status=0
115+
php includes/lib/automerge-php/tests/run.php > automerge-conformance.json || status=$?
116+
php -r '
117+
$report = json_decode( (string) file_get_contents( "automerge-conformance.json" ), true );
118+
if ( ! is_array( $report ) ) {
119+
echo "Runner produced no JSON report:\n";
120+
echo file_get_contents( "automerge-conformance.json" ), "\n";
121+
exit( 1 );
122+
}
123+
printf( "passing=%d failing=%d (%s%%)\n", $report["passingTests"] ?? -1, $report["failingTests"] ?? -1, $report["passPercent"] ?? "?" );
124+
'
125+
if [ "$status" -ne 0 ]; then
126+
echo "=== failing tests ==="
127+
grep -B 8 '"passed": false' automerge-conformance.json || true
128+
fi
129+
exit "$status"
130+
95131
# Tests target the dedicated tests environment (.wp-env.tests.json)
96132
# — the default .wp-env.json is the dev environment, whose afterStart
97133
# hook (the websocket daemon) has no business in CI.

‎AGENTS.md‎

Lines changed: 74 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,17 @@ the editor falls back to the classic exclusive post lock.
1818
This plugin provides:
1919

2020
- **Engines:** `intent-log` (server-authoritative log of typed intents; merges
21-
by transform, sets genuine conflicts aside for review) and `yjs-server`
21+
by transform, sets genuine conflicts aside for review), `yjs-server`
2222
(server-authoritative CRDT: the vendored y-php library merges every update
2323
into a canonical room document server-side, compacts by itself, and
2424
materializes post content — lock-free ingest; it inherited the retired
25-
naive-relay yjs-relay engine's client CRDT machinery and wire format).
25+
naive-relay yjs-relay engine's client CRDT machinery and wire format), and
26+
`de-rtc` (Distributed Editing's save-centric model on the room protocol:
27+
clients propose whole content against a named base version; the server
28+
three-way-merges every proposal with the merge core ported verbatim from
29+
the wordpress-develop `add/distributed-editing` branch and broadcasts
30+
canonical content rows; genuine conflicts escalate instead of silently
31+
merging).
2632
The framework's conventional default engine
2733
(`WP_Sync_Engine_Registry::DEFAULT_ENGINE`) is **intent-log** — that's
2834
what runs when the `wp_sync_engine` option is unset. Registration order
@@ -42,9 +48,21 @@ The framework/plugin split is complete: the framework ships **neither** engines
4248
## Repo layout
4349

4450
- `gutenberg-sync-engines.php` — plugin entry.
45-
- `includes/` — server PHP: `engines/{intent-log,yjs-server}/`,
51+
- `includes/` — server PHP: `engines/{intent-log,yjs-server,de-rtc}/`,
4652
`transports/{...,websocket/}`, `admin/` (the Collaboration settings screen),
4753
and `lib/`:
54+
- `engines/de-rtc/merge-core.php` — the DE-RTC merge core, ported
55+
VERBATIM from the Gutenberg `chriszarate/refreshed-de-rtc` branch's
56+
`de-rtc.php` (itself a verbatim port of wordpress-develop
57+
`add/distributed-editing`): the exact call-graph closure (113
58+
functions) of the engine-facing entry points — serialized-block +
59+
block-identity three-way merges, the rich-text merge model, update
60+
construction/validation, version snapshots, sync-meta parse/format,
61+
canonicalization/hashing. Frozen like the intent-log core and excluded
62+
from phpcs; the only deltas are the vendored-library path and loader
63+
delegation (marked `DELTA` in place). Loaded behind a
64+
`function_exists( 'wp_de_rtc_get_reason_codes' )` guard so a
65+
Core/Gutenberg build that ships DE-RTC itself wins.
4866
- `lib/y-php/` — **vendored y-php** (PHP port of Yjs 13.6.31), imported
4967
verbatim from <https://github.com/alecgeatches/y-php> (MIT; upstream
5068
commit recorded in the import commit). Excluded from our phpcs (it
@@ -61,6 +79,33 @@ The framework/plugin split is complete: the framework ships **neither** engines
6179
- `lib/y-php-loader.php` — lazy runtime loader (PSR-4 autoloader +
6280
Composer-`files` equivalents) so the plugin can use y-php without a
6381
Composer autoloader.
82+
- `lib/automerge-php/` — **vendored automerge-php** (native PHP port of
83+
Automerge for DE-RTC research), imported verbatim from the Gutenberg
84+
`chriszarate/refreshed-de-rtc` branch (originally wordpress-develop
85+
`add/distributed-editing`, PR WordPress/wordpress-develop#12334; MIT,
86+
PHP 8.2+ with mbstring, namespace
87+
`WordPress\DistributedEditing\Automerge`, no WordPress dependency).
88+
Excluded from phpcs; frozen like y-php. Its own conformance suite runs
89+
in CI: `php includes/lib/automerge-php/tests/run.php` (<1 s, no
90+
WordPress; 680 mapped upstream tests). FULL parity needs the fixed
91+
GB11 grapheme rules of PCRE2 ≥ 10.43, which PHP bundles from 8.4 —
92+
under PHP ≤ 8.3 builds carrying PCRE2 10.42 (stock GitHub runners,
93+
Ubuntu 24.04 packages) two adjacent emoji-ZWJ sequences count as ONE
94+
`\X` cluster and exactly 2 of the 680 tests fail (grapheme cursor
95+
tracking + a UTF-16-boundary splice); CI pins PHP 8.4 for this step.
96+
The 11 upstream fixture files
97+
the runner reads live under `automerge-php/upstream/automerge/`
98+
(fetched from automerge/automerge; pin recorded in
99+
`VENDORED_FROM_COMMIT.txt` — the source branches referenced an
100+
upstream submodule that was never committed). Running the suite
101+
rewrites the tracked `PORTING_STATUS.json` (timestamps); revert that
102+
side-effect after local runs. NOTE: the DE-RTC
103+
*shipping* merge path (`native-automerge-blocks-v1`) never calls this
104+
library — it backs only the dead legacy whole-text lane and
105+
external-repair; it is vendored for fidelity and future use.
106+
- `lib/automerge-php-loader.php` — lazy PSR-4 loader shim +
107+
`gutenberg_sync_engines_automerge_php_is_supported()` (PHP ≥ 8.2 +
108+
mbstring gate).
64109
- `src/` — client JS/TS (webpack entry `src/index.ts` → `build/sync-engines.js`,
65110
externalizes `@wordpress/sync`→`wp.sync` and `yjs`→`wp.sync.Y`):
66111
- `engines/intent-log/` — the **frozen cross-language core** (byte-matched
@@ -189,15 +234,19 @@ everywhere, so auth even succeeds); the first visible failure is
189234
`activatePlugin( 'gutenberg' )` in global-setup dying with
190235
"Unexpected end of JSON input". Always pass `WP_BASE_URL` in that case.
191236

192-
Current green baseline: **Jest 342**, **PHPUnit 148 (659 assertions)**,
193-
**e2e 40/40** (occasional flake under full-suite load — a save notice or a
194-
fixture login navigation; green solo), **e2e:websocket 1 skipped** (see
237+
Current green baseline: **Jest 347**, **PHPUnit 174 (763 assertions)**,
238+
**e2e 32/32** (occasional flake under full-suite load — a save notice, a
239+
fixture login navigation, or `http-only/collaboration-sync-body-size`
240+
failing after a preceding engine-flip suite [verified pre-existing: the
241+
yjs suite followed by body-size reproduces it without de-rtc involved];
242+
each green solo), **e2e:websocket 1 skipped** (see
195243
below — the peer-relay WS fixture needs a client-merging engine and none
196-
remains), plus the
197-
vendored y-php library's own conformance suite (**428 tests**, run
198-
separately — see `includes/lib/` above). CI (`.github/workflows/ci.yml`)
199-
certifies all suites on pushes to `main` and PRs; the e2e job leans on the
200-
base config's 2-retries-in-CI to absorb the flake.
244+
remains), plus the vendored libraries' own conformance suites run
245+
separately: y-php (**428 tests**) and automerge-php (**680 mapped
246+
upstream tests**, `php includes/lib/automerge-php/tests/run.php`). CI
247+
(`.github/workflows/ci.yml`) certifies all suites on pushes to `main` and
248+
PRs; the e2e job leans on the base config's 2-retries-in-CI to absorb the
249+
flakes.
201250

202251
The transport-specific e2e suites live here (relocated from the framework):
203252
`tests/e2e/specs/http-only/` runs in the default suite; `tests/e2e/specs/
@@ -336,6 +385,20 @@ and the test suites never start a daemon.
336385
materialization carries the same Phase-2a wrapper simplification as
337386
intent-log. Genesis blocks must set `isValid: true` or the editor renders
338387
them as invalid-content recovery blocks (has bitten).
388+
- **de-rtc known gaps** (docs/engine-comparison.md has the full list): no
389+
title sync (proposals carry content only); no review lane yet — server
390+
escalations (`manual-conflict-required`, `requires-unfiltered-html`)
391+
surface as dispositions but no UI presents them, and the client abandons
392+
an escalated proposal once canonical applies; truly concurrent SAME-block
393+
edits resolve block-level last-writer-wins client-side (yjs-server's
394+
silent-register-LWW class, coarser grain); the client sends
395+
`clientUpdate: null` and relies on the server's engine-unaware-writer
396+
lane to derive block-native operations (the client-side descriptor
397+
builder + cross-language fingerprint vectors are unported); no benchmark
398+
authoring profile yet (`tests/benchmarks` cannot drive de-rtc); ingest
399+
serializes per room under the intent-log-style GET_LOCK, and every
400+
accepted proposal broadcasts FULL content rows (storage bounded by
401+
checkpoints, but row bytes scale with document size).
339402
- **Intent-log echo race:** editor pushes racing live keystrokes can corrupt
340403
canvas text (observed under load; worst over websocket's per-keystroke
341404
cadence — benchmark that transport under yjs-server meanwhile). Deferring or

‎README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,11 @@ Engines (how concurrent edits merge):
2525
- **yjs-server** — a server-authoritative CRDT: the vendored y-php library
2626
merges every update into a canonical room document server-side, compacts
2727
by itself, and materializes post content.
28+
- **de-rtc** — Distributed Editing's save-centric model: clients propose
29+
whole content against a named base version, the server three-way-merges
30+
every proposal with the merge core ported from wordpress-develop's
31+
`add/distributed-editing` branch, and genuine conflicts escalate instead
32+
of silently merging.
2833

2934
Transports (how updates move):
3035

0 commit comments

Comments
 (0)