Repository navigation
Commit 6a65ec4
Migrate DE-RTC as the third sync engine (#9)
* Vendor automerge-php at includes/lib with lazy loader
Imported verbatim from the Gutenberg chriszarate/refreshed-de-rtc branch
(lib/compat/wordpress-7.1/distributed-editing/automerge-php, itself a
verbatim port of wordpress-develop add/distributed-editing, PR
WordPress/wordpress-develop#12334). MIT, PHP 8.2+, namespace
WordPress\DistributedEditing\Automerge, no WordPress dependency.
The library's conformance suite (php tests/run.php) referenced an
upstream/automerge submodule never committed on either source branch;
the 11 fixture files it reads (3 storage fixtures + 8 fuzz-crashers)
are now vendored under upstream/automerge/, fetched from
automerge/automerge@8d7b12f (MIT;
recorded in VENDORED_FROM_COMMIT.txt). Suite passes 680/680 locally.
includes/lib/automerge-php-loader.php mirrors y-php-loader.php: a lazy
PSR-4 shim plus gutenberg_sync_engines_automerge_php_is_supported()
(PHP >= 8.2 + mbstring) for callers to gate on. Excluded from phpcs
like y-php.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Port the DE-RTC merge core as a frozen engine slice
includes/engines/de-rtc/merge-core.php is the exact call-graph closure
(113 functions, ~6.4k lines) of the engine-facing entry points in the
Gutenberg chriszarate/refreshed-de-rtc branch's de-rtc.php (itself a
verbatim port of wordpress-develop add/distributed-editing): update
construction/normalization, the serialized-block and block-identity
three-way merges, the rich-text merge model, version snapshots,
sync-meta format/parse, canonicalization/hashing, and the automerge
runtime gate. Extracted token-exactly; the only deltas are the vendored
library path and delegating autoloading to automerge-php-loader.php
(marked DELTA in place). Excluded from phpcs and frozen like the
intent-log cross-language core.
The plugin loads it behind a function_exists guard so a Core/Gutenberg
build that ships DE-RTC itself wins.
tests/phpunit/wpDeRtcMergeCore.php drives the merge seam directly:
disjoint block edits merge, non-overlapping rich-text edits within one
paragraph merge, overlapping edits escalate (de_rtc_rebase_failed /
manual_conflict_required), concurrent differing appends escalate by
design, tampered/mismatched updates are rejected, engine-unaware saves
get server-generated updates, and version snapshots trim at the limit.
Two upstream-faithful behaviors are pinned rather than fixed: the
legacy native-automerge-php-v1 lane is dead code (NativePort::merge
accepts only document pairs, so operations-array updates are rejected
as unverifiable) and same-edge concurrent appends require review.
Full PHPUnit: 162 tests / 707 assertions green (baseline 148 + 14 new).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add WP_De_RTC_Engine: the DE-RTC merge core behind the engine SPI
The third engine (slug de-rtc): server-governed three-way merges of
whole-content proposals. Clients send proposal updates ({proposalId,
baseVersion, proposedContent, clientUpdate}); under the per-room ingest
lock (three-way merges are order-dependent, so ingest serializes like
intent-log, unlike yjs-server's lock-free CRDT path) the engine resolves
the named base from its version snapshots, merges with
wp_de_rtc_get_automerge_retry_save_result(), and either advances the
canonical document (disposition applied + a server-authored content row
peers replay) or settles the proposal as escalated
(manual-conflict-required, requires-unfiltered-html) or voided
(unknown-base-version, invalid-payload, tamper/malformed details).
Canonical state ({version, content, sync_meta.version_snapshots}) lives
in room meta — not in a post_content sync-meta block, which is what
made most of upstream DE-RTC's periphery (revision scan, kses allowance,
preservation filters) unnecessary here. Genesis is deterministic from
post content and ADOPTS an existing upstream sync-meta block (version
lineage continues, content stripped). Server-driven checkpoints reuse
the shared retention invariant (trim behind the previous checkpoint,
floor-clamp stale cursors).
The capability lane escalates whole proposals kses would rewrite
(upstream sequesters just the risky blocks for review — documented gap
until the review lane exists). Registered in the plugin and on the
settings screen.
PHPUnit: 174 tests / 763 assertions green (12 new engine tests: SPI
identity, genesis+lineage, apply/broadcast, stale-base three-way rebase,
conflict escalation, kses escalation, per-row voids, type fencing,
persistence across instances, checkpoint trim + floor clamp).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add the de-rtc client engine: proposal wire over a local doc bridge
The client half of DE-RTC keeps Distributed Editing's core inversion:
the client NEVER merges. A local Y.Doc (shared engines/yjs schema —
same undo manager, awareness anchor, and record mapping as the yjs
engines) bridges the editor; the session codec speaks the proposal
wire:
- One coalesced in-flight proposal: local edits mark the doc dirty and
a whole-content proposal is built only when none is pending, so a
typing burst costs one proposal per poll cycle. The base version is
the version last APPLIED to the doc — stale bases are exactly what
the server's three-way merge is for.
- Canonical rows are deferred while local edits are dirty/in flight
(applying them would overwrite edits the server has not seen); the
newest deferred row applies when the local state settles. Escalated
proposals are abandoned locally once canonical applies — the upstream
review/decision UI is a listed follow-up, as is the client-side
block-native descriptor builder (proposals send clientUpdate: null
and the server derives operations via its engine-unaware writer
lane).
- Server genesis bootstraps the doc (hydrate is a no-op; pre-bootstrap
local changes buffer and replay), mirroring yjs-server.
- Collections are inert (the intent-log precedent): awareness flows,
rows are ignored, nothing is proposed.
The doc bridge serializes/parses through the editor's own block library
(wp.blocks at runtime; resolved from the subtree in Jest/tsc, mocked in
tests). Registered as the third adapter in src/index.ts.
Jest: 347 tests green (8 new: identity, no-seed hydrate + genesis
bootstrap, pre-bootstrap buffering, proposal coalescing + settle,
remote apply + onRemoteChange + rebased base version, deferred
canonical under in-flight, own-row slot clearing, inert collections).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix de-rtc client rebase races; add the de-rtc collaboration e2e spec
Three client fixes found by driving the real editor two-up (each
verified with the spec + Playwright traces):
1. Own accepted rows advance the base version. A client's successive
typing chunks were proposed against the same old base, so an
author's continuation conflicted with their own just-accepted chunk
(rich-text overlap at the same position). When our own row
round-trips unchanged, advance the version WITHOUT touching the doc
(newer keystrokes survive); the next chunk proposes against it.
2. Block-level incorporation of own merged rows. When the server merges
peers' work into our proposal, adopt canonical blocks we have not
edited since proposing, keep locally-edited blocks (reconciled by
the next proposal), handle peer tail-insertions, and bail to
deferral on any other structure change. Truly concurrent same-block
edits resolve block-level last-writer-wins — the same class of
silent register policy yjs-server carries, at coarser grain
(documented in the engine comparison).
3. Dispositions only settle the proposal they name. A previous
proposal's disposition arrives in the response AFTER the one whose
rows already settled it; clearing the in-flight slot unconditionally
let a peer's canonical row clobber unproposed local edits, which the
next incorporation then wrongly pinned as 'local'.
The e2e spec (three scenarios: server-side genesis + bidirectional text
sync, concurrent different-block edits surviving the three-way merge,
and the save flow persisting both users' settled edits) passes
repeatedly in-suite and solo. It opens sessions through a local helper
that force-closes the second user's welcome guide when the fixture's
preference dispatch loses to late preference hydration (an open modal
aria-hides the page and fails discovery even though sync is healthy).
Deliberately absent scenarios are the v1 gaps: title sync, and the
empty-post concurrent-append case (a BY-DESIGN escalation under DE-RTC
policy).
Known pre-existing full-suite flake, verified NOT caused by these
changes: http-only/collaboration-sync-body-size can fail after any
engine-flip suite under load (control: yjs suite + body-size fails
identically without de-rtc involved); solo it is green, CI retries
absorb it.
Jest 347 green; de-rtc e2e 3/3; full e2e 31/32 (the flake above).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document the de-rtc engine and run automerge-php conformance in CI
- AGENTS.md: engines list, repo-layout entries for the frozen merge core
and the vendored automerge-php (incl. the PORTING_STATUS.json rewrite
side-effect and the fact that the shipping merge path never calls the
library), new green baselines (Jest 347, PHPUnit 174/763, e2e 32/32
with the pre-existing body-size-after-engine-flip flake called out),
and the de-rtc known-gaps entry.
- docs/engine-comparison.md: de-rtc added to the one-sentence list, the
parity and host-resource tables, known gaps (no benchmark profile, no
review UI/title sync, block-level LWW on same-block concurrency, no
client-side proof descriptors), and the bottom line.
- README.md: de-rtc in the engines list.
- CI: run the vendored automerge-php conformance suite (680 mapped
upstream tests, plain php runner, no WordPress) alongside y-php's.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix conflict syntax error
* Actually fix conflict
* Pin PHP 8.4 for the conformance suites; surface the automerge report
The automerge-php conformance step failed on CI while passing locally:
its grapheme-cluster paths split text with PCRE2's \X, and PCRE2 10.42
(bundled by PHP <= 8.3 builds, including the stock runner's 8.3.6 and
Ubuntu 24.04 packages) carries a GB11 bug where two ADJACENT emoji-ZWJ
sequences count as ONE extended grapheme cluster. PCRE2 10.44 (bundled
from PHP 8.4) counts two. Exactly 2 of the 680 mapped upstream tests
fail under the old tables — grapheme cursor tracking through an
inserted family emoji, and a UTF-16-boundary splice — reproduced in
php:8.2/8.3-cli containers (678/680) vs php:8.4-cli (680/680).
CI now pins PHP 8.4 via setup-php for the conformance suites (y-php is
7.4+-compatible and unaffected), and the automerge step keeps its JSON
report visible — a pass prints the summary line, a failure dumps the
failing entries — instead of discarding stdout, which had swallowed the
report entirely (the step 'failed silently' in 1.6 s). AGENTS.md
documents the PCRE2 >= 10.43 parity requirement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>1 parent 021e37f commit 6a65ec4
59 files changed
Lines changed: 52974 additions & 59 deletions
File tree
- .github/workflows
- docs
- includes
- admin
- engines/de-rtc
- lib
- automerge-php
- src
- tests
- tools
- upstream/automerge
- rust/automerge/tests
- fixtures
- fuzz-crashers
- src
- engines
- de-rtc
- tests
- e2e/specs
- js/engines/de-rtc
- phpunit
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
81 | 81 | | |
82 | 82 | | |
83 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
84 | 97 | | |
85 | 98 | | |
86 | 99 | | |
87 | 100 | | |
88 | 101 | | |
89 | | - | |
| 102 | + | |
90 | 103 | | |
91 | 104 | | |
92 | 105 | | |
93 | 106 | | |
94 | 107 | | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
95 | 131 | | |
96 | 132 | | |
97 | 133 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
26 | 32 | | |
27 | 33 | | |
28 | 34 | | |
| |||
42 | 48 | | |
43 | 49 | | |
44 | 50 | | |
45 | | - | |
| 51 | + | |
46 | 52 | | |
47 | 53 | | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
48 | 66 | | |
49 | 67 | | |
50 | 68 | | |
| |||
61 | 79 | | |
62 | 80 | | |
63 | 81 | | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
64 | 109 | | |
65 | 110 | | |
66 | 111 | | |
| |||
189 | 234 | | |
190 | 235 | | |
191 | 236 | | |
192 | | - | |
193 | | - | |
194 | | - | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
195 | 243 | | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
201 | 250 | | |
202 | 251 | | |
203 | 252 | | |
| |||
336 | 385 | | |
337 | 386 | | |
338 | 387 | | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
339 | 402 | | |
340 | 403 | | |
341 | 404 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
28 | 33 | | |
29 | 34 | | |
30 | 35 | | |
| |||
0 commit comments