|
| 1 | +# Live Failover and Audit UI Verification Design |
| 2 | + |
| 3 | +## Objective |
| 4 | + |
| 5 | +Exercise Orchestrator's history UI with real operational data from the existing three-node MySQL Docker lab. The pass must populate and inspect general audit operations, failure detections, recovery summaries, and a recovery detail page without recreating or deleting the MySQL containers or their volumes. |
| 6 | + |
| 7 | +## Chosen approach |
| 8 | + |
| 9 | +Use one controlled hard-primary failure because it produces the complete state chain needed by the UI: |
| 10 | + |
| 11 | +1. enable backend audit persistence in the functional Orchestrator configuration; |
| 12 | +2. recreate only the Orchestrator container with `--no-deps`; |
| 13 | +3. rediscover the healthy `mysql1` primary with `mysql2` and `mysql3` replicas; |
| 14 | +4. stop only `mysql1`; |
| 15 | +5. wait for `DeadMaster` detection and a successful automated recovery; |
| 16 | +6. restart `mysql1` and restore the original lab topology and ProxySQL hostgroups; |
| 17 | +7. retain the Orchestrator SQLite session so its audit and recovery history remains available for browser testing. |
| 18 | + |
| 19 | +This is preferable to a synthetic fixture because it validates the browser against the real API payloads. It is preferable to a forced logical failover because a logical failover does not cover the actual failure-detection history. |
| 20 | + |
| 21 | +## Safety boundaries |
| 22 | + |
| 23 | +- Record container IDs, health, topology roles, and ProxySQL writer state before mutation. |
| 24 | +- Never run a broad `docker compose up`, `down`, volume removal, or dependency recreation. |
| 25 | +- Recreate only Orchestrator and always use `--no-deps`. |
| 26 | +- Stop and restart only the resolved current primary container. |
| 27 | +- Preserve all MySQL volumes and container identities. |
| 28 | +- Use bounded waits for detection, recovery, restart, and topology restoration. |
| 29 | +- If recovery fails, restart the stopped primary immediately, capture diagnostics, and restore the original topology before further investigation. |
| 30 | +- Confirm the original three MySQL container IDs and healthy state after restoration. |
| 31 | + |
| 32 | +## UI states to verify |
| 33 | + |
| 34 | +### Audit operations |
| 35 | + |
| 36 | +- At least one table row renders with timestamp, operation type, instance, and message. |
| 37 | +- Pagination controls remain disabled/enabled according to available pages. |
| 38 | +- Instance links use working registered routes. |
| 39 | +- Long messages and identifiers do not create document-level horizontal overflow. |
| 40 | + |
| 41 | +### Failure detections |
| 42 | + |
| 43 | +- A `DeadMaster` detection row renders with failed instance, affected replicas, cluster, and detection time. |
| 44 | +- Expanding the detection reveals recorded context and a working related-recovery link. |
| 45 | +- API-derived values are displayed as text rather than executable markup. |
| 46 | + |
| 47 | +### Recoveries |
| 48 | + |
| 49 | +- A successful recovery row renders with failed instance and promoted successor. |
| 50 | +- Opening the recovery renders the summary, acknowledgement state, related detection, and recovery steps. |
| 51 | +- Empty, populated, and unavailable states remain mutually exclusive. |
| 52 | + |
| 53 | +### Existing topology pages |
| 54 | + |
| 55 | +- Cluster dashboard and topology reflect the promoted primary during recovery. |
| 56 | +- After cleanup, the lab returns to `mysql1` as writable primary and `mysql2`/`mysql3` as running replicas. |
| 57 | + |
| 58 | +## Remediation policy |
| 59 | + |
| 60 | +Browser defects discovered with populated data will be reproduced by a focused failing test before production changes. Fixes will preserve existing API contracts and routes, keep CSS scoped below the history workspace, escape API-derived content, and retain narrow-screen table scrolling without document overflow. |
| 61 | + |
| 62 | +## Verification |
| 63 | + |
| 64 | +- Focused regression tests for every discovered defect, including a demonstrated red-to-green cycle. |
| 65 | +- Full `go test ./go/http -count=1`. |
| 66 | +- All Node UI behavior tests and JavaScript syntax checks. |
| 67 | +- Functional smoke suite after topology restoration. |
| 68 | +- Browser audit at desktop and 390px widths for the three populated history pages and recovery detail. |
| 69 | +- Browser console audit for errors and warnings. |
| 70 | +- Pre/post MySQL container-ID comparison and final role/replication checks. |
| 71 | + |
| 72 | +## Out of scope |
| 73 | + |
| 74 | +- PostgreSQL failover UI verification. |
| 75 | +- Agent-enabled UI verification. |
| 76 | +- Redesigning the topology workspace or global navigation. |
| 77 | +- Acknowledging or deleting the generated recovery record solely to make the page look cleaner. |
0 commit comments