Skip to content

Commit fe80870

Browse files
committed
docs(ui): design live failover audit verification
1 parent 4e0550a commit fe80870

1 file changed

Lines changed: 77 additions & 0 deletions

File tree

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# Live Failover and Audit UI Verification Design
2+
3+
## Objective
4+
5+
Exercise Orchestrator's history UI with real operational data from the existing three-node MySQL Docker lab. The pass must populate and inspect general audit operations, failure detections, recovery summaries, and a recovery detail page without recreating or deleting the MySQL containers or their volumes.
6+
7+
## Chosen approach
8+
9+
Use one controlled hard-primary failure because it produces the complete state chain needed by the UI:
10+
11+
1. enable backend audit persistence in the functional Orchestrator configuration;
12+
2. recreate only the Orchestrator container with `--no-deps`;
13+
3. rediscover the healthy `mysql1` primary with `mysql2` and `mysql3` replicas;
14+
4. stop only `mysql1`;
15+
5. wait for `DeadMaster` detection and a successful automated recovery;
16+
6. restart `mysql1` and restore the original lab topology and ProxySQL hostgroups;
17+
7. retain the Orchestrator SQLite session so its audit and recovery history remains available for browser testing.
18+
19+
This is preferable to a synthetic fixture because it validates the browser against the real API payloads. It is preferable to a forced logical failover because a logical failover does not cover the actual failure-detection history.
20+
21+
## Safety boundaries
22+
23+
- Record container IDs, health, topology roles, and ProxySQL writer state before mutation.
24+
- Never run a broad `docker compose up`, `down`, volume removal, or dependency recreation.
25+
- Recreate only Orchestrator and always use `--no-deps`.
26+
- Stop and restart only the resolved current primary container.
27+
- Preserve all MySQL volumes and container identities.
28+
- Use bounded waits for detection, recovery, restart, and topology restoration.
29+
- If recovery fails, restart the stopped primary immediately, capture diagnostics, and restore the original topology before further investigation.
30+
- Confirm the original three MySQL container IDs and healthy state after restoration.
31+
32+
## UI states to verify
33+
34+
### Audit operations
35+
36+
- At least one table row renders with timestamp, operation type, instance, and message.
37+
- Pagination controls remain disabled/enabled according to available pages.
38+
- Instance links use working registered routes.
39+
- Long messages and identifiers do not create document-level horizontal overflow.
40+
41+
### Failure detections
42+
43+
- A `DeadMaster` detection row renders with failed instance, affected replicas, cluster, and detection time.
44+
- Expanding the detection reveals recorded context and a working related-recovery link.
45+
- API-derived values are displayed as text rather than executable markup.
46+
47+
### Recoveries
48+
49+
- A successful recovery row renders with failed instance and promoted successor.
50+
- Opening the recovery renders the summary, acknowledgement state, related detection, and recovery steps.
51+
- Empty, populated, and unavailable states remain mutually exclusive.
52+
53+
### Existing topology pages
54+
55+
- Cluster dashboard and topology reflect the promoted primary during recovery.
56+
- After cleanup, the lab returns to `mysql1` as writable primary and `mysql2`/`mysql3` as running replicas.
57+
58+
## Remediation policy
59+
60+
Browser defects discovered with populated data will be reproduced by a focused failing test before production changes. Fixes will preserve existing API contracts and routes, keep CSS scoped below the history workspace, escape API-derived content, and retain narrow-screen table scrolling without document overflow.
61+
62+
## Verification
63+
64+
- Focused regression tests for every discovered defect, including a demonstrated red-to-green cycle.
65+
- Full `go test ./go/http -count=1`.
66+
- All Node UI behavior tests and JavaScript syntax checks.
67+
- Functional smoke suite after topology restoration.
68+
- Browser audit at desktop and 390px widths for the three populated history pages and recovery detail.
69+
- Browser console audit for errors and warnings.
70+
- Pre/post MySQL container-ID comparison and final role/replication checks.
71+
72+
## Out of scope
73+
74+
- PostgreSQL failover UI verification.
75+
- Agent-enabled UI verification.
76+
- Redesigning the topology workspace or global navigation.
77+
- Acknowledging or deleting the generated recovery record solely to make the page look cleaner.

0 commit comments

Comments
 (0)