fix: ensure node and services are watcing for the same shutdown signal from the context cancel - #1406
Conversation
WalkthroughThe changes across various Go files reflect a shift towards improved context management, with the introduction of context with cancel in node creation functions and context cancellation checks in block processing. Error handling has been refined, and a new debugging tool for context cancellation has been added. The changes enhance the robustness and debuggability of the system, ensuring graceful shutdowns and better error tracking. Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on X ? TipsChat with CodeRabbit Bot (
|
…e from the context cancel
f6bd6c7 to
4cb83d1
Compare
There was a problem hiding this comment.
Review Status
Actionable comments generated: 0
Configuration used: CodeRabbit UI
Files selected for processing (2)
- node/full.go (2 hunks)
- node/light.go (2 hunks)
Additional comments: 5
node/full.go (1)
- 95-107: The changes to the
newFullNodefunction correctly implement the creation of a context with cancel and defer the cancel function to ensure it is called in case of an error. This aligns with the PR objectives to ensure that nodes and services respond to the same shutdown signal.node/light.go (4)
49-61: The changes to the
newLightNodefunction signature and the addition of context handling with a deferred cancel function are consistent with the PR objectives. This ensures that the services receive the shutdown signal when the node shuts down. The use of named return values for error handling is idiomatic in Go.55-60: The implementation of the context creation with
context.WithCancel(ctx)and the deferred cancel function is correct. It ensures that the cancel function is called if an error occurs, which is a good practice for resource cleanup.81-81: The error handling in the
newLightNodefunction correctly usesfmt.Errorfwith the%wverb to wrap errors, which allows for error unwrapping and is considered a best practice in Go.81-86: The
OnStopmethod correctly calls the cancel function to initiate the shutdown process. However, it's important to ensure that all errors from stopping services are handled properly. The use ofmultierr.Appendis a good practice to collect all errors.
There was a problem hiding this comment.
Review Status
Actionable comments generated: 8
Configuration used: CodeRabbit UI
Files selected for processing (6)
- block/manager.go (3 hunks)
- node/full_client.go (2 hunks)
- node/full_node_integration_test.go (4 hunks)
- node/node_test.go (1 hunks)
- state/txindex/indexer_service.go (2 hunks)
- test/context/context.go (1 hunks)
Additional comments: 12
block/manager.go (3)
386-393: The addition of context cancellation checks at the start of the
trySyncNextBlockfunction is a good practice to ensure that the function can exit early if the context is canceled. This is in line with the PR's objective to improve shutdown signal handling.523-528: The context cancellation check added to the
processNextDABlockfunction is consistent with the changes made intrySyncNextBlock. This ensures that the function respects the context's cancellation signal and can return early if necessary.634-639: The context cancellation check added to the
publishBlockfunction is consistent with the other changes and follows the same pattern of checking for context cancellation at the start of the function. This ensures that the function does not proceed if the context has been canceled.node/full_client.go (2)
112-118: The change to use the provided context for unsubscribing from the event bus in the
BroadcastTxCommitmethod aligns with the PR's objective to improve context management. This ensures that the unsubscribe operation is aware of the same context that was used for subscription, which can help prevent potential leaks or unexpected behavior during shutdown.885-891: The update to the
resubscribemethod to use the context from the client's node (c.node.ctx) instead ofcontext.Background()is consistent with the PR's goal of ensuring that nodes and services are watching for the same shutdown signal. This change should help prevent issues where services might not receive a shutdown signal, as they will now use the same context as the node.node/full_node_integration_test.go (3)
71-76: The addition of context creation and cancellation before the
newFullNodefunction call is consistent with the PR's objective to ensure that nodes and services respond to the same shutdown signal. This change should help prevent the issue where services were not shutting down properly during tests.488-497: The retry mechanism added for ensuring that blocks are DA included is a good addition for robustness in tests. It's important to verify that the retry logic is correctly implemented and that the error messages are clear for debugging purposes.
71-76: The changes made to the context handling in the test functions align with the PR's objective and improve the consistency of shutdown signal handling. It's important to ensure that the context is cancelled at the right time to avoid resource leaks, especially in the context of testing where multiple tests may run in sequence or parallel.
node/node_test.go (1)
- 20-24: The removal of the deferred
node.Cancel()call incleanUpNodefunction changes the cleanup behavior. Ensure that the node's resources are properly released and that there are no side effects due to this change, such as resource leaks or incomplete shutdowns.state/txindex/indexer_service.go (2)
47-60: The change to use
is.ctxin theSubscribeUnbufferedcalls aligns with the PR's objective to ensure consistent context management across nodes and services. This should allow for proper shutdown handling when the context is canceled.130-135: The update to use
is.ctxin theUnsubscribeAllcall within theOnStopmethod is consistent with the PR's goal of unified context management, which should help in proper resource cleanup during shutdown.test/context/context.go (1)
- 10-31: The implementation of
WithDebugCancelFunclooks correct and should provide valuable debugging information when the context is canceled. It captures the stack trace of the last 5 calls before the cancel function is invoked and prints them, which can help in tracing the source of premature or unexpected cancellations.
|
test failure: https://github.com/rollkit/rollkit/actions/runs/7174771927/job/19536842207?pr=1406 |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #1406 +/- ##
==========================================
- Coverage 51.09% 50.79% -0.30%
==========================================
Files 46 46
Lines 6422 6448 +26
==========================================
- Hits 3281 3275 -6
- Misses 2802 2831 +29
- Partials 339 342 +3 ☔ View full report in Codecov by Sentry. |
This is a known flaky test, can be resolved in a follow up. |
Manav-Aggarwal
left a comment
There was a problem hiding this comment.
Looks good, passed all tests.
Overview
The nodes were creating a new context with cancel after passing the original context through to the services. This meant that when the node shutdown and cancelled its context, its services did not receive a shutdown signal until the original caller cancelled its context. This was leading to panics in testing about writing to a logger test file that was already closed.
This change was verified by reverting the changes in #1402 and testing in a loop 100 times with no panics.
EDIT 1
Since no good deed goes unpunished 🙃 some timeouts surfaced. This was due to
context.Background()being used for subscribe and unsubscribe events. Under the hood those events are watching forctx.Done()events, so whencontext.Background()is passed in, they can hang indefinitely if the other signals aren't triggered. The test that was most prone to this timeout was run in a loop 1000 times to verify the issue was fixed.Additionally, added some more checks for
ctx.Done()for faster shutdowns.Summary by CodeRabbit
New Features
Refactor
Tests
Chores
Documentation