Skip to content

fix: ensure node and services are watcing for the same shutdown signal from the context cancel - #1406

Merged
gupadhyaya merged 6 commits into
mainfrom
sevey/node-ctx-fix
Dec 15, 2023
Merged

gupadhyaya merged 6 commits into
mainfrom
sevey/node-ctx-fix

Conversation

@MSevey

@MSevey MSevey commented Dec 11, 2023

Copy link
Copy Markdown
Contributor

Overview

The nodes were creating a new context with cancel after passing the original context through to the services. This meant that when the node shutdown and cancelled its context, its services did not receive a shutdown signal until the original caller cancelled its context. This was leading to panics in testing about writing to a logger test file that was already closed.

This change was verified by reverting the changes in #1402 and testing in a loop 100 times with no panics.

EDIT 1
Since no good deed goes unpunished 🙃 some timeouts surfaced. This was due to context.Background() being used for subscribe and unsubscribe events. Under the hood those events are watching for ctx.Done() events, so when context.Background() is passed in, they can hang indefinitely if the other signals aren't triggered. The test that was most prone to this timeout was run in a loop 1000 times to verify the issue was fixed.

Additionally, added some more checks for ctx.Done() for faster shutdowns.

Summary by CodeRabbit

  • New Features

    • Enhanced node shutdown process with context cancellation for improved service termination.
  • Refactor

    • Updated error handling and context management in node creation functions.
    • Improved context usage in block synchronization and event subscription methods.
  • Tests

    • Adjusted full node integration tests to reflect new context management.
  • Chores

    • Removed redundant node cancellation in test cleanup function.
  • Documentation

    • No visible changes to end-users.

@coderabbitai

coderabbitai Bot commented Dec 11, 2023

Copy link
Copy Markdown
Contributor

Walkthrough

The changes across various Go files reflect a shift towards improved context management, with the introduction of context with cancel in node creation functions and context cancellation checks in block processing. Error handling has been refined, and a new debugging tool for context cancellation has been added. The changes enhance the robustness and debuggability of the system, ensuring graceful shutdowns and better error tracking.

Changes

File(s) Summary
node/full.go, node/light.go Updated newFullNode and newLightNode functions to create a context with cancel and handle errors differently.
block/manager.go, node/full_client.go Added context cancellation checks and adjusted context usage in event subscription and block synchronization functions.
node/full_node_integration_test.go, node/node_test.go Modified tests to adjust context creation and cancellation, and removed deferred node cancellation.
state/txindex/indexer_service.go Replaced context.Background() with is.ctx for better context management in IndexerService.
test/context/context.go Introduced WithDebugCancelFunc for enhanced debugging of context cancellation.

🐇✨
In the realm of code, where contexts entwine,
A rabbit hopped through, leaving changes so fine.
With a cancel in hand, and errors at bay,
It leapt through the Go, making contexts obey. 🌿🚀

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on X ?


Tips

Chat with CodeRabbit Bot (@coderabbitai)

  • If you reply to a review comment from CodeRabbit, the bot will automatically respond.
  • To engage with CodeRabbit bot directly around the specific lines of code in the PR, mention @coderabbitai in your review comment
  • Note: Review comments are made on code diffs or files, not on the PR overview.
  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Note: For conversation with the bot, please use the review comments on code diffs or files.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • The JSON schema for the configuration file is available here.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

@MSevey
MSevey force-pushed the sevey/node-ctx-fix branch from f6bd6c7 to 4cb83d1 Compare December 11, 2023 17:13
@MSevey MSevey added T:bug Something isn't working C:ci Component: Continuous Integration labels Dec 11, 2023

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 0

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 2f6ed32 and f6bd6c7.
Files selected for processing (2)
  • node/full.go (2 hunks)
  • node/light.go (2 hunks)
Additional comments: 5
node/full.go (1)
  • 95-107: The changes to the newFullNode function correctly implement the creation of a context with cancel and defer the cancel function to ensure it is called in case of an error. This aligns with the PR objectives to ensure that nodes and services respond to the same shutdown signal.
node/light.go (4)
  • 49-61: The changes to the newLightNode function signature and the addition of context handling with a deferred cancel function are consistent with the PR objectives. This ensures that the services receive the shutdown signal when the node shuts down. The use of named return values for error handling is idiomatic in Go.

  • 55-60: The implementation of the context creation with context.WithCancel(ctx) and the deferred cancel function is correct. It ensures that the cancel function is called if an error occurs, which is a good practice for resource cleanup.

  • 81-81: The error handling in the newLightNode function correctly uses fmt.Errorf with the %w verb to wrap errors, which allows for error unwrapping and is considered a best practice in Go.

  • 81-86: The OnStop method correctly calls the cancel function to initiate the shutdown process. However, it's important to ensure that all errors from stopping services are handled properly. The use of multierr.Append is a good practice to collect all errors.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Status

Actionable comments generated: 8

Configuration used: CodeRabbit UI

Commits Files that changed from the base of the PR and between 4cb83d1 and 6ba6b90.
Files selected for processing (6)
  • block/manager.go (3 hunks)
  • node/full_client.go (2 hunks)
  • node/full_node_integration_test.go (4 hunks)
  • node/node_test.go (1 hunks)
  • state/txindex/indexer_service.go (2 hunks)
  • test/context/context.go (1 hunks)
Additional comments: 12
block/manager.go (3)
  • 386-393: The addition of context cancellation checks at the start of the trySyncNextBlock function is a good practice to ensure that the function can exit early if the context is canceled. This is in line with the PR's objective to improve shutdown signal handling.

  • 523-528: The context cancellation check added to the processNextDABlock function is consistent with the changes made in trySyncNextBlock. This ensures that the function respects the context's cancellation signal and can return early if necessary.

  • 634-639: The context cancellation check added to the publishBlock function is consistent with the other changes and follows the same pattern of checking for context cancellation at the start of the function. This ensures that the function does not proceed if the context has been canceled.

node/full_client.go (2)
  • 112-118: The change to use the provided context for unsubscribing from the event bus in the BroadcastTxCommit method aligns with the PR's objective to improve context management. This ensures that the unsubscribe operation is aware of the same context that was used for subscription, which can help prevent potential leaks or unexpected behavior during shutdown.

  • 885-891: The update to the resubscribe method to use the context from the client's node (c.node.ctx) instead of context.Background() is consistent with the PR's goal of ensuring that nodes and services are watching for the same shutdown signal. This change should help prevent issues where services might not receive a shutdown signal, as they will now use the same context as the node.

node/full_node_integration_test.go (3)
  • 71-76: The addition of context creation and cancellation before the newFullNode function call is consistent with the PR's objective to ensure that nodes and services respond to the same shutdown signal. This change should help prevent the issue where services were not shutting down properly during tests.

  • 488-497: The retry mechanism added for ensuring that blocks are DA included is a good addition for robustness in tests. It's important to verify that the retry logic is correctly implemented and that the error messages are clear for debugging purposes.

  • 71-76: The changes made to the context handling in the test functions align with the PR's objective and improve the consistency of shutdown signal handling. It's important to ensure that the context is cancelled at the right time to avoid resource leaks, especially in the context of testing where multiple tests may run in sequence or parallel.

node/node_test.go (1)
  • 20-24: The removal of the deferred node.Cancel() call in cleanUpNode function changes the cleanup behavior. Ensure that the node's resources are properly released and that there are no side effects due to this change, such as resource leaks or incomplete shutdowns.
state/txindex/indexer_service.go (2)
  • 47-60: The change to use is.ctx in the SubscribeUnbuffered calls aligns with the PR's objective to ensure consistent context management across nodes and services. This should allow for proper shutdown handling when the context is canceled.

  • 130-135: The update to use is.ctx in the UnsubscribeAll call within the OnStop method is consistent with the PR's goal of unified context management, which should help in proper resource cleanup during shutdown.

test/context/context.go (1)
  • 10-31: The implementation of WithDebugCancelFunc looks correct and should provide valuable debugging information when the context is canceled. It captures the stack trace of the last 5 calls before the cancel function is invoked and prints them, which can help in tracing the source of premature or unexpected cancellations.

Comment thread test/context/context.go
Comment thread test/context/context.go
Comment thread node/full_node_integration_test.go
Comment thread node/full_node_integration_test.go
Comment thread node/full_node_integration_test.go
Comment thread node/node_test.go
Comment thread node/full_node_integration_test.go
Comment thread node/full_node_integration_test.go
@MSevey

MSevey commented Dec 12, 2023

Copy link
Copy Markdown
Contributor Author

test failure: https://github.com/rollkit/rollkit/actions/runs/7174771927/job/19536842207?pr=1406

=== RUN   TestStatus
panic: 

mock: Unexpected Method Call
-----------------------------

CheckTx(context.todoCtx,*types.RequestCheckTx)
		0: context.todoCtx{emptyCtx:context.emptyCtx{}}
		1: &types.RequestCheckTx{Tx:[]uint8{0x67, 0x6f, 0x6f, 0x64}, Type:1}

The closest call I have is: 

CheckTx(string,*types.RequestCheckTx)
		0: "mock.Anything"
		1: &types.RequestCheckTx{Tx:[]uint8{0x62, 0x61, 0x64}, Type:0}

Difference found in argument 1:

--- Expected
+++ Actual
@@ -1,2 +1,2 @@
-(*types.RequestCheckTx)(tx:"bad" )
+(*types.RequestCheckTx)(tx:"good" type:RECHECK )
 

Diff: 0: PASS:  (context.todoCtx=context.TODO) == (string=mock.Anything)
	1: FAIL:  (*types.RequestCheckTx=tx:"good" type:RECHECK ) != (*types.RequestCheckTx=tx:"bad" )
goroutine 3711 [running]:
github.com/stretchr/testify/mock.(*Mock).fail(0xc0483a18b0, {0x23d7c49, 0x6f}, {0xc026f1f4c0, 0x4, 0x4})
	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/mock/mock.go:332 +0x1b9
github.com/stretchr/testify/mock.(*Mock).MethodCalled(0xc0483a18b0, {0x2ad07ae, 0x7}, {0xc01be59260, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/mock/mock.go:492 +0x69f
github.com/stretchr/testify/mock.(*Mock).Called(0x36ff780?, {0xc01be59260, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/mock/mock.go:464 +0x193
github.com/rollkit/rollkit/test/mocks.(*Application).CheckTx(0xc0483a18b0, {0x2792828?, 0x36ff780}, 0xc01be59240)
	/home/runner/work/rollkit/rollkit/test/mocks/Application.go:45 +0x12a
github.com/cometbft/cometbft/abci/client.(*localClient).CheckTxAsync(0xc015fba0c0, {0x2792828, 0x36ff780}, 0xc01be59240)
	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.0-rc3/abci/client/local_client.go:51 +0x12a
github.com/cometbft/cometbft/proxy.(*appConnMempool).CheckTxAsync(0xc0429acf48, {0x2792828, 0x36ff780}, 0x0?)
	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.0-rc3/proxy/app_conn.go:147 +0x27c
github.com/rollkit/rollkit/mempool.(*CListMempool).recheckTxs(0xc0161f8a90)
	/home/runner/work/rollkit/rollkit/mempool/clist_mempool.go:653 +0x2c4
github.com/rollkit/rollkit/mempool.(*CListMempool).Update(0xc0161f8a90, 0x2, {0x36ff780, 0x0, 0xc0429938f4?}, {0x36ff780, 0x0, 0x1d5264a?}, 0xc0270d53a0, 0xc0270d53b0)
	/home/runner/work/rollkit/rollkit/mempool/clist_mempool.go:626 +0x80f
github.com/rollkit/rollkit/state.(*BlockExecutor).commit(_, {_, _}, {{{0xb, 0x0}, {0x222fbc6, 0xa}}, {0x2201ed4, 0x4}, 0x1, ...}, ...)
	/home/runner/work/rollkit/rollkit/state/executor.go:294 +0x618
github.com/rollkit/rollkit/state.(*BlockExecutor).Commit(_, {_, _}, {{{0xb, 0x0}, {0x222fbc6, 0xa}}, {0x2201ed4, 0x4}, 0x1, ...}, ...)
	/home/runner/work/rollkit/rollkit/state/executor.go:246 +0xab
github.com/rollkit/rollkit/block.(*Manager).trySyncNextBlock(0xc0001b6f00, {0x27929c8, 0xc0536e5ea0}, 0x1)
	/home/runner/work/rollkit/rollkit/block/manager.go:415 +0x4d8
github.com/rollkit/rollkit/block.(*Manager).SyncLoop(0xc0001b6f00, {0x27929c8, 0xc0536e5ea0}, 0x7fe2917e30d0?)
	/home/runner/work/rollkit/rollkit/block/manager.go:355 +0x850
created by github.com/rollkit/rollkit/node.(*FullNode).OnStart in goroutine 3215
	/home/runner/work/rollkit/rollkit/node/full.go:350 +0xdeb
FAIL	github.com/rollkit/rollkit/node	3.846s

@codecov

codecov Bot commented Dec 12, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 24 lines in your changes are missing coverage. Please review.

Comparison is base (2f6ed32) 51.09% compared to head (6ba6b90) 50.79%.

Files Patch % Lines
block/manager.go 0.00% 16 Missing ⚠️
node/full.go 66.66% 2 Missing and 1 partial ⚠️
node/light.go 66.66% 2 Missing and 1 partial ⚠️
node/full_client.go 0.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1406      +/-   ##
==========================================
- Coverage   51.09%   50.79%   -0.30%     
==========================================
  Files          46       46              
  Lines        6422     6448      +26     
==========================================
- Hits         3281     3275       -6     
- Misses       2802     2831      +29     
- Partials      339      342       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Comment thread test/context/context.go
@MSevey
MSevey requested a review from tuxcanfly December 12, 2023 00:38
@Manav-Aggarwal

Copy link
Copy Markdown
Member

test failure: https://github.com/rollkit/rollkit/actions/runs/7174771927/job/19536842207?pr=1406

=== RUN   TestStatus
panic: 

mock: Unexpected Method Call
-----------------------------

CheckTx(context.todoCtx,*types.RequestCheckTx)
		0: context.todoCtx{emptyCtx:context.emptyCtx{}}
		1: &types.RequestCheckTx{Tx:[]uint8{0x67, 0x6f, 0x6f, 0x64}, Type:1}

The closest call I have is: 

CheckTx(string,*types.RequestCheckTx)
		0: "mock.Anything"
		1: &types.RequestCheckTx{Tx:[]uint8{0x62, 0x61, 0x64}, Type:0}

Difference found in argument 1:

--- Expected
+++ Actual
@@ -1,2 +1,2 @@
-(*types.RequestCheckTx)(tx:"bad" )
+(*types.RequestCheckTx)(tx:"good" type:RECHECK )
 

Diff: 0: PASS:  (context.todoCtx=context.TODO) == (string=mock.Anything)
	1: FAIL:  (*types.RequestCheckTx=tx:"good" type:RECHECK ) != (*types.RequestCheckTx=tx:"bad" )
goroutine 3711 [running]:
github.com/stretchr/testify/mock.(*Mock).fail(0xc0483a18b0, {0x23d7c49, 0x6f}, {0xc026f1f4c0, 0x4, 0x4})
	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/mock/mock.go:332 +0x1b9
github.com/stretchr/testify/mock.(*Mock).MethodCalled(0xc0483a18b0, {0x2ad07ae, 0x7}, {0xc01be59260, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/mock/mock.go:492 +0x69f
github.com/stretchr/testify/mock.(*Mock).Called(0x36ff780?, {0xc01be59260, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.4/mock/mock.go:464 +0x193
github.com/rollkit/rollkit/test/mocks.(*Application).CheckTx(0xc0483a18b0, {0x2792828?, 0x36ff780}, 0xc01be59240)
	/home/runner/work/rollkit/rollkit/test/mocks/Application.go:45 +0x12a
github.com/cometbft/cometbft/abci/client.(*localClient).CheckTxAsync(0xc015fba0c0, {0x2792828, 0x36ff780}, 0xc01be59240)
	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.0-rc3/abci/client/local_client.go:51 +0x12a
github.com/cometbft/cometbft/proxy.(*appConnMempool).CheckTxAsync(0xc0429acf48, {0x2792828, 0x36ff780}, 0x0?)
	/home/runner/go/pkg/mod/github.com/cometbft/cometbft@v0.38.0-rc3/proxy/app_conn.go:147 +0x27c
github.com/rollkit/rollkit/mempool.(*CListMempool).recheckTxs(0xc0161f8a90)
	/home/runner/work/rollkit/rollkit/mempool/clist_mempool.go:653 +0x2c4
github.com/rollkit/rollkit/mempool.(*CListMempool).Update(0xc0161f8a90, 0x2, {0x36ff780, 0x0, 0xc0429938f4?}, {0x36ff780, 0x0, 0x1d5264a?}, 0xc0270d53a0, 0xc0270d53b0)
	/home/runner/work/rollkit/rollkit/mempool/clist_mempool.go:626 +0x80f
github.com/rollkit/rollkit/state.(*BlockExecutor).commit(_, {_, _}, {{{0xb, 0x0}, {0x222fbc6, 0xa}}, {0x2201ed4, 0x4}, 0x1, ...}, ...)
	/home/runner/work/rollkit/rollkit/state/executor.go:294 +0x618
github.com/rollkit/rollkit/state.(*BlockExecutor).Commit(_, {_, _}, {{{0xb, 0x0}, {0x222fbc6, 0xa}}, {0x2201ed4, 0x4}, 0x1, ...}, ...)
	/home/runner/work/rollkit/rollkit/state/executor.go:246 +0xab
github.com/rollkit/rollkit/block.(*Manager).trySyncNextBlock(0xc0001b6f00, {0x27929c8, 0xc0536e5ea0}, 0x1)
	/home/runner/work/rollkit/rollkit/block/manager.go:415 +0x4d8
github.com/rollkit/rollkit/block.(*Manager).SyncLoop(0xc0001b6f00, {0x27929c8, 0xc0536e5ea0}, 0x7fe2917e30d0?)
	/home/runner/work/rollkit/rollkit/block/manager.go:355 +0x850
created by github.com/rollkit/rollkit/node.(*FullNode).OnStart in goroutine 3215
	/home/runner/work/rollkit/rollkit/node/full.go:350 +0xdeb
FAIL	github.com/rollkit/rollkit/node	3.846s

This is a known flaky test, can be resolved in a follow up.

@Manav-Aggarwal Manav-Aggarwal left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, passed all tests.

Comment thread node/full_node_integration_test.go
@MSevey
MSevey added this pull request to the merge queue Dec 15, 2023
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Dec 15, 2023
@gupadhyaya
gupadhyaya added this pull request to the merge queue Dec 15, 2023
Merged via the queue into main with commit d7359b0 Dec 15, 2023
@gupadhyaya
gupadhyaya deleted the sevey/node-ctx-fix branch December 15, 2023 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C:ci Component: Continuous Integration T:bug Something isn't working

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

4 participants