Skip to content

Make event invoke function update test more predicatable#12444

Draft
joe4dev wants to merge 3 commits intomainfrom
fix-flaky-lambda-test-event-function-update
Draft

Make event invoke function update test more predicatable#12444
joe4dev wants to merge 3 commits intomainfrom
fix-flaky-lambda-test-event-function-update

Conversation

@joe4dev
Copy link
Member

@joe4dev joe4dev commented Mar 26, 2025 •
edited
Loading

Note

This test improvement PR is currently blocked by a Lambda parity issue (pending invokes are aborted instead of gracefully finished upon a function update)

Depends on: #12441

Motivation

The test test_async_invoke_queue_upon_function_update has been flaky in CircleCI (e.g., old example) and locally (assertion error 14 != 15).

Changes

  • Migrate test to new SQS-based lambda notifier infrastructure introduced in Fix flaky lambda test event retry reserved concurrency #12441
    • This allows us to wait until the first invoke is actually being processed rather than using a flaky sleep.
    • This should speeds up the test because the SQS message collection uses long polling
  • Migrate to label-based snapshots for easier snapshot skip

TODO

  • Fix buggy LocalStack behavior not respecting the SQS message deletion and therefore causing a snapshot key issue (guarded via assert). It seems that the SQS message does not get deleted properly although the delete_message call was successful 🤔 . The same test passes against AWS.
    • Increasing the visibility timeout didn't help sqs_create_queue(Attributes={"VisibilityTimeout": "90"}) (the default 20s should be ok as long as the cold start doesn't take more than 10s)
    • The sleep-01 key appears multiple times because the initial invoke is aborted with the update instead of continuted and then the retry triggers another invoke. To fix this, Lambda should not abort pending invokes upon function update.

@joe4dev joe4dev added the semver: patch Non-breaking changes which can be included in patch releases label Mar 26, 2025
@joe4dev joe4dev added this to the 4.4 milestone Mar 26, 2025
@joe4dev joe4dev self-assigned this Mar 26, 2025
@github-actions
Copy link

github-actions bot commented Mar 26, 2025 •
edited
Loading

LocalStack Community integration with Pro

    2 files  ±    0      2 suites  ±0   1h 30m 14s ⏱️ - 21m 13s
3 144 tests  - 1 162  2 925 ✅  - 1 061  218 💤  - 102  1 ❌ +1 
3 146 runs   - 1 162  2 925 ✅  - 1 061  220 💤  - 102  1 ❌ +1 

For more details on these failures, see this check.

Results for commit 00d944c. ± Comparison against base commit ccddefd.

This pull request removes 1162 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

♻️ This comment has been updated with latest results.

Base automatically changed from fix-flaky-lambda-test-event-retry-reserved-concurrency to master March 28, 2025 15:48
@joe4dev joe4dev force-pushed the fix-flaky-lambda-test-event-function-update branch from c0ced8a to 00d944c Compare March 28, 2025 16:06
@joe4dev joe4dev modified the milestones: 4.4, Playground May 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dominikschubert dominikschubert Awaiting requested review from dominikschubert dominikschubert will be requested when the pull request is marked ready for review dominikschubert is a code owner

@dfangl dfangl Awaiting requested review from dfangl dfangl will be requested when the pull request is marked ready for review dfangl is a code owner

@gregfurman gregfurman Awaiting requested review from gregfurman gregfurman will be requested when the pull request is marked ready for review gregfurman is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@joe4dev joe4dev

Labels

semver: patch Non-breaking changes which can be included in patch releases

Projects

None yet

Milestone

Playground

Development

Successfully merging this pull request may close these issues.

1 participant

@joe4dev