CloudFormation: Store resource scaffolding in 'generated' directory. #13534

peter-smith-phd · 2025-12-16T21:07:15Z

Motivation

Generate CloudFormation resource scaffolding into a service's resource_providers/generated directory to separate auto-generated code from hand-written code. Resource provider classes should inherit from their auto-generated base class, permitting automatic updates of the base class and associated data types, without overwriting any hand-written code.

Changes

CloudFormation scaffolding script has been updated to:
- Introduce new generated/<resource>_base.py file that contains auto-generated type definitions, as well as abstract CRUD methods. These can be re-generated at any time, without overwriting hand-written code.
- Also generate the existing *_plugin.py and *.schema.json files into the generated subdirectory.
- Generate a default <resource>.py file that inherits types/methods from <resource>_base.py. This file is expected to be modified by hand and contain the full provider implementation.
Update the exist AWS::SQS::QueuePolicy provider to demonstrate that this new split works correctly (other resource providers for AWS::SQS will be updated in a later PR).

Tests

Tested by:

Migrating the existing AWS::SQS::QueuePolicy to use this new structure (included in this PR)
Testing with a Pro resource type, and visually inspecting the generated files.

Test Results - Preflight, Unit

22 982 tests +1 21 140 ✅ +1 6m 30s ⏱️ +16s
1 suites ±0 1 842 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit dc934b6. ± Comparison against base commit ee0d438.

♻️ This comment has been updated with latest results.

…o separate auto-generated code from human-written code.

github-actions · 2025-12-16T21:32:43Z

Test Results (amd64) - Acceptance

7 tests ±0 5 ✅ ±0 3m 4s ⏱️ -12s
1 suites ±0 2 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit dc934b6. ± Comparison against base commit ee0d438.

♻️ This comment has been updated with latest results.

github-actions · 2025-12-16T21:48:53Z

Test Results - Alternative Providers

583 tests - 885 325 ✅ - 562 18m 24s ⏱️ - 14m 32s
1 suites - 4 258 💤 - 323
1 files - 4 0 ❌ ± 0

Results for commit dc934b6. ± Comparison against base commit ee0d438.

This pull request removes 885 tests.

tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_basic_operations_multiple_protocols[json]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_basic_operations_multiple_protocols[query]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_basic_operations_multiple_protocols[smithy-rpc-v2-cbor]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_exception_serializing_with_no_shape_in_spec[json]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_exception_serializing_with_no_shape_in_spec[query]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_exception_serializing_with_no_shape_in_spec[smithy-rpc-v2-cbor]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_multi_protocol_client_fixture[json]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_multi_protocol_client_fixture[query]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudWatchMultiProtocol ‑ test_multi_protocol_client_fixture[smithy-rpc-v2-cbor]
tests.aws.services.cloudwatch.test_cloudwatch.TestCloudwatch ‑ test_alarm_lambda_target
…

♻️ This comment has been updated with latest results.

github-actions · 2025-12-16T22:01:07Z

Test Results (amd64) - Integration, Bootstrap

5 files ± 0 5 suites ±0 2h 4m 35s ⏱️ - 30m 58s
4 132 tests - 1 422 3 827 ✅ - 1 170 305 💤 - 251 0 ❌ - 1
4 138 runs - 1 422 3 827 ✅ - 1 170 311 💤 - 251 0 ❌ - 1

Results for commit dc934b6. ± Comparison against base commit ee0d438.

This pull request removes 1423 and adds 1 tests. Note that renamed tests count towards both.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

tests.aws.services.sns.test_sns.TestSNSTopicCrudV2 ‑ test_data_protection_policy_crud

This pull request removes 252 skipped tests and adds 1 skipped test. Note that renamed tests count towards both.

tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input4-FAILED]
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_deployed_infra_state
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_populate_data
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_user_clicks_are_stored
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_api_exceptions
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_create_exceptions
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_create_invalid_desiredstate
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_double_create_with_client_token
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_lifecycle
…

tests.aws.services.sns.test_sns.TestSNSTopicCrudV2 ‑ test_data_protection_policy_crud

♻️ This comment has been updated with latest results.

github-actions · 2025-12-16T22:10:17Z

LocalStack Community integration with Pro

2 files ± 0 2 suites ±0 1h 33m 13s ⏱️ - 23m 27s
4 108 tests - 1 044 3 799 ✅ - 957 309 💤 - 87 0 ❌ ±0
4 110 runs - 1 044 3 799 ✅ - 957 311 💤 - 87 0 ❌ ±0

Results for commit dc934b6. ± Comparison against base commit ee0d438.

This pull request removes 1045 and adds 1 tests. Note that renamed tests count towards both.

tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

tests.aws.services.sns.test_sns.TestSNSTopicCrudV2 ‑ test_data_protection_policy_crud

This pull request removes 88 skipped tests and adds 1 skipped test. Note that renamed tests count towards both.

tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input4-FAILED]
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_deployed_infra_state
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_populate_data
tests.aws.scenario.mythical_mysfits.test_mythical_misfits.TestMythicalMisfitsScenario ‑ test_user_clicks_are_stored
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_api_exceptions
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_create_exceptions
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_create_invalid_desiredstate
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_double_create_with_client_token
tests.aws.services.cloudcontrol.test_cloudcontrol_api.TestCloudControlResourceApi ‑ test_lifecycle
…

tests.aws.services.sns.test_sns.TestSNSTopicCrudV2 ‑ test_data_protection_policy_crud

♻️ This comment has been updated with latest results.

simonrw

I am hugely on board with this change, thank you! I have raised an issue about hiding the docstrings which I think will help developers build a conforming implementation (until we enforce an engine -> resource provider contract at least). I'd like to open a discussion about that.

I tried this out while implementing AWS::RDS::DBProxyEndpoint for a feature request. Thank you for making this change backwards compatible! It means that we can migrate the resource providers gradually! I really appreciate this.

I found that the files needed formatting. Can we maybe run make format-modified (or similar) after rendering the templates? That would make the initial experience much nicer.

simonrw · 2025-12-17T10:34:19Z

...lstack-core/localstack/services/cloudformation/scaffolding/templates/provider_template.py.j2

        self,
        request: ResourceRequest[{{ resource }}Properties],
    ) -> ProgressEvent[{{ resource }}Properties]:
-        """


issue: I don't like removing these docstrings to the base model. I realise that their content is autogenerated, and will get out of sync with the generated code but the docstrings are a great reminder of what

properties the model needs to set (writeOnlyProperties)

potential operations the method may need to invoke (through required iam actions)

I can see the docstring if I hover the method in PyCharm, but for people who have not written many resource providers it's not intuitive that this docstring is present. I bet the wouldn't look in the generated subdir anyway. To that end, relying on the schema.json file is also not a good solution.

simonrw · 2025-12-17T10:51:21Z

...k-core/localstack/services/cloudformation/scaffolding/templates/provider_base_template.py.j2

+        {%- endif %}
+        """
+        raise NotImplementedError
+


suggestion: can we add the list method (from CloudControl) into the scaffolding? And make sure it's overridden in the subclass?

Something like:

Suggested change

@abstractmethod

def list(

self,

request: ResourceRequest[{{ resource }}Properties],

) -> ProgressEvent[{{ resource }}Properties]:

"""

List available resources of this type

{% if list_permissions -%}

IAM permissions required:

{%- for permission in list_permissions %}

- {{ permission }}

{%- endfor %}

{%- endif %}

"""

raise NotImplementedError

peter-smith-phd · 2025-12-17T21:05:24Z

Thanks @simonrw , I've made your suggested changes.

peter-smith-phd requested review from baermat, dominikschubert, gregfurman, pinzon, simonrw and thrau as code owners December 16, 2025 21:07

peter-smith-phd added semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes aws:cloudformation AWS CloudFormation labels Dec 16, 2025

CloudFormation: Store resource scaffolding in 'generated' directory t…

d8931e2

…o separate auto-generated code from human-written code.

peter-smith-phd force-pushed the sqs/make-cfn-gold branch from 0713e38 to d8931e2 Compare December 16, 2025 21:13

simonrw requested changes Dec 17, 2025

View reviewed changes

simonrw reviewed Dec 17, 2025

View reviewed changes

Updates, as recommend in PR review.

dc934b6

thrau removed their request for review December 18, 2025 14:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

CloudFormation: Store resource scaffolding in 'generated' directory. #13534

CloudFormation: Store resource scaffolding in 'generated' directory. #13534

Uh oh!

peter-smith-phd commented Dec 16, 2025

Uh oh!

github-actions bot commented Dec 16, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 16, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 16, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 16, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 16, 2025 •

edited

Loading

Uh oh!

simonrw left a comment •

edited

Loading

Uh oh!

simonrw Dec 17, 2025

Uh oh!

simonrw Dec 17, 2025

Uh oh!

peter-smith-phd commented Dec 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

+    @abstractmethod
+    def list(
+        self,
+        request: ResourceRequest[{{ resource }}Properties],
+    ) -> ProgressEvent[{{ resource }}Properties]:
+        """
+        List available resources of this type
+        {% if list_permissions -%}
+        IAM permissions required:
+        {%- for permission in list_permissions %}
+          - {{ permission }}
+        {%- endfor %}
+        {%- endif %}
+        """
+        raise NotImplementedError

Uh oh!

CloudFormation: Store resource scaffolding in 'generated' directory. #13534

Are you sure you want to change the base?

CloudFormation: Store resource scaffolding in 'generated' directory. #13534

Uh oh!

Conversation

peter-smith-phd commented Dec 16, 2025

Motivation

Changes

Tests

Related

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Preflight, Unit

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Acceptance

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results - Alternative Providers

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results (amd64) - Integration, Bootstrap

Uh oh!

github-actions bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LocalStack Community integration with Pro

Uh oh!

simonrw left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonrw Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

simonrw Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

peter-smith-phd commented Dec 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Dec 16, 2025 •

edited

Loading

github-actions bot commented Dec 16, 2025 •

edited

Loading

github-actions bot commented Dec 16, 2025 •

edited

Loading

github-actions bot commented Dec 16, 2025 •

edited

Loading

github-actions bot commented Dec 16, 2025 •

edited

Loading

simonrw left a comment •

edited

Loading