Skip to content

Refactor persistence logic; use single file for persistence#2011

Merged
whummer merged 1 commit intomasterfrom
refactor-persistence
Feb 3, 2020
Merged

Refactor persistence logic; use single file for persistence#2011
whummer merged 1 commit intomasterfrom
refactor-persistence

Conversation

@whummer
Copy link
Member

@whummer whummer commented Feb 2, 2020

  • Refactor persistence logic
  • Use single file for persistence
  • Add response to list of arguments for persisting API calls

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.02%) to 50.66% when pulling fb877cc on refactor-persistence into f29c224 on master.

@whummer whummer merged commit 0a73a20 into master Feb 3, 2020
@whummer whummer deleted the refactor-persistence branch February 3, 2020 08:58
@varju
Copy link
Contributor

varju commented Mar 4, 2020

This change was quite the surprise when I upgraded and found all my data gone. Is it worth a mention in the release notes? Or am I doing it wrong, and nobody in their right mind keeps persistent data between releases?

jgbmattos pushed a commit to jgbmattos/localstack that referenced this pull request Mar 10, 2020
For  mine redrive policy to work on lambda. It was necessary or the raise the  lambda_api.process_sns_notification try except or to handle  the exception on sns_listener.
Since the only place that process_sns_notification is called is on sns_listener i decide to remove the try except inside lamda_api

Minor fix on tests.
Created tests for redrive policy in SNS.
Changed the position of try in case of sqs.
For the lambda execution i would appreciate an opinion.
I discover that the function proccess_sns_notification has two possible return, None, ou an HTTP response. I pretend to check if lambda behaviour that way, or not. Because i guess it would be better if it always return an HTTP Response.

resolve CloudFormation attributes starting with lower case (localstack#2008)

refactor persistence logic; use single file for persistence (localstack#2011)

Fix SQS queue creation attributes and specific attribute retrieval (localstack#2005)

Update elasticmq in order to fix sqs tag on creation (localstack#2017)

fix regex for replacement of S3 ETag hashes (localstack#2021)

Add windows support to the MakeFile (localstack#2024)

Refactor CloudFormation dependency resolution (localstack#2026)

fix resolution of CF stack parameters (localstack#2028)

fix CF unit test

Fix SNS tag listing to remove duplicate tags (localstack#2014)

Fix CloudFormation dependency resolution loop; async stack deployment (localstack#2031)

add ExportName to CloudFormation stack outputs (localstack#2033)

Fix CloudFormation support for IAM::Role (localstack#2034)

Remove None strings from SNS results; refactor resolution of CF resource name placeholders (localstack#2036)

fix persistence for ES API (localstack#2040)

Prefix CloudWatch event file names with timestamps (localstack#2035)

Mark Java LocalstackExtension deprecated (localstack#2047)

add persistence for Elasticsearch Service API calls (localstack#2048)

add API to confirm SNS subscriptions (localstack#2043)

fix setting of empty SQS queue attribute values (localstack#2052)

Expose java options for local lambda executors (localstack#2050)

Fix kinesis stream get_cfn_attribute (localstack#2063)

Upgrade testcontainers Maven dependency to version 1.12.5 (localstack#2059)

minor: fix base image and add sasl libs (localstack#2065)

support ExtendedS3DestinationConfiguration in Firehose streams (localstack#2068)

fix RawMessageDelivery subscription values for SNS - SQS integration (localstack#2067)

Check for None before iterating (localstack#2064)

Fix s3 notification event object size (localstack#2069)

Update PYTHONPATH for Python 3.8 (localstack#2070)

support static refs in CloudFormation Fn::Sub strings (localstack#2076)

Fix CloudWatch log streams lambda timstamp format (localstack#2078)

Return ConsumedCapacity for DynamoDB Query action (localstack#2071)

Add basic /health check endpoint (localstack#2080)

move Java sources to separate project (localstack#2084)

update README

Return SQS maxReceiveCount attribute as integer (localstack#2081)

Allow deleting a specific version of an object in S3 (localstack#2087)

update exports on CF stack update (localstack#2097)

fix handler lookup for "provided" Lambda runtime (localstack#2098)

Consider LAMBDA_REMOVE_CONTAINERS config for docker-reuse Lambda executor (localstack#2094)

fix returned attributes on ReturnValues=ALL_OLD for DynamoDB PutItem; Fix CreationTime for CloudFormation stacks (localstack#2103)

Configure nodejs Lambdas to skip SSL verification; add CF support for S3::BucketPolicy (localstack#2104)

Fix SNS subscription confirmation message to include signature details (localstack#2100)

Fix s3 notification event objectsize (localstack#2105)

Fix notifications for s3 uploads made with presigned post requests (localstack#1640)

Optimize plugin loading to speed up boot time (localstack#2109)

fix deployment of EC2 subnets with CidrIpv6 (localstack#2112)

Make Lambda batch size configurable for Kinesis event source mappings (localstack#2110)

Fix creation of SQS tags via CloudFormation (localstack#2114)

Release version 0.10.8; minor fix in DynamoDB ListStreams (localstack#2117)

Fix forwarding of Lambda output into CloudWatch Logs (localstack#2118)

Fix empty response for PutTargets in CloudWatch Events (localstack#2129)

Fix setting of attributes in existing SQS queues (localstack#2130)

integrate S3 starter into multiserver to improve performance (localstack#2132)

Generate default Lambda FunctionName in CloudFormation (localstack#2134)

Fix order of resource checks for CF deployment in single process (localstack#2136)

update elasticmq to 0.15.5 in base Docker image (localstack#2137)

add postgresql-dev to base Docker image

update elasticmq, assert messages with invalid characters are rejected by SQS (localstack#2135)

Support S3 bucket notifications in CF deployments (localstack#2138)

add generated ReceiptHandle for SQS messages forwarded to Lambda (localstack#2139)

Import fix.
@jbking jbking mentioned this pull request Mar 17, 2020
@bastula
Copy link

bastula commented Apr 22, 2020

@varju I see it in the release notes for version 0.10.8. However, just like you I wasn't able to keep my data when I upgraded. I was able to downgrade to the 0.10.7 docker image and my S3 data was back. I wonder if there is a way to force single file data persistence without editing source? I see that there is a flag USE_SINGLE_DUMP_FILE but it's not exposed as an environment variable.

Would maintainers be interested in a patch for this?

@collinsauve
Copy link
Contributor

collinsauve commented Apr 21, 2021

I see that there is a flag USE_SINGLE_DUMP_FILE but it's not exposed as an environment variable.

Would maintainers be interested in a patch for this?

As a user, I know I would!

edit: See PR #3913

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants