Skip to content

Fix for mis-named logstreams created by lambda invocations.#2078

Merged
whummer merged 1 commit intolocalstack:masterfrom
gfranxman:master
Feb 23, 2020
Merged

Fix for mis-named logstreams created by lambda invocations.#2078
whummer merged 1 commit intolocalstack:masterfrom
gfranxman:master

Conversation

@gfranxman
Copy link
Contributor

The CloudWatch log stream names created by lambda invocations were incorrect.
The invocation time is in milliseconds, which wasn't accounted for by the _store_logs method resulting in date prefixes like '52112/01/29'
instead of '2020/02/22'.

So the lambdas thought they ware logging to stream names like:

"2020/02/21/[$LATEST]549d751bde9d222e4917f4ac838986ed"

but were really logging to names like:

"+52111/12/10/[$LATEST]4b3dcbb1"

There will have to be an additional fix because there's still the issue of the short_uuid vs the long uuid.

The invocation time is in milliseconds, which wasn't accounted for
by the _store_logs method resulting in date prefixes like '52112/01/29'
instead of '2020/02/22'.
@gfranxman gfranxman changed the title Fix for mis-named logstreams created by lambda invokations. Fix for mis-named logstreams created by lambda invocations. Feb 22, 2020
@coveralls
Copy link

Coverage Status

Coverage increased (+0.1%) to 50.351% when pulling 0f6e44f on gfranxman:master into e8d7392 on localstack:master.

@gfranxman
Copy link
Contributor Author

It appears the version was hardcoded as well, and I have a fix for that too, but the suffix has been harder to track down.

The closest I've gotten seems to indicate the context.log_stream_name passed into the lambda is derived at invocation time like so (from the Moto package) then thrown away after it is used:

            invoke_id = uuid.uuid4().hex
            log_stream_name = "{date.year}/{date.month:02d}/{date.day:02d}/[{version}]{invoke_id}".format(
                date=datetime.datetime.utcnow(),
                version=self.version,
                invoke_id=invoke_id,
            )

so the localstack code may have no way of knowing what log stream was actually used. :(

@whummer
Copy link
Member

whummer commented Feb 23, 2020

Good catch - thanks for providing this fix @gfranxman !

@whummer whummer merged commit 9008f84 into localstack:master Feb 23, 2020
jgbmattos pushed a commit to jgbmattos/localstack that referenced this pull request Mar 10, 2020
For  mine redrive policy to work on lambda. It was necessary or the raise the  lambda_api.process_sns_notification try except or to handle  the exception on sns_listener.
Since the only place that process_sns_notification is called is on sns_listener i decide to remove the try except inside lamda_api

Minor fix on tests.
Created tests for redrive policy in SNS.
Changed the position of try in case of sqs.
For the lambda execution i would appreciate an opinion.
I discover that the function proccess_sns_notification has two possible return, None, ou an HTTP response. I pretend to check if lambda behaviour that way, or not. Because i guess it would be better if it always return an HTTP Response.

resolve CloudFormation attributes starting with lower case (localstack#2008)

refactor persistence logic; use single file for persistence (localstack#2011)

Fix SQS queue creation attributes and specific attribute retrieval (localstack#2005)

Update elasticmq in order to fix sqs tag on creation (localstack#2017)

fix regex for replacement of S3 ETag hashes (localstack#2021)

Add windows support to the MakeFile (localstack#2024)

Refactor CloudFormation dependency resolution (localstack#2026)

fix resolution of CF stack parameters (localstack#2028)

fix CF unit test

Fix SNS tag listing to remove duplicate tags (localstack#2014)

Fix CloudFormation dependency resolution loop; async stack deployment (localstack#2031)

add ExportName to CloudFormation stack outputs (localstack#2033)

Fix CloudFormation support for IAM::Role (localstack#2034)

Remove None strings from SNS results; refactor resolution of CF resource name placeholders (localstack#2036)

fix persistence for ES API (localstack#2040)

Prefix CloudWatch event file names with timestamps (localstack#2035)

Mark Java LocalstackExtension deprecated (localstack#2047)

add persistence for Elasticsearch Service API calls (localstack#2048)

add API to confirm SNS subscriptions (localstack#2043)

fix setting of empty SQS queue attribute values (localstack#2052)

Expose java options for local lambda executors (localstack#2050)

Fix kinesis stream get_cfn_attribute (localstack#2063)

Upgrade testcontainers Maven dependency to version 1.12.5 (localstack#2059)

minor: fix base image and add sasl libs (localstack#2065)

support ExtendedS3DestinationConfiguration in Firehose streams (localstack#2068)

fix RawMessageDelivery subscription values for SNS - SQS integration (localstack#2067)

Check for None before iterating (localstack#2064)

Fix s3 notification event object size (localstack#2069)

Update PYTHONPATH for Python 3.8 (localstack#2070)

support static refs in CloudFormation Fn::Sub strings (localstack#2076)

Fix CloudWatch log streams lambda timstamp format (localstack#2078)

Return ConsumedCapacity for DynamoDB Query action (localstack#2071)

Add basic /health check endpoint (localstack#2080)

move Java sources to separate project (localstack#2084)

update README

Return SQS maxReceiveCount attribute as integer (localstack#2081)

Allow deleting a specific version of an object in S3 (localstack#2087)

update exports on CF stack update (localstack#2097)

fix handler lookup for "provided" Lambda runtime (localstack#2098)

Consider LAMBDA_REMOVE_CONTAINERS config for docker-reuse Lambda executor (localstack#2094)

fix returned attributes on ReturnValues=ALL_OLD for DynamoDB PutItem; Fix CreationTime for CloudFormation stacks (localstack#2103)

Configure nodejs Lambdas to skip SSL verification; add CF support for S3::BucketPolicy (localstack#2104)

Fix SNS subscription confirmation message to include signature details (localstack#2100)

Fix s3 notification event objectsize (localstack#2105)

Fix notifications for s3 uploads made with presigned post requests (localstack#1640)

Optimize plugin loading to speed up boot time (localstack#2109)

fix deployment of EC2 subnets with CidrIpv6 (localstack#2112)

Make Lambda batch size configurable for Kinesis event source mappings (localstack#2110)

Fix creation of SQS tags via CloudFormation (localstack#2114)

Release version 0.10.8; minor fix in DynamoDB ListStreams (localstack#2117)

Fix forwarding of Lambda output into CloudWatch Logs (localstack#2118)

Fix empty response for PutTargets in CloudWatch Events (localstack#2129)

Fix setting of attributes in existing SQS queues (localstack#2130)

integrate S3 starter into multiserver to improve performance (localstack#2132)

Generate default Lambda FunctionName in CloudFormation (localstack#2134)

Fix order of resource checks for CF deployment in single process (localstack#2136)

update elasticmq to 0.15.5 in base Docker image (localstack#2137)

add postgresql-dev to base Docker image

update elasticmq, assert messages with invalid characters are rejected by SQS (localstack#2135)

Support S3 bucket notifications in CF deployments (localstack#2138)

add generated ReceiptHandle for SQS messages forwarded to Lambda (localstack#2139)

Import fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants