Skip to content

Make Lambda batch size configurable for Kinesis event source mappings#2110

Merged
whummer merged 4 commits intolocalstack:masterfrom
lsdev693:fix-lambda_event_source_mapping_from_kinesis
Mar 1, 2020
Merged

Make Lambda batch size configurable for Kinesis event source mappings#2110
whummer merged 4 commits intolocalstack:masterfrom
lsdev693:fix-lambda_event_source_mapping_from_kinesis

Conversation

@lsdev693
Copy link
Contributor

@lsdev693 lsdev693 commented Feb 29, 2020

  • Set default batch size event source = 10
  • Create multiple events according to the batch size

Fixes #1131

duongpv7 added 3 commits March 1, 2020 00:51
* Create multiple events according to the batch size
* Fix bug integration tests
* Fix long-line issue with lint
@coveralls
Copy link

coveralls commented Feb 29, 2020

Coverage Status

Coverage increased (+0.06%) to 50.401% when pulling b43c2d0 on lsdev693:fix-lambda_event_source_mapping_from_kinesis into 0a6e57a on localstack:master.

* Add default batch size
@whummer whummer changed the title Fix issue #1131 Make Lambda events batch size configurable Mar 1, 2020
@whummer whummer changed the title Make Lambda events batch size configurable Make Lambda batch size configurable for Kinesis event source mappings Mar 1, 2020
@whummer whummer merged commit a7f9ae2 into localstack:master Mar 1, 2020
jgbmattos pushed a commit to jgbmattos/localstack that referenced this pull request Mar 10, 2020
For  mine redrive policy to work on lambda. It was necessary or the raise the  lambda_api.process_sns_notification try except or to handle  the exception on sns_listener.
Since the only place that process_sns_notification is called is on sns_listener i decide to remove the try except inside lamda_api

Minor fix on tests.
Created tests for redrive policy in SNS.
Changed the position of try in case of sqs.
For the lambda execution i would appreciate an opinion.
I discover that the function proccess_sns_notification has two possible return, None, ou an HTTP response. I pretend to check if lambda behaviour that way, or not. Because i guess it would be better if it always return an HTTP Response.

resolve CloudFormation attributes starting with lower case (localstack#2008)

refactor persistence logic; use single file for persistence (localstack#2011)

Fix SQS queue creation attributes and specific attribute retrieval (localstack#2005)

Update elasticmq in order to fix sqs tag on creation (localstack#2017)

fix regex for replacement of S3 ETag hashes (localstack#2021)

Add windows support to the MakeFile (localstack#2024)

Refactor CloudFormation dependency resolution (localstack#2026)

fix resolution of CF stack parameters (localstack#2028)

fix CF unit test

Fix SNS tag listing to remove duplicate tags (localstack#2014)

Fix CloudFormation dependency resolution loop; async stack deployment (localstack#2031)

add ExportName to CloudFormation stack outputs (localstack#2033)

Fix CloudFormation support for IAM::Role (localstack#2034)

Remove None strings from SNS results; refactor resolution of CF resource name placeholders (localstack#2036)

fix persistence for ES API (localstack#2040)

Prefix CloudWatch event file names with timestamps (localstack#2035)

Mark Java LocalstackExtension deprecated (localstack#2047)

add persistence for Elasticsearch Service API calls (localstack#2048)

add API to confirm SNS subscriptions (localstack#2043)

fix setting of empty SQS queue attribute values (localstack#2052)

Expose java options for local lambda executors (localstack#2050)

Fix kinesis stream get_cfn_attribute (localstack#2063)

Upgrade testcontainers Maven dependency to version 1.12.5 (localstack#2059)

minor: fix base image and add sasl libs (localstack#2065)

support ExtendedS3DestinationConfiguration in Firehose streams (localstack#2068)

fix RawMessageDelivery subscription values for SNS - SQS integration (localstack#2067)

Check for None before iterating (localstack#2064)

Fix s3 notification event object size (localstack#2069)

Update PYTHONPATH for Python 3.8 (localstack#2070)

support static refs in CloudFormation Fn::Sub strings (localstack#2076)

Fix CloudWatch log streams lambda timstamp format (localstack#2078)

Return ConsumedCapacity for DynamoDB Query action (localstack#2071)

Add basic /health check endpoint (localstack#2080)

move Java sources to separate project (localstack#2084)

update README

Return SQS maxReceiveCount attribute as integer (localstack#2081)

Allow deleting a specific version of an object in S3 (localstack#2087)

update exports on CF stack update (localstack#2097)

fix handler lookup for "provided" Lambda runtime (localstack#2098)

Consider LAMBDA_REMOVE_CONTAINERS config for docker-reuse Lambda executor (localstack#2094)

fix returned attributes on ReturnValues=ALL_OLD for DynamoDB PutItem; Fix CreationTime for CloudFormation stacks (localstack#2103)

Configure nodejs Lambdas to skip SSL verification; add CF support for S3::BucketPolicy (localstack#2104)

Fix SNS subscription confirmation message to include signature details (localstack#2100)

Fix s3 notification event objectsize (localstack#2105)

Fix notifications for s3 uploads made with presigned post requests (localstack#1640)

Optimize plugin loading to speed up boot time (localstack#2109)

fix deployment of EC2 subnets with CidrIpv6 (localstack#2112)

Make Lambda batch size configurable for Kinesis event source mappings (localstack#2110)

Fix creation of SQS tags via CloudFormation (localstack#2114)

Release version 0.10.8; minor fix in DynamoDB ListStreams (localstack#2117)

Fix forwarding of Lambda output into CloudWatch Logs (localstack#2118)

Fix empty response for PutTargets in CloudWatch Events (localstack#2129)

Fix setting of attributes in existing SQS queues (localstack#2130)

integrate S3 starter into multiserver to improve performance (localstack#2132)

Generate default Lambda FunctionName in CloudFormation (localstack#2134)

Fix order of resource checks for CF deployment in single process (localstack#2136)

update elasticmq to 0.15.5 in base Docker image (localstack#2137)

add postgresql-dev to base Docker image

update elasticmq, assert messages with invalid characters are rejected by SQS (localstack#2135)

Support S3 bucket notifications in CF deployments (localstack#2138)

add generated ReceiptHandle for SQS messages forwarded to Lambda (localstack#2139)

Import fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lambda Event Source Mapping From Kinesis Does Not Always Honor Batch Size

4 participants