feat(jmespath): new built-in envelopes to unwrap S3 events - #2169
Conversation
Co-authored-by: Ruben Fonseca <fonseka@gmail.com> Signed-off-by: Leandro Damascena <leandro.damascena@gmail.com>
heitorlessa
left a comment
There was a problem hiding this comment.
Looks great!!! THANK YOU for that @leandrodamascena ;)
One necessary change, and two suggestions to make this better.
| | **`SNS`** | `Records[0].Sns.Message | powertools_json(@)` | | ||
| | **`SQS`** | `Records[*].powertools_json(body)` | | ||
|
|
||
| ???+ tip |
There was a problem hiding this comment.
Niiiice!! You can make it slightly better by:
- Adding a title so ESL readers know upfront if they should care or not e.g., "Using SNS?"
- Educate the reader that if they do need SNS event metadata they don't need to enable it
Write, review, cut the unnecessary words without losing its original intent, and ship it <3
There was a problem hiding this comment.
- Added the title
- I thought of the following mental model to improve this text, and see if it makes sense.
- Benefits - Reduce payload size
- How to do - enabling raw message
- why? - architectures that require sending event notifications from S3 to SNS but not require SNS metadata
|
Consider rewriting the PR title so anyone reading later know what this provides; play around with words |
Codecov ReportPatch and project coverage have no change.
Additional details and impacted files@@ Coverage Diff @@
## develop #2169 +/- ##
========================================
Coverage 97.46% 97.46%
========================================
Files 147 147
Lines 6872 6872
Branches 505 505
========================================
Hits 6698 6698
Misses 137 137
Partials 37 37 ☔ View full report in Codecov by Sentry. |
Thank you, @heitorlessa! Can you check again pls? |
Signed-off-by: Heitor Lessa <lessa@amazon.nl>
Issue number: #2168
Summary
Changes
In this pull request, 5 new envelopes were created to make the developer's experience simpler.
User experience
BEFORE
AFTER
Payloads to test
S3->SNS->SQS
{ "Records":[ { "messageId":"deefbac6-344a-4db5-b7dd-35894575e7d4", "receiptHandle":"AQEB0OWCS1/EYMFz92sKdu7g+DdjchivKC4bZ+sltCNq6ZodCecJ4VYtjvqilEP8D/gxbo4sEk8tTFPAZLoCHyat52E+vbPANwFkcAhObAQRmwWX7BRGzs/zFfqCW2nt/DwY2A+ElkYrdkmxRLGjQcCRekJ0hAaZrSWAmnWfO4siquB4lRsxlHL6TzLaYXTQlbQ/K/1NsUZ+Tnb7WSpUKs7GN44RjzJGbvZKK6C2Bg6BRs+S32to5mfIHFjPB4Mu1+8ao9Rt0ChintH/NQO+93lsgzCeLbokhKVqIMAwqgqAVG7l9Jk51hkvMT3zXk4jnwGFX7RfLmtFdU3O4MNizQVucQL6nrHj44Gb4ldvL5Jq6lSjz1BA+j0zwexh3EH4mj+R", "body":"{\n \"Type\" : \"Notification\",\n \"MessageId\" : \"1cd1d74a-f20d-5b05-b96b-8b897458b85b\",\n \"TopicArn\" : \"arn:aws:sns:us-east-1:200984112386:SQS\",\n \"Subject\" : \"Amazon S3 Notification\",\n \"Message\" : \"{\\\"Records\\\":[{\\\"eventVersion\\\":\\\"2.1\\\",\\\"eventSource\\\":\\\"aws:s3\\\",\\\"awsRegion\\\":\\\"us-east-1\\\",\\\"eventTime\\\":\\\"2023-04-26T20:05:39.433Z\\\",\\\"eventName\\\":\\\"ObjectCreated:Put\\\",\\\"userIdentity\\\":{\\\"principalId\\\":\\\"A1YQ72UWCM96UF\\\"},\\\"requestParameters\\\":{\\\"sourceIPAddress\\\":\\\"93.108.161.96\\\"},\\\"responseElements\\\":{\\\"x-amz-request-id\\\":\\\"ZPVZFYFWS102RWZ7\\\",\\\"x-amz-id-2\\\":\\\"AuER6ogIVeGOPF7bGWzi+U441aSBpLFf+fObE3Iq7/EbPXX2tLPOR2V4xL6TeXYPzwDH8Qo4PmX0Aj5Idfm7fIj48lw5FgZJ\\\"},\\\"s3\\\":{\\\"s3SchemaVersion\\\":\\\"1.0\\\",\\\"configurationId\\\":\\\"SNS\\\",\\\"bucket\\\":{\\\"name\\\":\\\"xxx\\\",\\\"ownerIdentity\\\":{\\\"principalId\\\":\\\"A1YQ72UWCM96UF\\\"},\\\"arn\\\":\\\"arn:aws:s3:::xxx\\\"},\\\"object\\\":{\\\"key\\\":\\\"xxx.pdf\\\",\\\"size\\\":103424,\\\"eTag\\\":\\\"cd3ae8d2aca42cb951566a82c615b639\\\",\\\"versionId\\\":\\\"FbCq_E1uOATR01o.H2ewD8KfTlxqoHCm\\\",\\\"sequencer\\\":\\\"0064498413657AEFCF\\\"}}}]}\",\n \"Timestamp\" : \"2023-04-26T20:05:41.101Z\",\n \"SignatureVersion\" : \"1\",\n \"Signature\" : \"RWcOmgOXLll01SCYMGHVL+Y5KtEDmFojlQ3DFYIDeAjr7Y2uQZm90hftD+OcYkLucRg1hEXljJAZnhBT5dTi8M7PVvAm+MEND0T8/s9JHidd1/TR3bB5BBPRkr0qe5D9bVsIZV/dZ1tAbOHPHYT8oBwXYeJ0CjUVKoOKb9TT6asvnx8nxq0V9qlY1b+hGojVtS92VMBXIj4WJRWI1XwSk2WRd+RYxRPuD4b9517SEgCdzdF+vuq/5FV/jAJudw0mT1oHE+hkZ315k5mTO3rIh9xiLQaKVUqTF01e16fnzLz798JYv3sazAydXjRkBhMP+h9Nj8hxadEZWMkPqryroA==\",\n \"SigningCertURL\" : \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem\",\n \"UnsubscribeURL\" : \"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:200984112386:SQS:4a2496a5-7b8b-4039-959a-3d3729b1d790\"\n}", "attributes":{ "ApproximateReceiveCount":"1", "SentTimestamp":"1682539541154", "SenderId":"AIDAIT2UOQQY3AUEKVGXU", "ApproximateFirstReceiveTimestamp":"1682539561154" }, "messageAttributes":{ }, "md5OfBody":"19bc26cd974eae7362ec27e31a9eefc2", "eventSource":"aws:sqs", "eventSourceARN":"arn:aws:sqs:us-east-1:xxx:SQS", "awsRegion":"us-east-1" } ] }S3->SNS(raw delivery message enabled)->SQS
{ "Records":[ { "messageId":"fc1d0d34-a507-4e10-b3e6-b501eb8f200c", "receiptHandle":"AQEBlkznH+C0zHuv6y453tltB30zTv+MpfH5Le3FSI2spABjWQmQXuMKBsE4ps/Fgh5tQ1VDnavknuW/dr5LcdCUc00E0wYF0sm9SiR35PxUs6Ybf1BzcGUxIAVPL+QV1/fBMRA3nvqY5B/gBoHvJX2uKcs9FNid0PtnOb11DQYh7FtkvzjufRXyTs0znLM2t7E1PJtwEghJr5z6yfiOlB1xw4K18av6RWOkG4dWIVzrNnHJKrmAUyHutZ6p+SdCMjucmR3MEEL848FPVUkUDlpg+tTCIQOMxjSr81FQFn0CJLaAFuoStN2+HoqfL+Uu78j1N1uCJHRIHUe0ZPa861nYtm0zcPX3Sr9Gswbku7bTnrkkd3VBgBQPl7t6+6kt1zGR", "body":"{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2023-02-14T16:48:40.076Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"A1YQ72UWCM96UF\"},\"requestParameters\":{\"sourceIPAddress\":\"93.108.161.96\"},\"responseElements\":{\"x-amz-request-id\":\"APCDMT4D1SBRGWJW\",\"x-amz-id-2\":\"dC8zU2g7TFS8p24lRvIgcFBAd1snHSs/1mrZ94oexCqDAzbVigpkUxrwbtzFU2IP0TcHq+8LIgRmFPhohYMn4KuS1DFXKHKSKruegMu42xM=\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"SNS\",\"bucket\":{\"name\":\"xxx\",\"ownerIdentity\":{\"principalId\":\"A1YQ72UWCM96UF\"},\"arn\":\"arn:aws:s3:::xxx\"},\"object\":{\"key\":\"xxx.pdf\",\"size\":683156,\"eTag\":\"01409bb790f925321c619170a4002cd6\",\"versionId\":\"6hIIuveLWIMHVD5qszL7MaDlsOxHOvVM\",\"sequencer\":\"0063EBBB67F02C246E\"}}}]}", "attributes":{ "ApproximateReceiveCount":"1", "SentTimestamp":"1676393321478", "SenderId":"AIDAIT2UOQQY3AUEKVGXU", "ApproximateFirstReceiveTimestamp":"1676393321485" }, "messageAttributes":{ }, "md5OfBody":"c17a39d9858961037e8d748de4c39f9d", "eventSource":"aws:sqs", "eventSourceARN":"arn:aws:sqs:us-east-1:xxx:SQS", "awsRegion":"us-east-1" } ] }S3->SNS->Kinesis Firehose
{ "invocationId":"064dbbd7-d5b9-42b4-8548-e2c87b49ad68", "deliveryStreamArn":"arn:aws:firehose:us-east-1:xxxx:deliverystream/PUT-S3-yltjg", "region":"us-east-1", "records":[ { "recordId":"49640211073817327783046060903306588296500113445104910338000000", "approximateArrivalTimestamp":1682550045572, "data":"eyJUeXBlIjoiTm90aWZpY2F0aW9uIiwiTWVzc2FnZUlkIjoiZGUyOWE1NmItZDhhYy01NDQwLWI4NDctZjlhODc3ZjFiNTFjIiwiVG9waWNBcm4iOiJhcm46YXdzOnNuczp1cy1lYXN0LTE6MjAwOTg0MTEyMzg2OlNRUyIsIlN1YmplY3QiOiJBbWF6b24gUzMgTm90aWZpY2F0aW9uIiwiTWVzc2FnZSI6IntcIlJlY29yZHNcIjpbe1wiZXZlbnRWZXJzaW9uXCI6XCIyLjFcIixcImV2ZW50U291cmNlXCI6XCJhd3M6czNcIixcImF3c1JlZ2lvblwiOlwidXMtZWFzdC0xXCIsXCJldmVudFRpbWVcIjpcIjIwMjMtMDQtMjZUMjM6MDA6NDQuMjYwWlwiLFwiZXZlbnROYW1lXCI6XCJPYmplY3RDcmVhdGVkOlB1dFwiLFwidXNlcklkZW50aXR5XCI6e1wicHJpbmNpcGFsSWRcIjpcIkExWVE3MlVXQ005NlVGXCJ9LFwicmVxdWVzdFBhcmFtZXRlcnNcIjp7XCJzb3VyY2VJUEFkZHJlc3NcIjpcIjkzLjEwOC4xNjEuOTZcIn0sXCJyZXNwb25zZUVsZW1lbnRzXCI6e1wieC1hbXotcmVxdWVzdC1pZFwiOlwiMDBYQ0FWNEpaTVlHMjlEWlwiLFwieC1hbXotaWQtMlwiOlwiUHI2U2U5eGdSaEYvMFY5OFhzZlMvMENCaStxczlubDN5OXhXVW9FOUlBRmtMc0pld3ZVYUUrTGtqSSsvTmg5VTN6cVBXQlZGQ25VWDhOR3NpcTg5RnIzZDJmaUw2amkzXCJ9LFwiczNcIjp7XCJzM1NjaGVtYVZlcnNpb25cIjpcIjEuMFwiLFwiY29uZmlndXJhdGlvbklkXCI6XCJkZGRkXCIsXCJidWNrZXRcIjp7XCJuYW1lXCI6XCJhYWFhYmJjY2NkZGRkXCIsXCJvd25lcklkZW50aXR5XCI6e1wicHJpbmNpcGFsSWRcIjpcIkExWVE3MlVXQ005NlVGXCJ9LFwiYXJuXCI6XCJhcm46YXdzOnMzOjo6YWFhYWJiY2NjZGRkZFwifSxcIm9iamVjdFwiOntcImtleVwiOlwibGVvL2UyZS91dGlscy9fX3B5Y2FjaGVfXy9hdXRoLmNweXRob24tMzkucHljXCIsXCJzaXplXCI6NzMxLFwiZVRhZ1wiOlwiNWMxZGE3ODYzNDYzMGRjNTU2YmI1MDJkNWNmOGRmZGJcIixcInZlcnNpb25JZFwiOlwiWllaaWhMa2g3X2V4V2paT1d3dmhremZscWFBY2w4aTNcIixcInNlcXVlbmNlclwiOlwiMDA2NDQ5QUQxQzM1RjJCOThDXCJ9fX1dfSIsIlRpbWVzdGFtcCI6IjIwMjMtMDQtMjZUMjM6MDA6NDUuNTExWiIsIlVuc3Vic2NyaWJlVVJMIjoiaHR0cHM6Ly9zbnMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1VbnN1YnNjcmliZSZTdWJzY3JpcHRpb25Bcm49YXJuOmF3czpzbnM6dXMtZWFzdC0xOjIwMDk4NDExMjM4NjpTUVM6ZjIwNmI0ZmItNmQ3YS00NmQxLTlhODktNGU2ZjRkOGJjYTNkIn0K" } ] }S3->SNS(raw delivery message enabled)->Kinesis Firehose
{ "invocationId":"2b93ed6a-4f9a-4b9e-80a8-bbaa78167ef9", "deliveryStreamArn":"arn:aws:firehose:us-east-1:200984112386:deliverystream/PUT-S3-yltjg", "region":"us-east-1", "records":[ { "recordId":"49640211073817327783046060904329339539894136249992085506000000", "approximateArrivalTimestamp":1682550723287, "data":"eyJSZWNvcmRzIjpbeyJldmVudFZlcnNpb24iOiIyLjEiLCJldmVudFNvdXJjZSI6ImF3czpzMyIsImF3c1JlZ2lvbiI6InVzLWVhc3QtMSIsImV2ZW50VGltZSI6IjIwMjMtMDQtMjZUMjM6MTI6MDIuNDQzWiIsImV2ZW50TmFtZSI6Ik9iamVjdENyZWF0ZWQ6UHV0IiwidXNlcklkZW50aXR5Ijp7InByaW5jaXBhbElkIjoiQTFZUTcyVVdDTTk2VUYifSwicmVxdWVzdFBhcmFtZXRlcnMiOnsic291cmNlSVBBZGRyZXNzIjoiOTMuMTA4LjE2MS45NiJ9LCJyZXNwb25zZUVsZW1lbnRzIjp7IngtYW16LXJlcXVlc3QtaWQiOiI3MzkyQVo0OE5NSDRKOUVZIiwieC1hbXotaWQtMiI6ImZNeHNZaTFtYjdjZ0ozNDUvNHhqMEdtRzBHd2pDTFhsOXYreHNWOFRWZTNUU3pkZ2JmYm9OQ0Vla1pZNzhhUkhwMU9QN2NoR0VYYzVkUUppa1JpUnVoNzBFYmtnYmgvWSJ9LCJzMyI6eyJzM1NjaGVtYVZlcnNpb24iOiIxLjAiLCJjb25maWd1cmF0aW9uSWQiOiJkZGRkIiwiYnVja2V0Ijp7Im5hbWUiOiJhYWFhYmJjY2NkZGRkIiwib3duZXJJZGVudGl0eSI6eyJwcmluY2lwYWxJZCI6IkExWVE3MlVXQ005NlVGIn0sImFybiI6ImFybjphd3M6czM6OjphYWFhYmJjY2NkZGRkIn0sIm9iamVjdCI6eyJrZXkiOiJsZW8vZTJlL3V0aWxzL2RhdGFfZmV0Y2hlci9fX3B5Y2FjaGVfXy9tZXRyaWNzLmNweXRob24tMzkucHljIiwic2l6ZSI6MjM4NCwiZVRhZyI6IjgwYzM5ZjYzYWM5ZWE1MWRlZjJhMzk4MGM0NDhlNjRiIiwidmVyc2lvbklkIjoiWVB2TlVtUVIuRTVqWVY1eVd1NTQzNXB0OFVySEhMalciLCJzZXF1ZW5jZXIiOiIwMDY0NDlBRkMyNjFGNEYzNTkifX19XX0=" } ] }S3->EventBridge->SQS
{ "Records":[ { "messageId":"8a39d5ec-3a82-4ffb-9c2b-8c1506983f98", "receiptHandle":"AQEBwgzrAtZhxdYYHNQdbUK9bHVQrk/bO/kXqdxuJ8IyfBstaTo2yigXmvuebqxAlbkgYLfvrQZl36JfZ4ZJ/oe4MKRY94H0HfN7K5byxQvGvNtXZ4LMEYuGa1CTL5zgS28jihj/hcfdt6OOHbmnxVlJz07nTGf+QH3wFTAuVEMDE96aEX2YUtP8TtX1c3+UxbaVRWuXWu41aoq6G10G0Nez1XkAUmUFsn5cTrGURNXksIVbccOLdGNG8coXaH1vwqxCz6ExavLXE5UegMoppLty2sSFB8LSzBVdFkDcQ0g4cSkJsrd/GaQwZN5E8fKm0lcIKN+tDNIDAoJE4G16ylAVTtBla+vI1n7tImQpvP4BeLZ0e7dXKPtIC7ArFqKu5ZYQ", "body":"{\"version\":\"0\",\"id\":\"680166b5-82a6-f60a-d99f-71a9d41dbb33\",\"detail-type\":\"Object Created\",\"source\":\"aws.s3\",\"account\":\"200984112386\",\"time\":\"2023-04-26T21:18:05Z\",\"region\":\"us-east-1\",\"resources\":[\"arn:aws:s3:::xxx\"],\"detail\":{\"version\":\"0\",\"bucket\":{\"name\":\"aaaabbcccdddd\"},\"object\":{\"key\":\"xx-xxx.pdf\",\"size\":9468,\"etag\":\"5f9ca96e2ed74b596b3754325c069812\",\"version-id\":\"_S90fg_wD3dgMCVp5xcLdlwEFYtCIVpM\",\"sequencer\":\"006449950D7FBE1F45\"},\"request-id\":\"71F7NXFJ1RA274BQ\",\"requester\":\"200984112386\",\"source-ip-address\":\"93.108.161.96\",\"reason\":\"PutObject\"}}", "attributes":{ "ApproximateReceiveCount":"1", "SentTimestamp":"1682543887329", "SenderId":"AIDAJXNJGGKNS7OSV23OI", "ApproximateFirstReceiveTimestamp":"1682543907329" }, "messageAttributes":{ }, "md5OfMessageAttributes":"None", "md5OfBody":"3bffe7eef50ac2a32e5eb4725e8093d5", "eventSource":"aws:sqs", "eventSourceARN":"arn:aws:sqs:us-east-1:xxx:SNS", "awsRegion":"us-east-1" } ] }Checklist
If your change doesn't seem to apply, please leave them unchecked.
Is this a breaking change?
RFC issue number:
Checklist:
Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.