Skip to content

Commit 27ae3a6

Browse files
mattmakaimattmakai
committed
add new resources and latest blog post
1 parent be01f80 commit 27ae3a6

26 files changed

+357
-2
lines changed

‎content/pages/03-data/04-sqlite.markdown‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,10 @@ tutorials will help you get started.
8686
It's a great short read which shows that the code is well-tested and
8787
maintained.
8888

89+
* [SQLite is not a toy database](https://antonz.org/sqlite-is-not-a-toy-database/)
90+
is a whirlwind overview of some of the best aspects of SQLite and why
91+
you should use it.
92+
8993
* [Data Analysis of 8.2 Million Rows with Python and SQLite](https://plot.ly/ipython-notebooks/big-data-analytics-with-pandas-and-sqlite/)
9094
explains how you can load a large dataset in to SQLite and visualize it
9195
using the Plotly service.
@@ -139,6 +143,9 @@ you are having with SQLite rather than going through a general tutorial.
139143
digs into the internals of SQLite and shows some bugs found (and
140144
since fixed) while the author was researching the SQLite source code.
141145

146+
* [How to Store Multimedia Files in a SQLite3 Database with Python](https://www.twilio.com/blog/intro-multimedia-file-upload-python-sqlite3-database)
147+
goes through the Python code for storing and accessing BLOB-type objects.
148+
142149
* [Going Fast with SQLite and Python](http://charlesleifer.com/blog/going-fast-with-sqlite-and-python/)
143150
shares essential knowledge for working effectively with SQLite in Python,
144151
particularly when it comes to transactions, concurrency and commits.

‎content/pages/10-working/01-event-streams.markdown‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ meta: An event stream is a log of one or more events.
99

1010
Event streams are a log of one or more "things that happen", which are
1111
usually referred to as events. Event streams are
12-
conceptually focused around events than "things" or objects, which are
12+
conceptually focused around events than objects or tables, which are
1313
the typical storage unit of [relational databases](/databases.html).
1414

1515
Apache Kafka and Gazette are a popular open source implementations of event

‎content/pages/meta/00-change-log.markdown‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,10 @@ view commit-level changes via the
1414
on GitHub.
1515

1616
## 2021
17+
### April
18+
* Published new post on [How to Monitor Python Functions on AWS Lambda with Sentry](/blog/monitor-python-functions-aws-lambda-sentry.html).
19+
* New [SQLite](/sqlite.html) resource.
20+
1721
### March
1822
* New [webhook](/webhooks.html) resource.
1923

‎content/posts/170428-python-2-7-aws-lambda.markdown‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ slug: aws-lambda-python-2-7
33
meta: Learn how to create and deploy your first Amazon Web Services (AWS) Lambda function with Python 2.7.
44
category: post
55
date: 2017-04-28
6-
modified: 2017-04-29
6+
modified: 2021-03-30
77
newsletter: False
88
headerimage: /img/170428-aws-lambda-python-2-7/header.jpg
99
headeralt: AWS, AWS Lambda and Python logos, copyright their respective owners.
@@ -22,6 +22,11 @@ function that executes some simple Python 2.7 code and handles environment
2222
variables. The code can then be modified to build far more complicated
2323
Python applications.
2424

25+
*Note*: AWS
26+
[ended support for Python 2.7 Lambda functions in 2021](https://aws.amazon.com/blogs/compute/announcing-end-of-support-for-python-2-7-in-aws-lambda/)
27+
and Python 2.7 no longer receives support so you should really be using
28+
[Python 3.8 or above](https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-now-supports-python-3-8/).
29+
2530

2631
## Tools We Need
2732
We do not need any local development environment tools to get through

‎content/posts/210422-monitor-python-aws-lambda-sentry.markdown‎

Lines changed: 333 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,333 @@
1+
title: How to Monitor Python Functions on AWS Lambda with Sentry
2+
slug: monitor-python-functions-aws-lambda-sentry
3+
meta: Learn how to monitor your Python 3 functions on AWS Lambda using Sentry.
4+
category: post
5+
date: 2021-04-22
6+
modified: 2021-04-23
7+
newsletter: False
8+
headerimage: /img/headers/python-lambda-sentry.jpg
9+
headeralt: The Python, AWS Lambda and Sentry logos are copyright their respective owners.
10+
11+
12+
[Amazon Web Services (AWS) Lambda](/aws-lambda.html) is a usage-based
13+
compute service that can run [Python 3](/why-use-python.html) code. Errors
14+
can happen in any environment you are running your application in, so
15+
it is necessary to have reliable [monitoring](/monitoring.html) in place
16+
to have visibility when a problem occurs.
17+
18+
In this post we will install and configure
19+
[Sentry](https://sentry.io/welcome/)'s application monitoring
20+
service that works specifically for code running on AWS Lambda.
21+
22+
23+
## Application Dependencies
24+
A local [development environment](/development-environments.html) is not
25+
required to follow this tutorial because all of the coding and configuration
26+
can happen in a web browser through the
27+
[AWS Console](https://console.aws.amazon.com/console/).
28+
29+
The example code can be copy and pasted from this blog post or you
30+
can access it on GitHub under the
31+
[Full Stack Python blog-post-examples](https://github.com/fullstackpython/blog-code-examples)
32+
repository within the
33+
[monitor-python-aws-lambda-sentry directory](https://github.com/fullstackpython/blog-code-examples/tree/master/monitor-python-aws-lambda-sentry).
34+
35+
36+
## Accessing the AWS Lambda Service
37+
[Sign into your existing AWS account](https://aws.amazon.com/console)
38+
or sign up for a [new account](https://aws.amazon.com/). Lambda
39+
gives you the first 1 million requests for free so that you can execute
40+
basic applications without no or low cost.
41+
42+
<img src="/img/210406-python-sentry-aws-lambda/aws-lambda-landing.jpg" width="100%" class="shot rnd outl" alt="The AWS Lambda landing page.">
43+
44+
When you log into your account, use the search box to enter
45+
"lambda" and select "Lambda" when it appears to get to the right
46+
page.
47+
48+
<img src="/img/210406-python-sentry-aws-lambda/lambda-search-bar.png" width="100%" class="shot rnd outl" alt="Use the search bar to find AWS Lambda.">
49+
50+
If you have already used Lambda before, you will see your existing Lambda
51+
functions in a searchable table. We're going to create a new function so
52+
click the "Create function" button.
53+
54+
<img src="/img/210406-python-sentry-aws-lambda/create-function.png" width="100%" class="shot rnd outl" alt="Click the create function button.">
55+
56+
The create function page will give you several options for starting a new
57+
Lambda function.
58+
59+
<img src="/img/210406-python-sentry-aws-lambda/create-function-detail.png" width="100%" class="shot rnd outl" alt="The create function details page.">
60+
61+
Click the "Browse Serverless App Repository" selection box, then choose
62+
the "hello-world-python3" starter app from within the
63+
"Public applications" section.
64+
65+
<img src="/img/210406-python-sentry-aws-lambda/create-function-detail.png" width="100%" class="shot rnd outl" alt="The create function details page.">
66+
67+
The hello-world-python3 starter app details page should look something
68+
like the following screen:
69+
70+
<img src="/img/210406-python-sentry-aws-lambda/hello-world-python3.png" width="100%" class="shot rnd outl" alt="Hello world Python3 example app and Lambda function.">
71+
72+
Fill in some example text such as "test" under `IdentityNameParameter`
73+
and click the "Deploy" button:
74+
75+
<img src="/img/210406-python-sentry-aws-lambda/deploy-starter-app.png" width="100%" class="shot rnd outl" alt="Click the deploy button to use the starter app.">
76+
77+
The function will now be deployed. As soon as it is ready we can
78+
customize it and test it out before adding Sentry to capture any errors
79+
that occur during execution.
80+
81+
82+
## Testing the starter Python app
83+
Go back to the Lambda functions main page and select your new deployed
84+
starter app from the list.
85+
86+
<img src="/img/210406-python-sentry-aws-lambda/functions-list.jpg" width="100%" class="shot rnd outl" alt="List of AWS Lambda functions you have created.">
87+
88+
Find the orange "Test" button with a down arrow next to it like you
89+
see in the image below, and then click the down arrow. Select
90+
"Configure Test Event".
91+
92+
<img src="/img/210406-python-sentry-aws-lambda/configure-test.jpg" width="100%" class="shot rnd outl" alt="Configure the test event.">
93+
94+
Fill in the Event name as "FirstTest" or something similar, then
95+
press the "Create" button at the bottom of the modal window.
96+
97+
Click the "Test" button and it will run the Lambda function with
98+
the parameters from that new test event. You should see something
99+
like the following output:
100+
101+
```python
102+
Response
103+
"value1"
104+
105+
Function Logs
106+
START RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Version: $LATEST
107+
value1 = value1
108+
value2 = value2
109+
value3 = value3
110+
END RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914
111+
REPORT RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Duration: 0.30 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 43 MB Init Duration: 1.34 ms
112+
113+
Request ID
114+
62fa2f25-669c-47b7-b4e7-47353b0bd914
115+
```
116+
117+
That means the test case was successful, but what happens even if there
118+
is a straightforward mistake in the code, such as trying to access an
119+
undeclared variable?
120+
121+
Go into the code editor and you should see the starter code like this:
122+
123+
<img src="/img/210406-python-sentry-aws-lambda/lambda-code-editor.jpg" width="100%" class="shot rnd outl" alt="Code editor within AWS Lambda.">
124+
125+
Update the code with the new highlighted line, which tries to access
126+
a fourth variable, which does not exist in the test configuration
127+
we try to run it with.
128+
129+
```python
130+
import json
131+
132+
print('Loading function')
133+
134+
135+
def lambda_handler(event, context):
136+
#print("Received event: " + json.dumps(event, indent=2))
137+
print("value1 = " + event['key1'])
138+
print("value2 = " + event['key2'])
139+
print("value3 = " + event['key3'])
140+
~~ print("value4 = " + event['key4'])
141+
return event['key1'] # Echo back the first key value
142+
#raise Exception('Something went wrong')
143+
```
144+
145+
After adding that one new line of code, hit the "Deploy" button,
146+
then the "Test" button. You should see some error output:
147+
148+
```
149+
Response
150+
{
151+
"errorMessage": "'key4'",
152+
"errorType": "KeyError",
153+
"stackTrace": [
154+
[
155+
"/var/task/lambda_function.py",
156+
11,
157+
"lambda_handler",
158+
"print(\"value4 = \" + event['key4'])"
159+
]
160+
]
161+
}
162+
163+
Function Logs
164+
START RequestId: a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb Version: $LATEST
165+
value1 = value1
166+
value2 = value2
167+
value3 = value3
168+
'key4': KeyError
169+
Traceback (most recent call last):
170+
File "/var/task/lambda_function.py", line 11, in lambda_handler
171+
print("value4 = " + event['key4'])
172+
KeyError: 'key4'
173+
174+
END RequestId: a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb
175+
REPORT RequestId: a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb Duration: 0.81 ms Billed Duration: 1 ms Memory Size: 128 MB Max Memory Used: 43 MB Init Duration: 1.61 ms
176+
177+
Request ID
178+
a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb
179+
```
180+
181+
It is obvious when we are working in the Console that an error just
182+
occurred. However, in most cases an error will happen sporadically
183+
which is why we need a monitoring system in place to catch and report
184+
on those exceptions.
185+
186+
187+
## AWS Lambda function monitoring with Sentry
188+
The easiest way to add Sentry to Lambda for this application
189+
is to configure an
190+
[AWS Lambda Layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html)
191+
with the necessary dependency for Sentry. Sentry has concise
192+
[documentation on addin gvia Lambda Layers](https://docs.sentry.io/platforms/python/guides/aws-lambda/layer/)
193+
so we will walk through that way to configure it and test it
194+
out.
195+
196+
First, scroll down to the "Layers" section while in your Lambda
197+
function configuration. Click the "Add a layer" button":
198+
199+
<img src="/img/210406-python-sentry-aws-lambda/add-lambda-layer.png" width="100%" class="shot rnd outl" alt="Add Lambda layer.">
200+
201+
In the "Add layer" screen, select the "Specify an ARN" option.
202+
203+
<img src="/img/210406-python-sentry-aws-lambda/add-layer-specify-arn.jpg" width="100%" class="shot rnd outl" alt="Select Specify ARN in the Add Layer screen.">
204+
205+
Now to specify the Amazon Resource Name (ARN), we need to use
206+
the Sentry documentation to get the right configuration string.
207+
208+
US-East-1 is the oldest and most commonly-used region so I'll
209+
use that here in this tutorial but you should check which one
210+
you are in if you are not certain.
211+
212+
<img src="/img/210406-python-sentry-aws-lambda/arn-region.png" width="100%" class="shot rnd outl" alt="Select the AWS for the ARN string.">
213+
214+
Copy that value into the Lambda Layer configuration, like this:
215+
216+
<img src="/img/210406-python-sentry-aws-lambda/layer-with-arn.png" width="100%" class="shot rnd outl" alt="Select the AWS for the ARN string.">
217+
218+
Then press the "Add" button. Now you have the Sentry dependency
219+
in your environment so code that relies upon that library can be
220+
used in the Lambda function.
221+
222+
Next we need to go into the Sentry dashboard to create a project,
223+
get our unique identifer, and connect it to our Lambda function.
224+
225+
Sentry can be [self-hosted](https://github.com/getsentry/onpremise) or
226+
used as a cloud service through [Sentry.io](https://sentry.io). We will
227+
use the cloud hosted version because it is quicker than
228+
setting up your own server as well as free for smaller projects.
229+
230+
Go to [Sentry.io's homepage](https://sentry.io).
231+
232+
<img src="/img/210406-python-sentry-aws-lambda/sentry-homepage.jpg" width="100%" class="shot rnd outl" alt="Sentry.io homepage where you can sign up for a free account.">
233+
234+
Sign into your account or sign up for a new free account. You will be at
235+
the main account dashboard after logging in or completing the Sentry sign
236+
up process.
237+
238+
There are no errors logged on our account dashboard yet, which is as
239+
expected because we have not yet connected our account to our Lambda
240+
function.
241+
242+
Click "Projects" on the left navigation bar, then "Create Project"
243+
in the top right corner.
244+
245+
Under "Choose a Platform", select "Serverless" and then "AWS Lambda (Python)"
246+
as shown below:
247+
248+
<img src="/img/210406-python-sentry-aws-lambda/aws-lambda-python.jpg" width="100%" class="shot rnd outl" alt="Choose AWS Lambda (Python) under the platform options.">
249+
250+
Decide under what criteria it should send error information out of
251+
Lambda. For this tutorial, we will have it send every exception.
252+
Then click the "Create Project." button.
253+
254+
You can have Sentry handle the instrumentation automatically but
255+
we will handle it manually for our function. On the next screen, Sentry
256+
will provide you with your unique DSN string, which we will need for
257+
our function.
258+
259+
<img src="/img/210406-python-sentry-aws-lambda/sentry-dsn-string.jpg" width="100%" class="shot rnd outl" alt="Copy the Sentry DSN string so we can export it as an environment variable.">
260+
261+
Typically you will want to
262+
[use environment variables on AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html)
263+
to store and access values like your Sentry key.
264+
265+
Copy the contents of the Sentry DSN string, and go into the Lambda console
266+
to create a new environment variable. To do that, click the "Configuration"
267+
tab within Lambda like you see here:
268+
269+
<img src="/img/210406-python-sentry-aws-lambda/aws-lambda-configuration.jpg" width="100%" class="shot rnd outl" alt="Click the Lambda Configuration tab.">
270+
271+
Then click "Edit" and add a new environment variable with the key of `SENTRY_DSN`
272+
and the value of the DSN string that you copied from the Sentry screen.
273+
274+
<img src="/img/210406-python-sentry-aws-lambda/add-env-var.jpg" width="100%" class="shot rnd outl" alt="Add the environment variable in AWS Lambda.">
275+
276+
Click the "Save" button and go back to your Lambda function code.
277+
278+
Update your Lambda function with the following highlighted new lines of code
279+
to send errors to Sentry.
280+
281+
```python
282+
import json
283+
~~import os
284+
~~import sentry_sdk
285+
~~from sentry_sdk.integrations.aws_lambda import AwsLambdaIntegration
286+
287+
~~SENTRY_DSN = os.environ.get('SENTRY_DSN')
288+
~~sentry_sdk.init(
289+
~~ dsn=SENTRY_DSN,
290+
~~ integrations=[AwsLambdaIntegration()]
291+
~~)
292+
293+
print('Loading function')
294+
295+
296+
def lambda_handler(event, context):
297+
#print("Received event: " + json.dumps(event, indent=2))
298+
print("value1 = " + event['key1'])
299+
print("value2 = " + event['key2'])
300+
print("value3 = " + event['key3'])
301+
print("value4 = " + event['key4'])
302+
return event['key1'] # Echo back the first key value
303+
#raise Exception('Something went wrong')
304+
```
305+
306+
Click the "Deploy" button and then "Test". The code will throw
307+
an error and when we go back to our Sentry dashboard we will
308+
see it captured and viewable for further inspection.
309+
310+
<img src="/img/210406-python-sentry-aws-lambda/sentry-error-dashboard.jpg" width="100%" class="shot rnd outl" alt="AWS Lambda exception in the Sentry dashboard.">
311+
312+
It works! Next you will likely want to tune your exception reporting
313+
criteria to make sure you get alerted to the right number of exceptions
314+
if you do not want to see all of them.
315+
316+
317+
## What's Next?
318+
We just wrote and executed a Python 3 function on AWS Lambda then
319+
captured the exception message into the Sentry logs. You can
320+
now continue building out your Python code knowing that when something
321+
goes wrong you will have full visibility on what happened.
322+
323+
Check out the [AWS Lambda section](/aws-lambda.html) for
324+
more tutorials by other developers.
325+
326+
Further questions? Contact me on Twitter
327+
[@fullstackpython](https://twitter.com/fullstackpython)
328+
or [@mattmakai](https://twitter.com/mattmakai). I am also on GitHub with
329+
the username [mattmakai](https://github.com/mattmakai).
330+
331+
Something wrong with this post? Fork
332+
[this page's source on GitHub](https://github.com/mattmakai/fullstackpython.com/blob/master/content/posts/210422-monitor-python-aws-lambda-sentry.markdown)
333+
and submit a pull request.

‎static/img/210406-python-sentry-aws-lambda/add-env-var.jpg‎

62.7 KB
Loading

‎static/img/210406-python-sentry-aws-lambda/add-lambda-layer.png‎

30.4 KB
Loading

‎static/img/210406-python-sentry-aws-lambda/add-layer-specify-arn.jpg‎

78 KB
Loading

‎static/img/210406-python-sentry-aws-lambda/arn-region.png‎

57.5 KB
Loading

‎static/img/210406-python-sentry-aws-lambda/aws-lambda-configuration.jpg‎

27.7 KB
Loading

0 commit comments

Comments
 (0)