forked from mattmakai/fullstackpython.com
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathmonitoring.html
More file actions
357 lines (347 loc) · 22.1 KB
/
monitoring.html
File metadata and controls
357 lines (347 loc) · 22.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
<!DOCTYPE html>
<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="author" content="Matt Makai">
<link rel="shortcut icon" href="theme/img/fsp-fav.png">
<title>Monitoring - Full Stack Python</title>
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
<![endif]-->
<link href="theme/css/f.min.css" rel="stylesheet">
<link href="//netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.css" rel="stylesheet">
<style>
.toc-indent {padding-left: 25px; padding-bottom: 1px;}
.toc-more-indent {padding-left: 37px; padding-bottom: 1px;}
</style>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-19910497-7', 'auto');
ga('send', 'pageview');
</script> </head>
<body>
<a href="https://github.com/makaimc/fullstackpython.github.com"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/a6677b08c955af8400f44c6298f40e7d19cc5b2d/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f677261795f3664366436642e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_gray_6d6d6d.png"></a>
<div class="container">
<div class="row">
<div class="col-md-12">
<div class="logo-header-section">
<a href="/" style="text-decoration: none; border: none;"><img src="theme/img/fsp-logo.png" height="52" width="52" class="logo-image" style="padding-top: 1px;" alt="Full Stack Python logo"></a>
<span class="logo-title"><a href="/">Full Stack Python</a></span>
</div>
</div>
</div><div class="row">
<div class="col-md-8">
<h1>Monitoring</h1>
<p>Monitoring tools capture, analyze and display information for a web
application's execution. Every application has issues arise throughout
all levels of the web stack. Monitoring tools provide transparency so
developers and operations teams can respond and fix problems.</p>
<h2>Why is monitoring necessary?</h2>
<p>Capturing and analyzing data about your production environment is critical
to proactively deal with stability, performance, and errors in a web
application.</p>
<h2>Difference between monitoring and logging</h2>
<p>Monitoring and logging are very similar in their purpose of helping to
diagnose issues with an application and aid the debugging process. One way
to think about the difference is that logging happens based on explicit events
while monitoring is a passive background collection of data. </p>
<p>For example, when an error occurs, that event is explicitly logged through
code in an exception handler. Meanwhile, a monitoring agent instruments the
code and gathers data not only about the logged exception but also the
performance of the functions.</p>
<p>This distinction between logging and monitoring is vague and not necessarily
the only way to look at it. Pragmatically, both are useful for maintaining a
production web application.</p>
<h2>Monitoring layers</h2>
<p>There are several important resources to monitor on the operating system
and network level of a web stack.</p>
<ol>
<li>CPU utilization</li>
<li>Memory utilization</li>
<li>Persistence storage consumed versus free</li>
<li>Network bandwidth and latency</li>
</ol>
<p>Application level monitoring encompasses several aspects. The amount of time
and resources dedicated to each aspect will vary based on whether an
application is read-heavy, write-heavy, or subject to rapid swings in traffic.</p>
<ol>
<li>Application warnings and errors (500-level HTTP errors)</li>
<li>Application code performance</li>
<li>Template rendering time</li>
<li>Browser rendering time for the application</li>
<li>Database querying performance</li>
</ol>
<h2>Open source monitoring projects</h2>
<ul>
<li>
<p><a href="https://github.com/etsy/statsd/">statsd</a> is a node.js network daemon that
listens for metrics and aggregates them for transfer into another service
such as Graphite.</p>
</li>
<li>
<p><a href="https://graphite.readthedocs.org/en/latest/overview.html">Graphite</a> stores
time-series data and displays them in graphs through a Django web application.</p>
</li>
<li>
<p><a href="http://github.hubspot.com/bucky/">Bucky</a> measures the performance of a
web application from end user's browsers and sends that data back to the
server for collection.</p>
</li>
<li>
<p><a href="http://sensuapp.org/">Sensu</a> is an open source monitoring framework
written in Ruby but applicable to any programming language web application.</p>
</li>
<li>
<p><a href="http://vimeo.github.io/graph-explorer/">Graph Explorer</a> by Vimeo is a
Graphite-based dashboard with added features and a slick design.</p>
</li>
<li>
<p><a href="http://packetbeat.com/">PacketBeat</a> sniffs protocol packets. Elasticsearch
then allows developers to search the collected data and visualize what's
happening inside their web application using the Kibana user interface.</p>
</li>
<li>
<p><a href="http://munin-monitoring.org/">Munin</a> is a client plugin-based monitoring
system that sends monitoring traffic to the Munin node where the data can
be analyzed and visualized. Note this project is written in Perl so Perl 5
must be installed on the node collecting the data.</p>
</li>
</ul>
<h2>Hosted monitoring services</h2>
<ul>
<li>
<p><a href="http://newrelic.com/">New Relic</a>. Application and database monitoring as
well as plug ins for capturing and analyzing additional data about tools in
your stack.</p>
</li>
<li>
<p><a href="http://copperegg.com/">CopperEgg</a> is lower-level monitoring on server and
infrastructure. It's popular with DevOps shops that are making changes to
their production environments and want immediate feedback on the results
of those modifications.</p>
</li>
<li>
<p><a href="http://status.io/">Status.io</a> focuses on uptime and response metrics
transparency for web applications.</p>
</li>
<li>
<p><a href="https://www.statuspage.io/">StatusPage.io</a> (yes, there's both a Status and
StatusPage.io) provides easy set up status pages for monitoring application
up time.</p>
</li>
<li>
<p><a href="http://www.pagerduty.com/">PagerDuty</a> alerts a designated person or group
if there are stability, performance, or uptime issues with an application.</p>
</li>
<li>
<p><a href="https://appenlight.com/">App Enlight</a> provides performance, exception and
error monitoring and is currently specific to Python web applications.</p>
</li>
</ul>
<h2>Monitoring resources</h2>
<ul>
<li>
<p><a href="http://www.paperplanes.de/2011/1/5/the_virtues_of_monitoring.html">The Virtues of Monitoring</a></p>
</li>
<li>
<p><a href="http://blog.docker.io/2013/07/effortless-monitoring-with-collectd-graphite-and-docker/">Effortless Monitoring with collectd, Graphite, and Docker</a></p>
</li>
<li>
<p><a href="http://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/">Practical Guide to StatsD/Graphite Monitoring</a>
is a detailed guide with code examples for monitoring infrastructure.</p>
</li>
<li>
<p>Bit.ly describes the
"<a href="http://word.bitly.com/post/74839060954/ten-things-to-monitor">10 Things They Forgot to Monitor</a>"
beyond the standard metrics such as disk & memory usage.</p>
</li>
<li>
<p><a href="http://aarvik.dk/four-linux-server-monitoring-and-management-tools/">Four Linux server monitoring tools</a></p>
</li>
<li>
<p><a href="https://blog.serverdensity.com/how-to-design-useful-monitoring-graphs-and-visualizations/">How to design useful monitoring and graphing visualizations</a></p>
</li>
<li>
<p><a href="https://speakerdeck.com/auxesis/5-years-of-metrics-and-monitoring">5 years of metrics and monitoring</a>
is a great presentation highlighting that visualization so humans can
understand measurements is a hard problem. Line graphs are often not
the best solution and they are overused.</p>
</li>
<li>
<p>The Collector Highlight Series has an article on <a href="http://blog.librato.com/posts/statsd">StatsD</a>
that explains how to install it and how it works.</p>
</li>
<li>
<p>This <a href="http://kartar.net/2014/11/monitoring-survey---tools/">survey on monitoring tools</a>
has some nice data and graphs on what developers and operations folks use
in their environments.</p>
</li>
<li>
<p>Ryan Frantz wrote a nice post on
<a href="http://ryanfrantz.com/posts/solving-monitoring/">Solving Monitoring</a>
with a new definition of what monitoring means based on today's complex
systems and how the practice should evolve going forward.</p>
</li>
</ul>
<h2>Monitoring learning checklist</h2>
<p><i class="fa fa-check-square-o"></i>
Review the software-as-a-service and open source monitoring tools below. Third
party services tend to be easier to set up and host the data for you. Open
source projects give you more control but you'll need to have additional
servers ready for the monitoring.</p>
<p><i class="fa fa-check-square-o"></i>
My recommendation is to install <a href="http://newrelic.com/">New Relic</a>'s free
option with the trial period to see how it works with your app. It'll give you
a good idea of the capabilities for application-level monitoring tools.</p>
<p><i class="fa fa-check-square-o"></i>
As your app scales take a look at setting up one of the the open source
monitoring projects such as StatsD with Graphite. The combination of those
two projects will give you fine-grained control over the system metrics
you're collecting and visualizing.</p>
<h3>What topic do you want to learn next?</h3>
<div class="row">
<div class="col-md-3">
<div class="well select-next">
<a href="/web-analytics.html" class="btn btn-success btn-full"><i class="fa fa-dashboard fa-2x"></i></a>
<p class="under-btn">
How do I learn more about the users of my app with web analytics?
</p>
</div>
</div>
<div class="col-md-3">
<div class="well select-next">
<a href="/web-application-security.html" class="btn btn-success btn-full"><i class="fa fa-lock fa-inverse fa-2x"></i></a>
</a>
<p class="under-btn">
What should I learn about web application security?
</p>
</div>
</div>
<div class="col-md-3">
<div class="well select-next">
<a href="/configuration-management.html" class="btn btn-success btn-full"><i class="fa fa-gears fa-inverse fa-2x"></i></a>
<p class="under-btn">
How do I automate the server configuration I've set up?
</p>
</div>
</div>
<div class="col-md-3">
<div class="well select-next">
<a href="/logging.html" class="btn btn-success btn-full"><i class="fa fa-align-left fa-inverse fa-2x"></i></a>
<p class="under-btn">
How should I log events in my application outside monitoring?
</p>
</div>
</div>
</div> <div id="mc_embed_signup">
<form action="//mattmakai.us2.list-manage.com/subscribe/post?u=b7e774f0c4f05dcebbfee183d&id=b22335388d" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate>
<div id="mc_embed_signup_scroll">
<h4>Sign up here to receive an email with major updates to this site and Python tutorials delivered to your inbox once a month.</h4>
<div class="row">
<div class="col-md-9">
<input type="email" value="" name="EMAIL" class="email form-control" id="mce-EMAIL" placeholder="email address" required>
<div style="position: absolute; left: -5000px;"><input type="text" name="b_b7e774f0c4f05dcebbfee183d_b22335388d" tabindex="-1" value=""></div>
</div>
<div class="col-md-3">
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="btn btn-success" style="font-family: 'Helvetica Neue';"></div>
</div>
</div>
</div>
</form>
</div>
</div>
<div class="col-md-offset-1 col-md-3" id="sidebar">
<div class="panel panel-success">
<div class="panel-heading">
<h3 class="panel-head">Sponsored By</h3>
</div>
<div class="panel-body">
<a href="https://www.loggly.com/log-management/python-logging/?utm_source=fsp&utm_medium=promo&utm_campaign=march"><img src="theme/img/sponsored/loggly.jpg" alt="Loggly with Beaver logo" width="100%"></a>
<p style="font-size: .8em; margin-top: 10px;">Want to improve monitoring
in your Python web app? I highly recommend
<a href="https://www.loggly.com/log-management/python-logging/?utm_source=fsp&utm_medium=promo&utm_campaign=march" onclick="trackOutboundLink('https://www.loggly.com/log-management/python-logging/?utm_source=fsp&utm_medium=promo&utm_campaign=march'); return false;">signing up for a free Loggly account</a>.
</p>
</div>
</div>
<div class="panel panel-success">
<div class="panel-heading">
<h3 class="panel-head"><a href="/table-of-contents.html" style="color: #fff;">Table of Contents</a></h3>
</div>
<div class="list-group">
<a href="/introduction.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Introduction</a>
<a href="/why-use-python.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Why Use Python?</a>
<a href="/best-python-resources.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Best Python Resources</a>
<a href="/best-python-videos.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Best Python Videos</a>
<a href="/development-environments.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Development Environments</a>
<a href="/vim.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Vim</a>
<a href="/emacs.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Emacs</a>
<a href="/generators.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Generators</a>
<a href="/comprehensions.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Comprehensions</a>
<a href="/web-frameworks.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Frameworks</a>
<a href="/django.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Django</a>
<a href="/flask.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Flask</a>
<a href="/bottle.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Bottle</a>
<a href="/pyramid.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Pyramid</a>
<a href="/morepath.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Morepath</a>
<a href="/other-web-frameworks.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Other Web Frameworks</a>
<a href="/web-design.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Design</a>
<a href="/cascading-style-sheets.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Cascading Style Sheets</a>
<a href="/javascript.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>JavaScript</a>
<a href="/websockets.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>WebSockets</a>
<a href="/web-application-security.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Application Security</a>
<a href="/databases.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Databases</a>
<a href="/no-sql-datastore.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>NoSQL Data Stores</a>
<a href="/application-programming-interfaces.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Application Programming Interfaces</a>
<a href="/api-integration.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>API Integration</a>
<a href="/api-creation.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>API Creation</a>
<a href="/deployment.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Deployment</a>
<a href="/servers.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Servers</a>
<a href="/platform-as-a-service.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Platform-as-a-service</a>
<a href="/operating-systems.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Operating Systems</a>
<a href="/web-servers.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Servers</a>
<a href="/wsgi-servers.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>WSGI Servers</a>
<a href="/source-control.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Source Control</a>
<a href="/application-dependencies.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Application Dependencies</a>
<a href="/static-content.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Static Content</a>
<a href="/task-queues.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Task Queues</a>
<a href="/configuration-management.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Configuration Management</a>
<a href="/continuous-integration.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Continuous Integration</a>
<a href="/logging.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Logging</a>
<a href="/monitoring.html" class="list-group-item smaller-item active" style='font-family: "Helvetica Neue",sans-serif;'>Monitoring</a>
<a href="/web-analytics.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Analytics</a>
<a href="/docker.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Docker</a>
<a href="/caching.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Caching</a>
<a href="/code-metrics.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Code Metrics</a>
<a href="/what-full-stack-means.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>What "Full Stack" Means</a>
<a href="/change-log.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Change Log</a>
<a href="/future-directions.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Future Directions</a>
<a href="/about-author.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>About the Author</a>
</div>
</div>
<div class="panel panel-success">
<div class="panel-heading"><h3 class="panel-head">Monitoring</h3></div>
<div class="panel-body">
Major updates are tweeted via
<a href="https://twitter.com/fullstackpython">@fullstackpython</a>.
<hr/>
Need more detailed tutorials than you see here?
<a href="/email.html">Sign up to receive an email when that content is created.</a>
</div>
</div>
</div></div>
<hr/>
<div class="footer pull-right">
<a href="http://www.mattmakai.com/" class="underline">Matt Makai</a>
2015
</div>
</div>
<script type='text/javascript'>
var trackOutboundLink = function(url) { ga('send', 'event', 'outbound', 'click', url, {'hitCallback': function () { document.location = url; } }); }
</script>
</body>
</html>