forked from mattmakai/fullstackpython.com
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathtask-queues.html
More file actions
199 lines (193 loc) · 9.74 KB
/
task-queues.html
File metadata and controls
199 lines (193 loc) · 9.74 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
<!DOCTYPE html>
<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="Full Stack Python explains each layer of the web application stack, from the server up through the rendering in a user's browser.">
<meta name="author" content="Matt Makai">
<link rel="shortcut icon" href="theme/img/full-stack-python-logo-bw.png">
<title>Full Stack Python: Task Queues</title>
<link href="theme/css/fsp.css" rel="stylesheet">
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
<![endif]-->
<style>
html,
body {
font-size: 18px;
color: #222;
background: #fefefe;
}
body {
padding-top: 30px;
}
.footer {
padding: 20px 0 30px 0;
}
a, a:hover {border-bottom: 1px dotted; color: #444;}
a:hover {text-decoration: none; color: #000;}
.logo-title {font-size: 56px; color: #403072; padding-top: 80px;
font-family: "News Cycle", "Arial Narrow Bold", sans-serif;
font-weight: bold; line-height: 30px; margin-left: 5px;}
.logo-title a, .logo-title a:hover {color: #000; text-decoration: none;
border-bottom: none;}
.logo-title a:hover {color: gray;}
.logo-image {vertical-align: top; border: none;}
a.list-group-item.active {background: #444; border: 1px solid #222;}
a.list-group-item.active:hover {background: #444; border: 1px solid #222;}
#sidebar {margin-top: 30px;}
@media (max-width: 600px) {
.logo-header-section {
margin: 20px 32px 0 0;
}
}
</style>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-19910497-7']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<a href="https://github.com/makaimc/fullstackpython.github.com" class="github">
<img style="position: absolute; top: 0; right: 0; border: 0;" src="http://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png" alt="Fork me on GitHub" />
</a>
<div class="container">
<div class="row">
<div class="col-md-12">
<div class="logo-header-section">
<a href="/" style="text-decoration: none; border: none;"><img src="theme/img/full-stack-python-logo-bw.png" height="42" width="42" class="logo-image" /></a>
<span class="logo-title"><a href="/">Full Stack Python</a></span>
</div>
</div>
</div>
<div class="row">
<div class="col-md-8">
<h1>Task queues</h1>
<p>Task queues handle background work that need to be processed outside the
usual HTTP request-response cycle. These tasks are handled asynchronously
because HTTP requests must be responded back to by the server as fast as
possible otherwise the user experience in the web browser will suffer. The
most common types of jobs for task queues include</p>
<ul>
<li>
<p>calculating computationally expensive data analytics</p>
</li>
<li>
<p>scheduling periodic jobs such as batch processes</p>
</li>
<li>
<p>spreading out large numbers of independent database inserts over time
instead of all at once</p>
</li>
<li>
<p>aggregating collected data values on a fixed interval, such as every
15 minutes</p>
</li>
</ul>
<h2>Task queue projects</h2>
<ul>
<li>
<p>The <a href="http://www.celeryproject.org/">Celery</a> distributed task queue is the
most commonly used Python library for handling asynchronous tasks and
scheduling.</p>
</li>
<li>
<p>The <a href="http://python-rq.org/">RQ (Redis Queue)</a> is a simple Python
library for queueing jobs and processing them in the background with workers.
RQ is backed by Redis and it is designed to have a low barrier to entry.
The <a href="http://nvie.com/posts/introducing-rq/">intro post</a> contains information
on design decisions and how to use RQ.</p>
</li>
<li>
<p><a href="https://github.com/dcramer/taskmaster">Taskmaster</a> is a lightweight simple
distributed queue for handling large volumes of one-off tasks. </p>
</li>
</ul>
<h2>Task queue resources</h2>
<ul>
<li>
<p><a href="http://justcramer.com/2012/05/04/distributing-work-without-celery/">Distributing work without Celery</a>
provides a scenario in which Celery and RabbitMQ are not the right tool
for scheduling asynchronous jobs.</p>
</li>
<li>
<p><a href="http://queues.io/">Queues.io</a> is a collection of task queue systems with
short summaries for each one. The task queues are not all compatible with
Python but ones that work with it are tagged with the "Python" keyword.</p>
</li>
<li>
<p><a href="http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619">Why Task Queues</a>
is a presentation for what task queues are and why they are needed. </p>
</li>
<li>
<p><a href="https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps">How to use Celery with RabbitMQ</a>
is a detail walkthrough for using these tools on an Ubuntu VPS.</p>
</li>
<li>
<p>Heroku has a clear walkthrough for using
<a href="https://devcenter.heroku.com/articles/python-rq">RQ for background tasks</a>.</p>
</li>
<li>
<p><a href="http://www.linuxforu.com/2013/12/introducing-celery-pythondjango/">Introducing Celery for Python+Django</a>
provides an introduction to the Celery task queue.</p>
</li>
<li>
<p>The "Django in Production" series by
<a href="https://twitter.com/robgolding63">Rob Golding</a> contains a post
specifically on <a href="http://www.robgolding.com/blog/2011/11/27/django-in-production-part-2---background-tasks/">Background Tasks</a>.</p>
</li>
<li>
<p><a href="http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/">Asynchronous Processing in Web Applications Part One</a>
and <a href="http://blog.thecodepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/">Part Two</a>
are great reads for understanding the difference between a task queue and
why you shouldn't use your database as one.</p>
</li>
</ul>
<br/>
Next read the
<a href="/api-integration.html">api integration</a> section.
</div>
<div class="col-md-offset-1 col-md-3" id="sidebar">
<div class="list-group">
<a href="/introduction.html" class="list-group-item ">Introduction</a>
<a href="/servers.html" class="list-group-item ">Servers</a>
<a href="/operating-systems.html" class="list-group-item ">Operating Systems</a>
<a href="/web-servers.html" class="list-group-item ">Web Servers</a>
<a href="/platform-as-a-service.html" class="list-group-item ">Platform-as-a-service</a>
<a href="/databases.html" class="list-group-item ">Databases</a>
<a href="/wsgi-servers.html" class="list-group-item ">WSGI Servers</a>
<a href="/web-frameworks.html" class="list-group-item ">Web Frameworks</a>
<a href="/application-dependencies.html" class="list-group-item ">Application Dependencies</a>
<a href="/cascading-style-sheets.html" class="list-group-item ">Cascading Style Sheets</a>
<a href="/static-content.html" class="list-group-item ">Static Content</a>
<a href="/source-control.html" class="list-group-item ">Source Control</a>
<a href="/caching.html" class="list-group-item ">Caching</a>
<a href="/task-queues.html" class="list-group-item active">Task Queues</a>
<a href="/api-integration.html" class="list-group-item ">API Integration</a>
<a href="/no-sql-datastore.html" class="list-group-item ">NoSQL Data Stores</a>
<a href="/web-application-security.html" class="list-group-item ">Web Security</a>
<a href="/logging.html" class="list-group-item ">Logging</a>
<a href="/monitoring.html" class="list-group-item ">Monitoring</a>
<a href="/configuration-management.html" class="list-group-item ">Configuration Management</a>
<a href="/web-analytics.html" class="list-group-item ">Web Analytics</a>
<a href="/best-python-resources.html" class="list-group-item ">Best Python Resources</a>
<a href="/about-author.html" class="list-group-item ">About the Author</a>
<a href="/change-log.html" class="list-group-item ">Change Log</a>
</div>
</div></div>
<hr/>
<div class="footer pull-right">
<a href="http://www.mattmakai.com/" class="underline">Matt Makai</a> 2014
</div>
</div>
<script src="http://code.jquery.com/jquery-2.1.0.min.js"></script>
<script src="theme/js/bootstrap.min.js"></script>
</body>
</html>