fullstackpython.com/task-queues.html at gh-pages · alvsgithub/fullstackpython.com

History

397 lines (396 loc) · 26.7 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

<!DOCTYPE html>

<head>

<title>Task Queues - Full Stack Python</title>

<!--[if lt IE 9]>

<![endif]-->

</head>

<body>

<h2 style="color: #fff; margin: 20px 40px 0 0;"><a href="https://www.gumroad.com/l/python-deployments" style="color: #fff">The Full Stack Python Guide to Deployments book</a> has been released!</h2>

</div>

</div> <a href="https://github.com/makaimc/fullstackpython.com"><img style="position: absolute; top: 0; right: 0; border: 0;" src="/theme/img/fork.png" alt="Fork me on GitHub"></a>

<span class="logo-title"><a href="/">Full Stack Python</a></span>

</div>

</div><div class="row">

<h1>Task queues</h1>

<p>Task queues manage background work that must be executed outside the usual

HTTP request-response cycle.</p>

<h2>Why are task queues necessary?</h2>

<p>Tasks are handled asynchronously either because they are not initiated by

an HTTP request or because they are long-running jobs that would dramatically

reduce the performance of an HTTP response.</p>

<p>For example, a web application could poll the GitHub API every 10 minutes to

collect the names of the top 100 starred repositories. A task queue would

handle invoking code to call the GitHub API, process the results and store them

in a persistent database for later use.</p>

<p>Another example is when a database query would take too long during the HTTP

request-response cycle. The query could be performed in the background on a

fixed interval with the results stored in the database. When an

HTTP request comes in that needs those results a query would simply fetch the

precalculated result instead of re-executing the longer query.

This precalculation scenario is a form of <a href="/caching.html">caching</a> enabled

by task queues.</p>

<p>Other types of jobs for task queues include</p>

<ul>

<li>

<p>spreading out large numbers of independent database inserts over time

instead of inserting everything at once</p>

</li>

<li>

<p>aggregating collected data values on a fixed interval, such as every

15 minutes</p>

</li>

<li>

<p>scheduling periodic jobs such as batch processes</p>

</li>

</ul>

<h2>Task queue projects</h2>

<p>The defacto standard Python task queue is Celery. The other task queue

projects that arise tend to come from the perspective that Celery is overly

complicated for simple use cases. My recommendation is to put the effort into

Celery's reasonable learning curve as it is worth the time it takes to

understand how to use the project.</p>

<ul>

<li>

<p>The <a href="http://www.celeryproject.org/">Celery</a> distributed task queue is the

most commonly used Python library for handling asynchronous tasks and

scheduling.</p>

</li>

<li>

<p>The <a href="http://python-rq.org/">RQ (Redis Queue)</a> is a simple Python

library for queueing jobs and processing them in the background with workers.

RQ is backed by Redis and is designed to have a low barrier to entry.

The <a href="http://nvie.com/posts/introducing-rq/">intro post</a> contains information

on design decisions and how to use RQ.</p>

</li>

<li>

<p><a href="https://github.com/dcramer/taskmaster">Taskmaster</a> is a lightweight simple

distributed queue for handling large volumes of one-off tasks. </p>

</li>

<li>

<p><a href="http://huey.readthedocs.org/en/latest/">Huey</a> is a simple task queue that

uses Redis on the backend but otherwise does not depend on other libraries.

The project was previously known as Invoker and the author changed the name.</p>

</li>

</ul>

<h2>Hosted message and task queue services</h2>

<p>Task queue third party services aim to solve the complexity issues that arise

when scaling out a large deployment of distributed task queues.</p>

<ul>

<li>

<p><a href="http://www.iron.io/">Iron.io</a> is a distributed messaging service platform

that works with many types of task queues such as Celery. It also is built

to work with other IaaS and PaaS environments such as Amazon Web Services

and Heroku.</p>

</li>

<li>

<p><a href="http://aws.amazon.com/sqs/">Amazon Simple Queue Service (SQS)</a> is a

set of five APIs for creating, sending, receiving, modifying and deleting

messages.</p>

</li>

<li>

<p><a href="http://www.cloudamqp.com/">CloudAMQP</a> is at its core managed servers with

RabbitMQ installed and configured. This service is an option if you are

using RabbitMQ and do not want to maintain RabbitMQ installations on your

own servers.</p>

</li>

</ul>

<h2>Task queue resources</h2>

<ul>

<li>

<p><a href="http://www.caktusgroup.com/blog/2014/06/23/scheduling-tasks-celery/">Getting Started Scheduling Tasks with Celery</a>

is a detailed walkthrough for setting up Celery with Django (although

Celery can also be used without a problem with other frameworks).</p>

</li>

<li>

<p><a href="http://justcramer.com/2012/05/04/distributing-work-without-celery/">Distributing work without Celery</a>

provides a scenario in which Celery and RabbitMQ are not the right tool

for scheduling asynchronous jobs.</p>

</li>

<li>

<p><a href="http://www.warski.org/blog/2014/07/evaluating-persistent-replicated-message-queues/">Evaluating persistent, replicated message queues</a>

is a detailed comparison of Amazon SQS, MongoDB, RabbitMQ, HornetQ and

Kafka's designs and performance.</p>

</li>

<li>

<p><a href="http://queues.io/">Queues.io</a> is a collection of task queue systems with

short summaries for each one. The task queues are not all compatible with

Python but ones that work with it are tagged with the "Python" keyword.</p>

</li>

<li>

<p><a href="http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619">Why Task Queues</a>

is a presentation for what task queues are and why they are needed. </p>

</li>

<li>

<p>Flask by Example <a href="https://realpython.com/blog/python/flask-by-example-implementing-a-redis-task-queue/">Implementing a Redis Task Queue</a>

provides a detailed walkthrough of setting up workers to use RQ with

Redis.</p>

</li>

<li>

<p><a href="https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps">How to use Celery with RabbitMQ</a>

is a detailed walkthrough for using these tools on an Ubuntu VPS.</p>

</li>

<li>

<p>Heroku has a clear walkthrough for using

<a href="https://devcenter.heroku.com/articles/python-rq">RQ for background tasks</a>.</p>

</li>

<li>

<p><a href="http://www.linuxforu.com/2013/12/introducing-celery-pythondjango/">Introducing Celery for Python+Django</a>

provides an introduction to the Celery task queue.</p>

</li>

<li>

<p><a href="https://denibertovic.com/posts/celery-best-practices/">Celery - Best Practices</a>

explains things you should not do with Celery and shows some underused

features for making task queues easier to work with.</p>

</li>

<li>

<p>The "Django in Production" series by

<a href="https://twitter.com/robgolding63">Rob Golding</a> contains a post

specifically on <a href="http://www.robgolding.com/blog/2011/11/27/django-in-production-part-2---background-tasks/">Background Tasks</a>.</p>

</li>

<li>

<p><a href="http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/">Asynchronous Processing in Web Applications Part One</a>

and <a href="http://blog.thecodepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/">Part Two</a>

are great reads for understanding the difference between a task queue and

why you shouldn't use your database as one.</p>

</li>

<li>

<p><a href="http://www.caktusgroup.com/blog/2014/09/29/celery-production/">Celery in Production</a>

on the Caktus Group blog contains good practices from their experience

using Celery with RabbitMQ, monitoring tools and other aspects not often

discussed in existing documentation.</p>

</li>

<li>

<p><a href="https://www.youtube.com/watch?v=68QWZU_gCDA">A 4 Minute Intro to Celery</a> is

a short introductory task queue screencast.</p>

</li>

<li>

<p>Heroku wrote about how to

<a href="https://engineering.heroku.com/blogs/2014-09-15-securing-celery">secure Celery</a>

when tasks are otherwise sent over unencrypted networks.</p>

</li>

<li>

<p>Miguel Grinberg wrote a nice post on using the

<a href="http://blog.miguelgrinberg.com/post/using-celery-with-flask">task queue Celery with Flask</a>.

He gives an overview of Celery followed by specific code to set up the task

queue and integrate it with Flask.</p>

</li>

<li>

<p><a href="http://wiredcraft.com/posts/2015/02/04/3-gotchas-for-celery.html">3 Gotchas for Working with Celery</a>

are things to keep in mind when you're new to the Celery task queue

implementation.</p>

</li>

<li>

<p><a href="https://godjango.com/63-deferred-tasks-and-scheduled-jobs-with-celery-31-django-17-and-redis/">Deferred Tasks and Scheduled Jobs with Celery 3.1, Django 1.7 and Redis</a>

is a video along with code that shows how to set up Celery with Redis as the

broker in a Django application.</p>

</li>

<li>

<p><a href="http://michal.karzynski.pl/blog/2014/05/18/setting-up-an-asynchronous-task-queue-for-django-using-celery-redis/">Setting up an asynchronous task queue for Django using Celery and Redis</a>

is a straightforward tutorial for setting up the Celery task queue for

Django web applications using the Redis broker on the back end.</p>

</li>

<li>

<p><a href="http://django.zone/posts/background-jobs-django-and-celery">Background jobs with Django and Celery</a>

shows the code and a simple explanation of how to use Celery with

<a href="/django.html">Django</a>.</p>

</li>

<li>

<p><a href="https://realpython.com/blog/python/asynchronous-tasks-with-django-and-celery/">Asynchronous Tasks With Django and Celery</a>

shows how to integrate Celery with <a href="/django.html">Django</a> and create Periodic Tasks.</p>

</li>

<li>

<p><a href="https://library.launchkit.io/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb">Three quick tips from two years with Celery</a>

provides some solid advice on retry delays, the -Ofair flag and global

task timeouts for Celery.</p>

</li>

</ul>

<h2>Task queue learning checklist</h2>

<ol>

<li>

<p>Pick a slow function in your project that is called during an HTTP

request.</p>

</li>

<li>

<p>Determine if you can precompute the results on a fixed interval instead

of during the HTTP request. If so, create a separate function you can call

from elsewhere then store the precomputed value in the database.</p>

</li>

<li>

<p>Read the Celery documentation and the links in the resources section below

to understand how the project works.</p>

</li>

<li>

<p>Install a message broker such as RabbitMQ or Redis and then add Celery to

your project. Configure Celery to work with the installed message broker.</p>

</li>

<li>

<p>Use Celery to invoke the function from step one on a regular basis.</p>

</li>

<li>

<p>Have the HTTP request function use the precomputed value instead of the

slow running code it originally relied upon.</p>

</li>

</ol>

<h3>What's next to learn after task queues?</h3>

<p class="under-btn">How do I log errors that occur in my application?</p> </div>

</div>

<p class="under-btn">I want to learn more about app users via web analytics.</p> </div>

</div>

<p class="under-btn">What tools exist for monitoring a deployed web app?</p> </div>

</div>

</div><div id="mc_embed_signup">

<h4>Sign up here to receive an email with major updates to this site and Python tutorials delivered to your inbox once a month.</h4>

</div>

</div>

</form>

</div>

<p style="font-size: .8em; margin-top: 10px;">Searching for a complete, step-by-step deployment walkthrough? Learn more about <a href="http://www.deploypython.com/">The Full Stack Python Guide to Deployments book</a>.

</p>

</div>

</div> <div class="panel panel-success">

<h3 class="panel-head"><a href="/table-of-contents.html" style="color: #fff;">Table of Contents</a></h3>

</div>

<a href="/introduction.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Introduction</a>

<a href="/learning-programming.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Learning Programming</a>

<a href="/why-use-python.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Why Use Python?</a>

<a href="/python-2-or-3.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Python 2 or 3?</a>

<a href="/enterprise-python.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Enterprise Python</a>

<a href="/best-python-resources.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Best Python Resources</a>

<a href="/best-python-videos.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Best Python Videos</a>

<a href="/development-environments.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Development Environments</a>

<a href="/emacs.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Emacs</a>

<a href="/generators.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Generators</a>

<a href="/comprehensions.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Comprehensions</a>

<a href="/web-frameworks.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Frameworks</a>

<a href="/django.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Django</a>

<a href="/flask.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Flask</a>

<a href="/bottle.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Bottle</a>

<a href="/pyramid.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Pyramid</a>

<a href="/morepath.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Morepath</a>

<a href="/other-web-frameworks.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Other Web Frameworks</a>

<a href="/web-design.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Design</a>

<a href="/cascading-style-sheets.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Cascading Style Sheets</a>

<a href="/javascript.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>JavaScript</a>

<a href="/websockets.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>WebSockets</a>

<a href="/web-application-security.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Application Security</a>

<a href="/static-site-generator.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Static Site Generator</a>

<a href="/databases.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Databases</a>

<a href="/no-sql-datastore.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>NoSQL Data Stores</a>

<a href="/object-relational-mappers-orms.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Object-relational Mappers (ORMs)</a>

<a href="/application-programming-interfaces.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Application Programming Interfaces</a>

<a href="/api-integration.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>API Integration</a>

<a href="/api-creation.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>API Creation</a>

<a href="/deployment.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Deployment</a>

<a href="/servers.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Servers</a>

<a href="/platform-as-a-service.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Platform-as-a-service</a>

<a href="/operating-systems.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Operating Systems</a>

<a href="/web-servers.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Servers</a>

<a href="/wsgi-servers.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>WSGI Servers</a>

<a href="/source-control.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Source Control</a>

<a href="/application-dependencies.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Application Dependencies</a>

<a href="/static-content.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Static Content</a>

<a href="/task-queues.html" class="list-group-item smaller-item active" style='font-family: "Helvetica Neue",sans-serif;'>Task Queues</a>

<a href="/configuration-management.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Configuration Management</a>

<a href="/continuous-integration.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Continuous Integration</a>

<a href="/logging.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Logging</a>

<a href="/monitoring.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Monitoring</a>

<a href="/web-analytics.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Web Analytics</a>

<a href="/docker.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Docker</a>

<a href="/caching.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Caching</a>

<a href="/microservices.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Microservices</a>

<a href="/testing.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Testing</a>

<a href="/unit-testing.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Unit Testing</a>

<a href="/integration-testing.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Integration Testing</a>

<a href="/code-metrics.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Code Metrics</a>

<a href="/debugging.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Debugging</a>

<a href="/what-full-stack-means.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>What "Full Stack" Means</a>

<a href="/change-log.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Change Log</a>

<a href="/future-directions.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>Future Directions</a>

<a href="/about-author.html" class="list-group-item smaller-item " style='font-family: "Helvetica Neue",sans-serif;'>About the Author</a>

</div>

</p>

</div>

</div> <div class="panel panel-success">

<div class="panel-heading"><h3 class="panel-head">Task Queues</h3></div>

Major updates are tweeted via

<a href="https://twitter.com/fullstackpython">@fullstackpython</a>.

<hr/>

Need more detailed tutorials than you see here?

<a href="http://www.deploypython.com/">Learn more about The Full Stack Python Guide to Deployments book.</a>

</div>

</div></div>

<hr/>

<a href="http://www.mattmakai.com/" class="underline">Matt Makai</a>

2015

</div>

(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){

(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),

m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)

})(window,document,'script','//www.google-analytics.com/analytics.js','ga');

ga('create', 'UA-19910497-7', 'auto');

ga('send', 'pageview');

</script>

var trackOutboundLink = function(url) { ga('send', 'event', 'outbound', 'click', url, {'hitCallback': function () { document.location = url; } }); }

</script>

</body>

</html>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

task-queues.html

Latest commit

History

task-queues.html

File metadata and controls