Skip to content

Commit 5debfdf

Browse files
mattmakaimattmakai
authored
Merge pull request mattmakai#143 from Bogdanp/master
add Dramatiq
2 parents 299d3da + 8cb8cf0 commit 5debfdf

File tree

4 files changed

+100
-39
lines changed

4 files changed

+100
-39
lines changed

‎content/pages/07-web-development/26-task-queues.markdown‎

Lines changed: 46 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ HTTP request-response cycle.
1313

1414

1515
## Why are task queues necessary?
16-
Tasks are handled asynchronously either because they are not initiated by
16+
Tasks are handled asynchronously either because they are not initiated by
1717
an HTTP request or because they are long-running jobs that would dramatically
1818
reduce the performance of an HTTP response.
1919

@@ -27,12 +27,12 @@ request-response cycle. The query could be performed in the background on a
2727
fixed interval with the results stored in the database. When an
2828
HTTP request comes in that needs those results a query would simply fetch the
2929
precalculated result instead of re-executing the longer query.
30-
This precalculation scenario is a form of [caching](/caching.html) enabled
30+
This precalculation scenario is a form of [caching](/caching.html) enabled
3131
by task queues.
3232

3333
Other types of jobs for task queues include
3434

35-
* spreading out large numbers of independent database inserts over time
35+
* spreading out large numbers of independent database inserts over time
3636
instead of inserting everything at once
3737

3838
* aggregating collected data values on a fixed interval, such as every
@@ -42,39 +42,42 @@ Other types of jobs for task queues include
4242

4343

4444
## Task queue projects
45-
The defacto standard Python task queue is [Celery](/celery.html). The other
46-
task queue projects that arise tend to come from the perspective that
47-
Celery is overly complicated for simple use cases. My recommendation is to
48-
put the effort into Celery's reasonable learning curve as it is worth the
45+
The defacto standard Python task queue is [Celery](/celery.html). The other
46+
task queue projects that arise tend to come from the perspective that
47+
Celery is overly complicated for simple use cases. My recommendation is to
48+
put the effort into Celery's reasonable learning curve as it is worth the
4949
time it takes to understand how to use the project.
5050

5151
* The [Celery](/celery.html) distributed task queue is the
52-
most commonly used Python library for handling asynchronous tasks and
52+
most commonly used Python library for handling asynchronous tasks and
5353
scheduling.
5454

5555
* The [RQ (Redis Queue)](/redis-queue-rq.html) is a simple Python
56-
library for queueing jobs and processing them in the background with
57-
workers. RQ is backed by Redis and is designed to have a low barrier to
56+
library for queueing jobs and processing them in the background with
57+
workers. RQ is backed by Redis and is designed to have a low barrier to
5858
entry.
5959

6060
* [Taskmaster](https://github.com/dcramer/taskmaster) is a lightweight simple
61-
distributed queue for handling large volumes of one-off tasks.
61+
distributed queue for handling large volumes of one-off tasks.
6262

63-
* [Huey](http://huey.readthedocs.org/en/latest/) is a Redis-based task
64-
queue that aims to provide a simple, yet flexible framework for
65-
executing tasks. Huey supports task scheduling, crontab-like repeating
63+
* [Huey](http://huey.readthedocs.org/en/latest/) is a Redis-based task
64+
queue that aims to provide a simple, yet flexible framework for
65+
executing tasks. Huey supports task scheduling, crontab-like repeating
6666
tasks, result storage and automatic retry in the event of failure.
6767

6868
* [Kuyruk](https://kuyruk.readthedocs.io) is simple and easy to use task queue
6969
system built on top of RabbitMQ. Although feature set is small, new features
7070
can be added by extensions.
7171

72+
* [Dramatiq](https://dramatiq.io) is a fast and reliable alternative
73+
to Celery. It supports RabbitMQ and Redis as message brokers.
74+
7275

7376
## Hosted message and task queue services
7477
Task queue third party services aim to solve the complexity issues that arise
7578
when scaling out a large deployment of distributed task queues.
7679

77-
* [Iron.io](http://www.iron.io/) is a distributed messaging service platform
80+
* [Iron.io](http://www.iron.io/) is a distributed messaging service platform
7881
that works with many types of task queues such as Celery. It also is built
7982
to work with other IaaS and PaaS environments such as Amazon Web Services
8083
and Heroku.
@@ -84,27 +87,32 @@ when scaling out a large deployment of distributed task queues.
8487
messages.
8588

8689
* [CloudAMQP](http://www.cloudamqp.com/) is at its core managed servers with
87-
RabbitMQ installed and configured. This service is an option if you are
88-
using RabbitMQ and do not want to maintain RabbitMQ installations on your
90+
RabbitMQ installed and configured. This service is an option if you are
91+
using RabbitMQ and do not want to maintain RabbitMQ installations on your
8992
own servers.
9093

9194

9295
## Open source examples that use task queues
93-
* Take a look at the code in this open source
94-
[Flask application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/flask)
95-
and
96-
[this Django application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/django)
96+
* Take a look at the code in this open source
97+
[Flask application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/flask)
98+
and
99+
[this Django application](https://www.twilio.com/docs/howto/walkthrough/appointment-reminders/python/django)
97100
for examples of how to use and deploy Celery with a Redis broker to
98-
send text messages with these frameworks.
101+
send text messages with these frameworks.
99102

100103
* [flask-celery-example](https://github.com/thrisp/flask-celery-example) is
101104
a simple Flask application with Celery as a task queue and Redis as
102105
the broker.
103106

107+
* [django_dramatiq_example](https://github.com/Bogdanp/django_dramatiq_example) and
108+
[flask_dramatiq_example](https://github.com/Bogdanp/flask_dramatiq_example)
109+
are simple apps that demo how you can use Dramatiq with Django and
110+
Flask, respectively.
111+
104112

105113
## Task queue resources
106114
* [International Space Station notifications with Python and Redis Queue (RQ)](https://www.twilio.com/blog/2015/11/international-space-station-notifications-with-python-redis-queue-and-twilio-copilot.html)
107-
shows how to combine the RQ task queue library with Flask to send
115+
shows how to combine the RQ task queue library with Flask to send
108116
text message notifications every time a condition is met - in this blog
109117
post's case that the ISS is currently flying over your location on
110118
Earth.
@@ -117,10 +125,10 @@ when scaling out a large deployment of distributed task queues.
117125
short summaries for each one. The task queues are not all compatible with
118126
Python but ones that work with it are tagged with the "Python" keyword.
119127

120-
* [Why Task Queues](http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619)
121-
is a presentation for what task queues are and why they are needed.
128+
* [Why Task Queues](http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619)
129+
is a presentation for what task queues are and why they are needed.
122130

123-
* [Asynchronous Processing in Web Applications Part One](http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/)
131+
* [Asynchronous Processing in Web Applications Part One](http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/)
124132
and [Part Two](http://blog.thecodepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/)
125133
are great reads for understanding the difference between a task queue and
126134
why you shouldn't use your database as one.
@@ -130,18 +138,18 @@ when scaling out a large deployment of distributed task queues.
130138
provides a detailed walkthrough of setting up workers to use RQ with
131139
Redis.
132140

133-
* Heroku has a clear walkthrough for using
141+
* Heroku has a clear walkthrough for using
134142
[RQ for background tasks](https://devcenter.heroku.com/articles/python-rq).
135143

136144
* [How to use Celery with RabbitMQ](https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps)
137145
is a detailed walkthrough for using these tools on an Ubuntu VPS.
138146

139147
* [Celery - Best Practices](https://denibertovic.com/posts/celery-best-practices/)
140-
explains things you should not do with Celery and shows some underused
148+
explains things you should not do with Celery and shows some underused
141149
features for making task queues easier to work with.
142150

143151
* [Celery in Production](http://www.caktusgroup.com/blog/2014/09/29/celery-production/)
144-
on the Caktus Group blog contains good practices from their experience
152+
on the Caktus Group blog contains good practices from their experience
145153
using Celery with RabbitMQ, monitoring tools and other aspects not often
146154
discussed in existing documentation.
147155

@@ -151,25 +159,25 @@ when scaling out a large deployment of distributed task queues.
151159
* This [Celery tasks checklist](http://celerytaskschecklist.com/) has
152160
some nice tips and resources for using Celery in your applications.
153161

154-
* Heroku wrote about how to
162+
* Heroku wrote about how to
155163
[secure Celery](https://engineering.heroku.com/blogs/2014-09-15-securing-celery)
156164
when tasks are otherwise sent over unencrypted networks.
157165

158-
* Miguel Grinberg wrote a nice post on using the
159-
[task queue Celery with Flask](http://blog.miguelgrinberg.com/post/using-celery-with-flask).
166+
* Miguel Grinberg wrote a nice post on using the
167+
[task queue Celery with Flask](http://blog.miguelgrinberg.com/post/using-celery-with-flask).
160168
He gives an overview of Celery followed by specific code to set up the task
161169
queue and integrate it with Flask.
162170

163171
* [3 Gotchas for Working with Celery](https://wiredcraft.com/blog/3-gotchas-for-celery/)
164-
are things to keep in mind when you're new to the Celery task queue
172+
are things to keep in mind when you're new to the Celery task queue
165173
implementation.
166174

167175
* [Deferred Tasks and Scheduled Jobs with Celery 3.1, Django 1.7 and Redis](https://godjango.com/63-deferred-tasks-and-scheduled-jobs-with-celery-31-django-17-and-redis/)
168176
is a video along with code that shows how to set up Celery with Redis as the
169177
broker in a Django application.
170178

171179
* [Setting up an asynchronous task queue for Django using Celery and Redis](http://michal.karzynski.pl/blog/2014/05/18/setting-up-an-asynchronous-task-queue-for-django-using-celery-redis/)
172-
is a straightforward tutorial for setting up the Celery task queue for
180+
is a straightforward tutorial for setting up the Celery task queue for
173181
Django web applications using the Redis broker on the back end.
174182

175183
* [Three quick tips from two years with Celery](https://library.launchkit.io/three-quick-tips-from-two-years-with-celery-c05ff9d7f9eb)
@@ -178,21 +186,20 @@ when scaling out a large deployment of distributed task queues.
178186

179187

180188
## Task queue learning checklist
181-
1. Pick a slow function in your project that is called during an HTTP
189+
1. Pick a slow function in your project that is called during an HTTP
182190
request.
183191

184-
1. Determine if you can precompute the results on a fixed interval instead
192+
1. Determine if you can precompute the results on a fixed interval instead
185193
of during the HTTP request. If so, create a separate function you can call
186194
from elsewhere then store the precomputed value in the database.
187195

188196
1. Read the Celery documentation and the links in the resources section below
189197
to understand how the project works.
190198

191-
1. Install a message broker such as RabbitMQ or Redis and then add Celery to
199+
1. Install a message broker such as RabbitMQ or Redis and then add Celery to
192200
your project. Configure Celery to work with the installed message broker.
193201

194202
1. Use Celery to invoke the function from step one on a regular basis.
195203

196-
1. Have the HTTP request function use the precomputed value instead of the
204+
1. Have the HTTP request function use the precomputed value instead of the
197205
slow running code it originally relied upon.
198-

‎content/pages/07-web-development/40-dramatiq.markdown‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
title: Dramatiq
2+
category: page
3+
slug: dramatiq
4+
sortorder: 0730
5+
toc: False
6+
sidebartitle: Dramatiq
7+
meta: Dramatiq is a fast and reliable alternative to Celery. It supports RabbitMQ and Redis as message brokers.
8+
9+
10+
# Dramatiq
11+
12+
[Dramatiq][dramatiq] is a fast and reliable Python [task queue][taskqueues]
13+
that came about as an alternative to Celery. It supports RabbitMQ and
14+
Redis as message brokers.
15+
16+
<div class="well see-also">Dramatiq is an implementation of the <a href="/task-queues.html">task queue</a> concept. Learn more in the <a href="/web-development.html">web development</a> chapter or view the <a href="/table-of-contents.html">table of contents</a> for all topics.</div>
17+
18+
## Dramatiq resources
19+
20+
* The [Dramatiq][dramatiq] documentation website.
21+
* The [source code][dramatiq-source].
22+
* [django_dramatiq][django_dramatiq] is a Django app for integrating
23+
Django and Dramatiq.
24+
* [django_dramatiq_example][django_dramatiq_example] is a simple
25+
Django app demoing how you can integrate Django and Dramatiq.
26+
* [flask_dramatiq_example][flask_dramatiq_example] is a simple app
27+
demoing how you can integrate Flask and Dramatiq.
28+
29+
30+
[taskqueues]: /task-queues.html
31+
[dramatiq]: https://dramatiq.io
32+
[dramatiq-source]: https://github.com/Bogdanp/dramatiq
33+
[django_dramatiq]: https://github.com/Bogdanp/django_dramatiq
34+
[django_dramatiq_example]: https://github.com/Bogdanp/django_dramatiq_example
35+
[flask_dramatiq_example]: https://github.com/Bogdanp/flask_dramatiq_example

‎theme/templates/choices/dramatiq.html‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
<h3>Do you want to learn more about task queues, or another topic?</h3>
2+
<div class="row">
3+
<div class="col-md-4">
4+
<div class="well select-next">
5+
{% include "choices/buttons/task-queues.html" %}
6+
</div>
7+
</div>
8+
<div class="col-md-4">
9+
<div class="well select-next">
10+
{% include "choices/buttons/deployment.html" %}
11+
</div>
12+
</div>
13+
<div class="col-md-4">
14+
<div class="well select-next">
15+
{% include "choices/buttons/logging.html" %}
16+
</div>
17+
</div>
18+
</div>

‎theme/templates/table-of-contents.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -178,6 +178,7 @@ <h4 class="toc-subsection">7.5 <a href="/javascript.html">JavaScript</a></h4>
178178
<h4 class="toc-subsection">7.6 <a href="/task-queues.html">Task queues</a></h4>
179179
<div class="toc"><a href="/celery.html">Celery</a></div>
180180
<div class="toc"><a href="/redis-queue-rq.html">Redis Queue (RQ)</a></div>
181+
<div class="toc"><a href="/dramatiq.html">Dramatiq</a></div>
181182
<div class="toc soon">Huey</div>
182183

183184
<h4 class="toc-subsection">7.7 <a href="/static-site-generator.html">Static site generators</a></h4>

0 commit comments

Comments
 (0)