Skip to content

Commit 4c122c6

Browse files
lambdapluslambdaplus
committed
add celery_learning.md
1 parent f9e4596 commit 4c122c6

1 file changed

Lines changed: 349 additions & 0 deletions

File tree

‎celery_learn.md‎

Lines changed: 349 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,349 @@
1+
# Celery 的简单使用
2+
3+
标签（空格分隔）： python celery
4+
5+
--------------------------------------------------------------------------------
6+
7+
** Celery 是一个简单、灵活并且可靠的处理大量消息的分发系统。并且是自带电池的，本身提供了维护和操作这个系统的工具。**
8+
9+
Celery 专注于实时处理的任务队列，并且支持任务调度。 优点：
10+
11+
1. 简单
12+
2. 高可用
13+
3. 快速
14+
4. 灵活
15+
16+
## Celery 架构
17+
18+
- Celery Beat: 任务调度器
19+
- Celery Worker: 消费者
20+
- Broker: 消息中间件，常用的是 RabbitMQ 和 Redis
21+
- Producer：任务生产者
22+
- Result Backend：用于结果保存。
23+
24+
## Celery 序列化
25+
26+
## 一个简单的简单例子
27+
28+
项目目录为
29+
30+
```bash
31+
celeries/proj/
32+
├── celeryconfig.py
33+
├── celery.py
34+
├── __init__.py
35+
└── tasks.py
36+
```
37+
38+
--------------------------------------------------------------------------------
39+
40+
主程序 celery.py
41+
42+
```python
43+
from __future__ import absolute_import
44+
from celery import Celery
45+
46+
app = Celery('proj', include=['proj.tasks'],
47+
app.config_from_object('proj.celeryconfig')
48+
49+
50+
if __name__ == "main":
51+
app.start()
52+
```
53+
54+
任务函数 tasks.py
55+
56+
```python
57+
# coding=utf-8
58+
from __future__ import absolute_import
59+
60+
from .celery import app
61+
62+
63+
@app.task
64+
def add(x, y):
65+
return x + y
66+
67+
68+
@app.task
69+
def mul(x, y):
70+
return x * y
71+
```
72+
73+
接下来是 配置文件 celeryconfig.py
74+
75+
```python
76+
# coding=utf-8
77+
BROKER_URL = 'amqp://localhost' # RabbitMQ 作为消息代理
78+
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0' # Redis 作为结果存储
79+
CELERY_TASK_SERIALIZER = 'msgpack'
80+
# 任务序列化和反序列化格式为 msgpack, 别忘了安装 msgpack-python
81+
CELERY_RESULT_SERIALIZER = 'json' # 结果存储序列化格式为 json
82+
CELERY_ACCEPT_CONTENT = ['msgpack', 'json'] # 任务接受格式类型
83+
```
84+
85+
因为没有任务调度，所以直接启动消费者就行了。在启动之前，要先去安装 RabbitMQ 和 Redis， 并启动。
86+
87+
现在启动我们的消费者函数, 命令行直接启动：
88+
89+
```
90+
> cd celeries
91+
> celery -A celeries worker -l info
92+
```
93+
94+
看到下面的提示信息，表示成功启动
95+
96+
```python
97+
-------------- celery@mouse-pc v4.0.2 (latentcall)
98+
---- **** -----
99+
--- * *** * -- Linux-4.9.15-1-MANJARO-x86_64-with-glibc2.2.5 2017-03-22 21:53:05
100+
-- * - **** ---
101+
- ** ---------- [config]
102+
- ** ---------- .> app: celeries:0x7f9737da7a58
103+
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
104+
- ** ---------- .> results: redis://localhost/
105+
- *** --- * --- .> concurrency: 2 (prefork)
106+
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
107+
--- ***** -----
108+
-------------- [queues]
109+
.> celery exchange=celery(direct) key=celery
110+
111+
112+
[tasks]
113+
. celeries.tasks.add
114+
. celeries.tasks.mul
115+
. celeries.tasks.xsum
116+
117+
[2017-03-22 21:53:06,011: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
118+
[2017-03-22 21:53:06,034: INFO/MainProcess] mingle: searching for neighbors
119+
[2017-03-22 21:53:07,088: INFO/MainProcess] mingle: all alone
120+
[2017-03-22 21:53:07,115: INFO/MainProcess] celery@mouse-pc ready.
121+
```
122+
123+
打开 IPython 测试一下我们的几个函数。
124+
125+
```python
126+
~ ▶︎︎ ipython
127+
Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 12:22:00)
128+
Type "copyright", "credits" or "license" for more information.
129+
130+
131+
In [1]: from celeries.tasks import add, mul, xsum
132+
133+
In [2]: add.delay(1, 9)
134+
Out[2]: <AsyncResult: 38022eec-2d3d-4ee0-8c7e-367ef92b5f1f>
135+
In [3]: r = mul.delay(2, 4)
136+
137+
In [4]: r.status
138+
Out[4]: 'SUCCESS'
139+
140+
In [5]: r.result
141+
Out[5]: 8
142+
143+
In [6]: r.successful
144+
Out[6]: <bound method AsyncResult.successful of <AsyncResult: 17af4e48-736d-44c9-a8be-a50a35bbc435>>
145+
146+
In [7]: r.backend
147+
Out[7]: <celery.backends.redis.RedisBackend at 0x7f5aebbbcba8> # 结果存储在 redis 里
148+
```
149+
150+
delay() 是 apply_async() 的快捷方式。你也直接调用 apply_async() ：
151+
152+
```python
153+
In [24]: r = mul.apply_async((2, 4))
154+
155+
In [25]: r.result
156+
Out[25]: 8
157+
```
158+
159+
delay() & apply_async 返回的都是 AsyncResult 实例，可用于查看任务的执行状态，但首先你要配置好 result backend. 此时，在worker终端上可以看到，任务信息和结果
160+
161+
```bash
162+
[2017-03-22 22:05:13,689: INFO/MainProcess] Received task: celeries.tasks.add[38022eec-2d3d-4ee0-8c7e-367ef92b5f1f]
163+
[2017-03-22 22:05:14,765: INFO/PoolWorker-2] Task celeries.tasks.add[38022eec-2d3d-4ee0-8c7e-367ef92b5f1f] succeeded in 0.007736653999018017s: 10
164+
[2017-03-22 22:08:36,378: INFO/MainProcess] Received task: celeries.tasks.mul[17af4e48-736d-44c9-a8be-a50a35bbc435]
165+
[2017-03-22 22:08:37,010: INFO/PoolWorker-2] Task celeries.tasks.mul[17af4e48-736d-44c9-a8be-a50a35bbc435] succeeded in 0.011531784999533556s: 8
166+
```
167+
168+
仔细看，每个任务都有一个 task_id。我们可以通过 task_id 获得任务的结果。
169+
170+
取 add 任务的 id：
171+
172+
```bash
173+
task_id = '38022eec-2d3d-4ee0-8c7e-367ef92b5f1f'
174+
In [8]: task_id = '38022eec-2d3d-4ee0-8c7e-367ef92b5f1f'
175+
176+
In [9]: add.AsyncResult(task_id).get()
177+
Out[9]: 10
178+
```
179+
180+
**关联任务**
181+
182+
```
183+
In [2]: m = mul.apply_async((2, 2), link=mul.s(3))
184+
```
185+
186+
在 Worker 终端里会看到两个值，关联之前和之后的。
187+
188+
```
189+
[2017-03-23 13:27:13,045: INFO/MainProcess] Received task: proj.tasks.mul[40492357-44bb-41e4-979f-6eb197107a5b]
190+
[2017-03-23 13:27:13,731: INFO/PoolWorker-2] Task proj.tasks.mul[40492357-44bb-41e4-979f-6eb197107a5b] succeeded in 0.0023383530005958164s: 4
191+
[2017-03-23 13:27:13,732: INFO/MainProcess] Received task: proj.tasks.mul[b01be1b8-f957-48b2-9d72-8187af6ac161]
192+
[2017-03-23 13:27:13,734: INFO/PoolWorker-2] Task proj.tasks.mul[b01be1b8-f957-48b2-9d72-8187af6ac161] succeeded in 0.0006868359996587969s: 12
193+
```
194+
195+
## 指定队列
196+
197+
在 celeries 目录下新建一个目录 projb, 代码使用 proj 中的。
198+
199+
```bash
200+
celeries/projb
201+
├── celeryconfig.py
202+
├── celery.py
203+
├── __init__.py
204+
└── tasks.py
205+
```
206+
207+
在 celeryconfig.py 添加些配置：
208+
209+
```
210+
# coding=utf-8
211+
from kombu import Queue
212+
213+
BROKER_URL = 'amqp://localhost' # RabbitMQ 作为消息代理
214+
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0' # Redis 作为结果存储
215+
CELERY_TASK_SERIALIZER = 'msgpack'
216+
# 任务序列化和反序列化格式为 msgpack, 别忘了安装 msgpack-python
217+
CELERY_RESULT_SERIALIZER = 'json' # 结果存储序列化格式为 json
218+
CELERY_ACCEPT_CONTENT = ['msgpack', 'json'] # 任务接受格式类型
219+
220+
CELERY_QUEUES = {
221+
Queue('foo', routing_key='task.#'), # 路由键以 task. 开头的消息进入此队列
222+
Queue('feed_task', routing_key='*.feed'), # 路由键以 .feed 结尾的消息进入此队列
223+
}
224+
CELERY_DEFAULT_QUEUE = 'foo' # 默认队列
225+
226+
CELERY_DEFAULT_EXCHANGE = 'tasks' # 默认交换机
227+
228+
CELERY_DEFAULT_EXCHANGE_TYPE = 'topic' # 默认交换机类型 topic
229+
230+
CELERY_DEFAULT_ROUTING_KEY = 'task.foooooooo' # 默认交换机路由键, task. 后的值不影响
231+
232+
CELERY_ROUTES = {
233+
'projb.tasks.mul': {
234+
'queue': 'feed_task', # 消息全都进入 feed_task 队列
235+
'routing_key': 'mul.feed',
236+
},
237+
}
238+
```
239+
240+
然后，我们以指定队列的方式启动：
241+
242+
```
243+
> celery -A projb worker -Q foo,feed_task -l info
244+
```
245+
246+
tasks.py 中的 mul 函数只会通过队列 feed_task 被执行。add 函数通过默认队列 foo 执行。
247+
248+
```python
249+
In [84]: from projb.tasks import mul, add
250+
251+
In [85]: r = add.delay(3, 3)
252+
253+
In [86]: r.result
254+
Out[86]: 6
255+
256+
In [87]: res = mul.delay(3, 3)
257+
258+
In [88]: res.result
259+
Out[88]: 9
260+
```
261+
262+
不过，我们可以使用 apply_async() 函数来指定队列。
263+
264+
```python
265+
In [90]: r = add.apply_async((3, 3), queue='feed_task', routing_key='mul.feed')
266+
267+
In [91]: r.result
268+
Out[91]: 6
269+
270+
In [92]: res = mul.apply_async((3, 3), queue='foo', routing_key='task.foooooo')
271+
272+
In [93]: res.result
273+
Out[93]: 9
274+
```
275+
276+
## 任务调度
277+
278+
依法炮制，基于 projb 的代码，创建目录 projc，在 proc/celeryconfig.py 中添加如下配置。
279+
280+
```
281+
CELERYBEAT_SCHEDULE = {
282+
'mul-every-30-seconds': {
283+
'task': 'projc.tasks.mul',
284+
'schedule': 30.0,
285+
'args': (2, 2),
286+
}
287+
}
288+
```
289+
290+
执行
291+
292+
```
293+
> celery -B -A projc worker -l info
294+
```
295+
296+
就可以在终端看到每 30s 执行一次任务。
297+
298+
```
299+
[2017-03-23 12:23:13,920: INFO/Beat] Scheduler: Sending due task mul-every-30-seconds (projc.tasks.mul)
300+
[2017-03-23 12:23:13,923: INFO/MainProcess] Received task: projc.tasks.mul[9c414257-d627-4c36-a9d8-9daed7e295c0]
301+
[2017-03-23 12:23:15,177: INFO/PoolWorker-3] Task projc.tasks.mul[9c414257-d627-4c36-a9d8-9daed7e295c0] succeeded in 0.0010301589991286164s: 4
302+
```
303+
304+
## 任务绑定、日志记录和错误重试
305+
306+
任务绑定、记录日志和重试是 Celery 3 个常用的高级功能。接下来，修改 proj 的 tasks.py 文件。添加一个 div 函数。
307+
308+
```
309+
@app.task(bind=True)
310+
def div(self, x, y):
311+
logger.info(
312+
'''
313+
Executing task : {0.id}
314+
task.args : {0.args!r}
315+
task.kwargs : {0.kwargs!r}
316+
'''.format(self.request)
317+
)
318+
try:
319+
res = x / y
320+
except ZeroDivisionError as e:
321+
raise self.retry(exc=e, countdown=3, max_retries=3)
322+
else:
323+
return res
324+
```
325+
326+
在 Ipython 调用：
327+
328+
```
329+
In [3]: d = div.delay(2, 1)
330+
```
331+
332+
在 worker 中可以看到
333+
334+
```
335+
[2017-03-23 14:57:17,361: INFO/PoolWorker-2] proj.tasks.div[68ef1584-16ac-4236-9858-b00842891bbc]:
336+
Executing task : 68ef1584-16ac-4236-9858-b00842891bbc
337+
task.args : [2, 1]
338+
task.kwargs : {}
339+
340+
[2017-03-23 14:57:17,369: INFO/PoolWorker-2] Task proj.tasks.div[68ef1584-16ac-4236-9858-b00842891bbc] succeeded in 0.007741746998362942s: 2.0
341+
```
342+
343+
换成可以引起异常的参数：
344+
345+
```
346+
In [4]: d = div.delay(2, 0)
347+
```
348+
349+
可以看到，在 worker 中每 3s 重试一次，总共重复三次(执行了 4 次)，然后抛出异常！

0 commit comments

Comments
 (0)