52 questions
1
vote
1
answer
39
views
Launch Celery Task from Database Object Method
In Miguel Grinberg's Flask Mega-Tutorial, chapter 22, he creates a generic launch_task method as a part of the User object model. His tutorial uses rq and I am trying to do something similar with ...
- 487
0
votes
1
answer
118
views
Scheduling periodic RQ tasks, using asyncio
I think that the architecture of RQ Scheduler is fundamentally flawed and it's much more complicated than it needs to be.
Schedules are stored in Redis
Even if you remove or modify your scheduling ...
- 4,674
1
vote
0
answers
250
views
rq worker error on Windows: AttributeError: module 'os' has no attribute 'fork'
I'm using FastAPI with Redis and RQ for background task processing. When I try to start the RQ worker using the following command:
rq worker --url <my redis url> task_queue
I receive the ...
0
votes
0
answers
129
views
Integrating OpenCV with FastAPI and Redis RQ Causes Task/Job Failures
I'm working on integrating a set of microservices into our application, focusing on a FastAPI server that interacts with Redis through RQ for task queueing. The setup involves three main files:
...
- 49
1
vote
0
answers
56
views
Can I use django RQ to delay execution of tasks like saving models to the database? When I try It causes weird issues
I have the following code:
@job
def create_model():
test_text = "Text1"
TestModel.objects.create(test_text=test_text)
print("Done!")
def test_view(request):
...
- 33
0
votes
0
answers
83
views
Task queues with Flask - Redis server quitting before executing
I'm tryng to set up a redis server in order to run a simple task through RQ
I start a RQ worker from the VSC terminal, then open a new terminal, kick in a py shell session and type the following:
>&...
- 1
1
vote
2
answers
564
views
RQ is unable to find the task using string like function ref
I've have a fastapi server and I want to implement background task using Redis and RQ.
I've created FASTAPI_SERVER, REDIS_SERVER and WORKER_SERVER in docker-compose.
Here is what my fastapi code looks ...
0
votes
0
answers
41
views
Python RQ csv file processing
What is more preferable way to process .csv files using RQ? File possible can have few or few hundred thousands rows
def process_row():
pass
# Is it better to push function per csv row as a job.
...
- 1
0
votes
1
answer
522
views
No Redis configuration! error for simple flask app using rq-dashboard
rq-dashboard works from the command line, but not integrated into flask.
The simple example from github
from flask import Flask
import rq_dashboard
app = Flask(__name__)
app.config.from_object(...
- 225
0
votes
0
answers
161
views
Fast Api Rq job vs Normal Api call
I am new to python Rq and I am trying to understand his usage in FastApi. Why it would be better to use a Rq job instead a normal api call?
In other words, why it would be better to do:
@app.post('/')
...
0
votes
1
answer
799
views
How to retry an Redis Queue job on a different queue?
I am using Python with RQ to manage jobs. Occasionally a job will fail for a variety of reasons, usually either insufficient memory or timeout. I would like to have these jobs retried but with more ...
- 31
2
votes
2
answers
1k
views
RQ Job Terminated Unexpectedly
I have defined the rq task on module task.py as:
import django_rq
import logging
import requests
log = logging.getLogger(__name__)
def req_call():
url = 'https://jsonplaceholder.typicode.com/...
0
votes
1
answer
546
views
rq worker unable to connect to EC2 through Heroku app (SSL: CERTIFICATE_VERIFY_FAILED)
My Heroku app uses Redis and an rq worker. I recently enabled SSL on my Heroku app via the dashboard, and was met with SSL errors. Initially I was met with the following error when hitting an endpoint ...
- 776
2
votes
0
answers
1k
views
How to handle ConnectionError error with redis-py connection while Redis pod (on kubernetes) is restarting?
I have a problem with redis-py connection into redis on kubernetes. Few times a day redis pod restarts/moves on new one pod, but my python processes catches ConnectionError error:
I know, that it ...
- 51
0
votes
1
answer
408
views
How to pause and resume RQ task
I want to enqueue the RQ task, to complete a task, suppose it takes around 10 minutes, Now i want to pause any task if it continues for 5 minutes and start another task and after another 5 minutes and ...
- 140
4
votes
1
answer
2k
views
Redis Queue task is queued but never executed
I am having issues getting RQ-python to run. Like in the example of the documentation (https://python-rq.org) I have a function in an external file
def createAndSaveIndex(url_list, index_path):
...
- 51
0
votes
1
answer
116
views
TypeError: cannot pickle 'LockType' object
I am trying to create a flask application with rq redis which stores tasks that returns the data scraped by playwright library.
What I am trying to do is to create a global browser instance of ...
- 140
0
votes
1
answer
1k
views
Upgrading Redis to 7.0.11 results in TypeError at /login/ __init__() got an unexpected keyword argument 'ssl_cert_reqs'
We use redis with django-rq to manage queues on our project. Heroku has recently forced upgrades to Redis 7.0.11 and that requires TLS. Per their docs we need to set 'ssl_cert_reqs'=None in order to ...
0
votes
1
answer
216
views
Django race conditions between create() and get()
Since our last release a range of what seems can only DB race conditions have been happening.We use python 3.10, django-rq, Django 4 and Postgres.
In one case:
x = Model.objects.create()
job.delay(...
- 603
0
votes
1
answer
97
views
Django-RQ API_TOKEN configuration to access to stats endpoint
I'm trying to retrieve some statistics from the django-rq queues and I'm having some difficulties to access to the provided URL from the Django-Rq documentation
These statistics are also available in ...
- 21
1
vote
0
answers
114
views
Multiple Background Jobs Redis Queue
I am trying to implement a fast API app in Heroku.
A user has the option to send a keyword with the API endpoint.
This triggers a background job (web scraping with the keyword).
If there are multiple ...
- 21
1
vote
0
answers
148
views
Test email does not arrive in mail.outbox when sent by Django-RQ worker
I've set up a test within Django that sends an email, in the background, using Django-RQ.
I call the code that enqueues the send_email task, then, I get the django-rq worker, and call .work(), with ...
- 1,401
-1
votes
1
answer
218
views
Sending Confirmation Emails with Flask, RQ (Working outside of application context.)
I am trying to send confirmation emails when a new user registers. When I send a confirmation email, everything works fine, however, when I try to run that task in the background I get the following ...
1
vote
2
answers
610
views
How to run Python RQ jobs on a DigitalOcean application sever?
I have a FastAPI application deployed on DigitalOcean, it has multiple API endpoints and in some of them, I have to run a scraping function as a background job using the RQ package in order not to ...
1
vote
2
answers
3k
views
Railway.app: Is Procfile Successfully Loading a Worker?
Migrating from Heroku to Railway.app: Python Flask app with Redis and Postgres. Using Redis as an asynchronous job queue, with the RQ Redis queue python library.
Procfile, which works in dev, looks ...
- 53
0
votes
1
answer
716
views
Flask and rq-scheduler: Working outside of application context
I want to start a background job that does a periodic sync when I start the flask application.
This works well, but in the task function I want to access a logger that was defined outside. While this ...
- 11
3
votes
1
answer
537
views
rq info showing job_id (None None):?
Whenever we are trying to see queue info using rq info -u <<redis_url>> command, we are getting lots of extra entries like this -
a331d42408099f7e5ec9c5864 (None None): ?
...
- 7,439
0
votes
1
answer
584
views
When trying to use a rq worker get rq.exceptions.DeserializationError
When trying to run django-rq worker I keep getting this error
15:35:26 Traceback (most recent call last):
File "/home/john/PycharmProjects/api.to/venv/lib/python3.8/site-packages/rq/job.py"...
- 2,954
0
votes
0
answers
420
views
Django RQ ModuleNotFoundError xxx is not a package
I'm using django rq for some long running task, which I would like to provide some feedback about its progress. I hence created a function which is enqueued as a new task and set some model's ...
- 59
1
vote
1
answer
2k
views
Python RQ and Redis Server: How does the worker know the function code to execute?
So I would like to have the following architecture:
Flask API (VM 1) -> Redis Server (VM 2) <- Worker (VM 3)
I have seen some examples and the idea I get is that on the API you define the tasks ...
- 11
0
votes
1
answer
1k
views
Redis queue Retry does not work with the interval argument
I am trying to use the rq Retry functionality by following the rq documentation but it does not work when using the interval argument
python version: 3.8.0
rq version: 1.10.0
The somewhere.py
def ...
- 2,039
0
votes
0
answers
562
views
Creating a decorator for python rq to queue tasks
I'm trying to create a decorator that will automatically queue certain functions.
import functools
from rq import Queue
from worker import conn
q = Queue(connection=conn)
def fire_and_forget(func):
...
- 1,274
0
votes
2
answers
442
views
Can we retrieve the parameters originally passed in scheduler.Schedule() (Like Args) in RQ scheduler?
What i want:
I am trying to retrieve the parameters i passed to my job in rq scheduler while scheduling like:
args passed, Function name, and other things
What i have already tried:
list_of_jobs=...
- 55
3
votes
0
answers
1k
views
RQ: success and failure handling when using `@job` decorator
I am using @job decorator to define a task as below:
rqueue = Queue(connection=Redis.from_url(REDIS_URL))
@job(queue=rqueue)
def add(x, y):
time.sleep(2)
return x+y
this works fine when I ...
- 756
0
votes
0
answers
196
views
What is a better way to create a redis - rq microservice container, to publish events from other independent containers?
If we use microservice architecture, ideally a single function definition should be at a single place.
So if I have a microservice containing the subscriber function.
And I want to publish an event to ...
- 441
0
votes
1
answer
550
views
Python rq module: AttributeError: 'tuple' object has no attribute '__module__'
I am trying to test post requests to a Flask server, with a queue generated with rq module.
Rq's queue keeps prompting this weird error:
Traceback (most recent call last):
File "/home/dor/...
- 11
1
vote
0
answers
234
views
Requeueing specific job by ID django_rq
I can't find a straight answer, is it possible to retry a job given just the job's redis id?
I want to make an endpoint in Django that can manually retry a specific job, because I don't always need ...
- 792
0
votes
0
answers
1k
views
TypeError: cannot pickle '_thread.lock' object with RQ (Redis Queue) [python 3.9 & 3.8 & 3.7]
I have been trying to use rq to queue API requests from BigQuery since they take so much time I get an H12 (Timeout) Error. The code keeps breaking down when a pass a dataframe to the next enqueue.
...
- 1
0
votes
2
answers
1k
views
This is not work Flask Rq Queue with SqlAlchemy database table
I Have a Flask project. Process are work normal. Database is PostgreSql. Everything is working on this Database. But when I Can use Python Rq I dont reach tables with SqlAlchemy. My project is on ...
0
votes
0
answers
265
views
How to use PostgreSQL with SQLAlchemy in a Flask application with Redis queued job
When executing SQL queries using SQLAlchemy directly on API calls in Flask app they are executed successfully but when I try to perform one inside my function that gets executed using Redis job queue ...
1
vote
1
answer
535
views
Flask context for RQ jobs (RuntimeError: Working outside of application)
First of all, questions about flask_context including context for RQ jobs seems to be common issue but I searched a lot and still couldn't solve my problem.
My decorator functions (tried both of them ...
0
votes
1
answer
2k
views
Scheduling/Queueing a job within another job with redis queue
Tasker class setups the initial job when instantiated. Basically what I want is put a job in the 'main_queue', decide if job is running or if there is already same job that is queued in the '...
- 682
0
votes
0
answers
314
views
Flask redis queue (RQ) worker update parent process from Vault
I'm working on the project which building system on k8s and running two separated process on two different pods (provider and server), one pod (provider) take responsible for collecting data, other (...
- 1
3
votes
1
answer
616
views
rq-dashboard on Heroku
I'm trying to get rq-dashboard to work on my Redis server created through Heroku. I added it to my Flask app using
from flask import Flask
import rq_dashboard
app = Flask(__name__)
app.config....
- 849
1
vote
0
answers
366
views
Approach to animate a progress bar using python/flask/bootstrap/rq/redis [duplicate]
I am using flask in an application that does some long-running computations. I've successfully offloaded the compute-intensive code to an rq worker with this code
job = q.enqueue(run_simulation)
This ...
- 213
2
votes
0
answers
3k
views
I am getting a "ValueError: unsupported pickle protocol: 5" when trying to use Redis rq
I am trying to get Redis and rq working.
This is my main code:
from rq import Queue
from redis import Redis
from util import count_words_at_url
from rq.registry import FailedJobRegistry, Job
import ...
- 213
0
votes
1
answer
2k
views
Setting environment variables with Redis Queue
I am new at RQ and I am trying to implement it within my flask app. The main goal of one of my routes is to update values from a database. To setup my worker I am using the following:
from rq import ...
- 427
0
votes
1
answer
129
views
RQ-Jobs Failing for No reason
This is info for simple print function.
Jobs with any function is failing without no distinct reason.
Status: failed
Job: <Job fdee8b46-3a9a-4014-b82f-370616bf3325: ams.ams(<Request 'http://172....
0
votes
1
answer
917
views
RQ task from Flask keeps adding to queue
My setup is:
Running Redis on docker locally
running RQ in virtualenv(rq)
running Flask in virtualenv(rq)
Goal is to launch RQ tasks by hitting the Flask app.
Redis seems to running fine, as does ...
- 3,852
3
votes
1
answer
2k
views
Proper way to pass python flask app context to rq jobs
I've created a flask app that is effectively a news search app. It currently serves up a number of routes, integrates with elasticsearch for document searching and indexing and integrates with a mySQL ...
- 123