14,813 questions
0
votes
0
answers
34
views
Multithreading becomes much slower than multiprocessing in free-threaded Python
Consider the following executable Python script mtmp.py:
import numpy as np
import os
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
# `x` is referenced `n_loop` times.
def ...
- 1,810
0
votes
0
answers
33
views
Readers-writer spinlock implementation for lab locks MIT OCW OS
I am trying to learn some operating system engineering so I came about MIT's operating system engineering course which has been wonderful so far.
The problem I'm having is with the second problem read-...
- 131
-3
votes
0
answers
30
views
Event loop vs cooperative scheduler [closed]
After learning about event loops (like in js or python) and Scheduler types, I am under the expression that an event loop is basically a simpler cooperative scheduler, with each task being able to ...
- 310
1
vote
0
answers
56
views
Can I use Celery for a task which calls an external library that uses multiprocessing?
I am hoping to use Celery to manage the task queue for my application, and am wondering if it is capable of managing tasks that themselves use multiprocessing, called from an external library. For ...
- 25
2
votes
1
answer
74
views
asyncio freezes when spawning multiprocessing process on Windows but works fine on Linux
I’m experimenting with mixing asyncio and multiprocessing in Python 3.12.
On Linux, the following code works as expected — the event loop stays responsive and the child process prints normally.
On ...
1
vote
1
answer
79
views
How to share a large CustomObject to workers in Python multiprocessing on Windows (spawn)?
I'm trying to run calculations using multiple cores in Python on multiple platforms (Linux, macOS, Windows). I need to pass a large CustomClass Object and a dict (both readonly) to all workers. So far ...
- 459
2
votes
2
answers
107
views
How does multiprocessing.Pool() create its child processes?
Overview:
I am trying to use a Pool internally in a module that is not __main__ and make it invisible to main that this pool exists.
Because of this, if __name__ == "__main__": protection is ...
- 23
3
votes
1
answer
95
views
JAX crashes with `CUDNN_STATUS_INTERNAL_ERROR` when using `joblib` or `multiprocessing`, but works in a single process
I am running into a FAILED_PRECONDITION: DNN library initialization failed error when trying to parallelize a JAX function using either Python's multiprocessing library or joblib.
The strange part is ...
0
votes
2
answers
181
views
Process in parallel multiple .ndjson files inside a .tar.gz file
I have a file enwiktionary_namespace_0.tar.gz that contains 86 .ndjson files
enwiktionary_namespace_0_0.ndjson
enwiktionary_namespace_0_1.ndjson
enwiktionary_namespace_0_2.ndjson
...
...
- 2,886
1
vote
0
answers
82
views
Parent process unexpectedly exits during debugging when child receives SIGTERM (works fine outside debugger)
I'm experiencing an issue where the parent Python process terminates unexpectedly when debugging, but only when a child process is sent SIGTERM. The same code works perfectly when run normally (...
- 35
1
vote
1
answer
103
views
Brightway 'bw2io.import_ecoinvent_release()' failing unless I disable multiprocessing
I tried to run bw2io.import_ecoinvent_release() to import ecoinvent as I used to, but the function is now stuck in a loop and return error messages indefinitely (see below).
What I run:
import bw2data ...
0
votes
1
answer
94
views
Python gets stuck in an infinite loop restarting multiprocessing pool workers on error in initilization routine
I am trying to setup a multiprocessing Python task on Windows 10, Python 3.13. I have "main.py" module, containing the main entry, "orchestration.py" module with worker ...
- 1,869
0
votes
1
answer
30
views
Can I avoid setting-up and tearing down processes when using PyTorch DataLoader?
In my scenario I use multiple DataLoaders with multiple Datasets to evaluate models against each other (I want to test models with multiple resolutions, which means each dataset has a distinct ...
- 3,598
1
vote
1
answer
188
views
is jax really incompatible with python multiprocesses? [closed]
I have a simple app with a main controller process, and a child process that handles API calls. They communicate using Python queues.
The app looks (something) like this:
import multiprocessing as mp
...
- 39
1
vote
2
answers
114
views
Why does NumPy load() take longer in a larger script than in another?
I scan for a trigger and when I get it, load a .npy file and process it. It started to take almost 2 seconds to load the NumPy file from within the process but when I tried to load the same file from ...
- 21
3
votes
1
answer
132
views
Isolation of a custom multiprocessing manager and how to update internal state
I try to use a custom multiprocessing manager, mostly following the example from the docs. The main difference is that my class updates internal state. It looks like this:
class IdIndex:
def ...
- 15.7k
2
votes
1
answer
86
views
Python multiprocessing parallelization
I have a class with methods to simulate sources across 16 detectors using the Gelsa package. In my main script, I call the method generate.sources. I am trying to use multiprocessing to speed up the ...
- 21
4
votes
2
answers
241
views
Python multiprocessing shared memory seems to hang or crash when interacting with small numpy arrays
I've been trying to parallelize some code that I wrote in python. The actual work is embarrassingly parallel, but I don't have much experience with multiprocessing in Python.
The actual code I'm ...
- 116
0
votes
0
answers
97
views
XCB error when using Python with Matplotlib and Multiprocessing
I ran into a problem when I'm trying to plot figures with matplotlib in separate processes using multiprocessing. I tried it on OpenBSD 7.7 with Python 3.12.11 and on Debian 12 with Python 3.11.2 and ...
- 1
0
votes
1
answer
68
views
subprocess.run() eventually reaches python's global thread limit
I have a python script that pings another device every so often.
PING_SIZE = "1"
PING_RETRYS = "1"
while True:
try:
result = subprocess.run(
["ping&...
2
votes
2
answers
166
views
How to safely check a multiprocessing.Event in an asyncio loop without blocking?
I'm working on an asynchronous server using asyncio, but I need to monitor a multiprocessing.Event (used to signal termination from another process) inside my async event loop. Here's the simplified ...
3
votes
0
answers
129
views
What is the best/correct way to handle shared memory objects in a heterogeneous multi-CPU embedded microprocessor context [closed]
I'm working on several projects that are targeting bare-metal multi-core microprocessors, with hardware provisions for shared memory.
In other words, the device has several separate CPU cores (of ...
- 5,943
3
votes
1
answer
77
views
Accessing any object type from multiprocessing shared_memory?
Suppose I create a shared memory object:
from multiprocessing import shared_memory
shm_a = shared_memory.SharedMemory(create=True, size=1024)
buffer = shm_a.buf
and put a generic object of a generic ...
- 5,844
3
votes
4
answers
168
views
Where should I allocate the pool in Python multiprocessing?
I want to distribute a python app over several kernels and from the documentation I understand that a pool is the way to do this.
My problem can be reproduced by the following code:
#!/usr/bin/python3
...
- 91
1
vote
1
answer
69
views
Сonnection timeout for function does not seem to work in Python script
for some reason the connection timeout does not seem to work when connecting to a Firebird database. For example, if an incorrect IP address is specified, the script hangs and waits for a long time.
#!...
- 11
2
votes
2
answers
157
views
Setting global variables for python multiprocessing
I have a large array and an object I'd like to call multiple times with multiprocessing. Neither the data nor the object internals get modified.
This works:
import numpy as np
from multiprocessing ...
- 995
2
votes
1
answer
175
views
Starting multiprocessing.Process() in a Pytest test/Python creates a Windows fatal exception: access violation
On windows, running the below code with Python or pytest makes it print out Windows fatal exception: access violation (but the script will continue with no issue)
Reproduction:
import multiprocessing ...
- 694
0
votes
1
answer
89
views
Cannot pickle weakref error in the multiprocessing in a class in Python
In the project I am currently working on, I am parsing a json file and according to client list, I need to start processes in parallel. However there might be the case where a inside a client list, I ...
- 13
0
votes
0
answers
96
views
FastAPI becomes unresponsive while running partition_pdf from unstructured in a background process
I have a FastAPI application where one of the API endpoints needs to parse PDF files using unstructured.partition_pdf. Since the parsing is CPU-heavy, I want to run it in the background so that the ...
1
vote
0
answers
114
views
Windows Python 3.12 Multiprocessing Deadlock
I'm having trouble getting this example working. In serial this function works just fine, but when I attempt to run it in a multiprocessing.Pool it locks up and will not return a simple random integer....
- 51
0
votes
0
answers
56
views
How do I sucessfully stop TCPDump recording?
I am very new to python. My current project is to create a basic Tkinter window to start and stop a TCPdump Ethernet recording using a couple of buttons. I have been able to start a recording using a ...
- 1
0
votes
1
answer
76
views
How should CPU time be computed for calculations parallelized with the multiprocessing module?
I am trying to measure the processing time, or CPU time, of a CPU-intensive computation that has been parallelized with multiprocessing. However, simply bookending the parallelization of the ...
- 437
2
votes
0
answers
64
views
Overthreading in Multiprocessing for ML models
The problem
I am running machine learning models in parallel using multiprocessing. When using models with parameters stating the number of threads used- num_threads, num_jobs, etc. - the code works ...
- 353
1
vote
0
answers
140
views
Python processes arent being evenly distributed over cpu nodes
I wasn't able to find anything regarding this on the internet: I am using multiprocessing (concurrent.futures. ProcessPoolExecutor(max_workers=(...)) as executor) to execute several DRL training ...
- 11
2
votes
2
answers
89
views
Is it possible to create numpy array in shared_memory using custom dtypes?
I can succesfully create numpy arrays with simple dtypes (int32 etc).
But when i try to use something like :
b_shared_memory = shared_memory.SharedMemory(create=True, name = "test235", size =...
- 21
0
votes
2
answers
62
views
Python threads killed when main thread times out with only one thread, but survive when more than one threads are created
I have a Python function decorated with a 1-minute timeout. Inside it I spawn one or more worker threads that each run a 2-minute sleep via a dynamically executed script (using exec() in a custom ...
2
votes
1
answer
52
views
Avoid PicklingError using multiprocessing in Python 3.8 libreoffice macro
I'm trying to use multiprocessing in a LibreOffice Macro in Python 3.8. When I create a Macro it is part of the 'ooo_script_framework' module. So, when the process is pickled it is in fact part of an ...
0
votes
2
answers
103
views
Pausing, unpausing, and stopping threads in python
I am trying to make a program that has equivalents for another module's code.
The module has it's own thread class, with the commands 'stop', and 'sleep_for'
For example, using this module, you could ...
2
votes
0
answers
83
views
Why is training multiple deep learning models (via multiple terminals) faster than training a single model? Is it because higher clock speeds?
I'm training deep learning models using TensorFlow (with GPU support) on my local machine. I noticed a surprising behavior:
When I train just one model (in a single terminal), it runs slower.
But ...
- 77
0
votes
1
answer
122
views
How to run multiple potentially multithreaded shell commands in python
I am trying to write a Python script to run another potentially multithreaded script in a way to further parallelise them.
Consider a (C++) executable which can be run as run_task <sample_number>...
0
votes
0
answers
75
views
How to speed up Keras Tuning on multi core machine?
This question is related to the Distributed hyperparameter tuning strategy of keras tuner to get the possible hyperparameters for my model. The search space is huge. The worker thread is as in the ...
- 173
0
votes
1
answer
197
views
How do I efficiently write results from python's ProcessPoolExecutor to a single parquet file?
I use python to read a large csv file, transform the data within (mostly string operations), and then write the results to a single parquet file. In the transformation process, rows are independent. 1 ...
- 132
0
votes
0
answers
56
views
Unknown DDP error when running multi-processing with pytorch on a Google GPU based colab kernel
I have created a colab (link: https://colab.research.google.com/drive/1gg57PS7KMLKvvx9wgDKLMDiyjhRICplp#scrollTo=zG2D7JO2OdEC) to play with the gpt2 fine-tuning.
And I was trying to practice the DDP ...
- 11
0
votes
0
answers
88
views
Sending large graph of size
client=Client()
ds=xr.open_dataset(file_paths[0], chunks='auto')
time_data = ds['ObservTimes'].compute()
As long as the compute operation is performed, the following warning will appear：
UserWarning: ...
- 1
1
vote
0
answers
61
views
Is there a reason not to replace the default logging.Handler lock with a multiprocessing.RLock to synchronize multiprocess logging
I've got some code that, for reasons not germane to the problem at hand:
Must write very large log messages
Must write them from multiple multiprocessing worker processes
Must not interleave the logs ...
- 158k
0
votes
1
answer
88
views
Python code coverage not working for multiprocessing inside site-packages
I have a simple code packages in a module/folder "src"
File sample.py
import multiprocessing
def f(x):
return x*x
def big_run(n):
with multiprocessing.Pool(5) as p:
p.map(f,...
1
vote
0
answers
152
views
How to trace PyTorch Dataloader workers with VizTracer?
I'm using VizTracer to debug performance issues in my PyTorch data loading pipeline. Specifically, I'm using a DataLoader with num_workers > 0 to load data in parallel using multiple subprocesses.
...
- 53
2
votes
1
answer
69
views
Can I use sa-learn from Spamassassin in parallel with mutliple CPUs?
I am writing a bash script for using Spamassassin.
I have a single folder with loads of emails stored as text files. (1744408703.1184229_99.txt). I want to feed all of them to Spamassassin through sa-...
- 387
1
vote
0
answers
45
views
Image Tensors Return As Zero When num_workers > 0
I am facing an issue with multiprocessing. I am trying to load my .pt data as dataloaders. Everything works fine when I set the num_workers = 0. But when I set it to a value greater than 0, the tensor ...
- 11
0
votes
0
answers
62
views
Why does my Python multiprocessing code run slower than the single-threaded version?
I'm working on a CPU-bound task in Python and attempted to speed it up using the multiprocessing module. I expected performance improvements due to parallel execution, but instead, the multiprocessing ...
