14,796 questions
1
vote
1
answer
59
views
Combining asyncio and multiprocessing queues
I have a task where I have to fetch a lot of files from a database (I/O-bound) and process each of them (CPU-bound). I thought of using a producer-consumer pattern, where fetch workers (producers) ...
- 33
3
votes
1
answer
82
views
How to robustly intercept PyTorch GPU OOM in a Python subprocess and dynamically adjust batch_size within an autonomous AI Agent loop?
I am building an autonomous AI Agent (managing training workflows) that automatically generates PyTorch/OpenMMLab training scripts and executes them in a background subprocess.
One of the common ...
- 31
0
votes
1
answer
56
views
Avoid re-running tests when using unittest with multiprocessing
We recently upgraded from Python 3.9 to 3.14 and it appears that the default start method for multiprocessing has switched from 'fork' to 'spawn' and it is causing some tests to fail, trying to ...
- 269
0
votes
0
answers
28
views
Difference in Measuring Latency of (torch.)Multiprocessing.Queue
I'm writing a program that requires transferring GPU tensors across processes in a pipeline manner.
I knew that using torch.Multiprocessing would automatically get the CudaIpcMemHandle for me and send ...
- 45
Advice
0
votes
4
replies
80
views
Understanding "I/O depth" aka "queues" (e.g. in the fio "flexible I/O tester")
I've been going a bit down a rabbit hole that started when I tried to get a real-world performance comparison of 2 SSD drives, under Linux. I discovered KDiskMark, which tries very hard to emulate ...
- 860
Best practices
2
votes
7
replies
154
views
How to add many fractions together as fast as possible?
This is related to this question.
Basically, I need to add many fractions together. Seems simple enough? Except it isn't simple.
I can't use fractions.Fraction class, that thing stupidly does fraction ...
- 4,171
0
votes
0
answers
41
views
torch dataloader next-method when using multiple workers
I have a Dataset that is based on IterableDataSet, looking like that
class MyDataSet(torch.utils.data.IterableDataset):
def __init__(self):
# doing init stuff here
def __iter__(self):
...
- 1,597
0
votes
1
answer
129
views
NLog 6.0.7 Multiple archives created in multi-process logging setup
I have a multi-process application (one orchestrator and multiple workers) where all processes write to the same log file using NLog 6.
The log file is configured to be archived:
daily (ArchiveEvery=...
14
votes
3
answers
717
views
Multithreading becomes much slower than multiprocessing in free-threaded Python
Consider the following executable Python script mtmp.py:
import numpy as np
import os
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
# `x` is referenced `n_loop` times.
def ...
- 1,962
2
votes
0
answers
92
views
Readers-writer spinlock implementation for lab locks MIT OCW OS
I am trying to learn some operating system engineering so I came about MIT's operating system engineering course which has been wonderful so far.
The problem I'm having is with the second problem read-...
- 141
1
vote
1
answer
79
views
Can I use Celery for a task which calls an external library that uses multiprocessing?
I am hoping to use Celery to manage the task queue for my application, and am wondering if it is capable of managing tasks that themselves use multiprocessing, called from an external library. For ...
- 25
2
votes
1
answer
114
views
asyncio freezes when spawning multiprocessing process on Windows but works fine on Linux
I’m experimenting with mixing asyncio and multiprocessing in Python 3.12.
On Linux, the following code works as expected — the event loop stays responsive and the child process prints normally.
On ...
1
vote
0
answers
98
views
How to share a large CustomObject to workers in Python multiprocessing on Windows (spawn)?
I'm trying to run calculations using multiple cores in Python on multiple platforms (Linux, macOS, Windows). I need to pass a large CustomClass Object and a dict (both readonly) to all workers. So far ...
- 459
2
votes
2
answers
118
views
How does multiprocessing.Pool() create its child processes?
Overview:
I am trying to use a Pool internally in a module that is not __main__ and make it invisible to main that this pool exists.
Because of this, if __name__ == "__main__": protection is ...
- 23
4
votes
1
answer
170
views
JAX crashes with `CUDNN_STATUS_INTERNAL_ERROR` when using `joblib` or `multiprocessing`, but works in a single process
I am running into a FAILED_PRECONDITION: DNN library initialization failed error when trying to parallelize a JAX function using either Python's multiprocessing library or joblib.
The strange part is ...
0
votes
1
answer
193
views
Process in parallel multiple .ndjson files inside a .tar.gz file
I have a file enwiktionary_namespace_0.tar.gz that contains 86 .ndjson files
enwiktionary_namespace_0_0.ndjson
enwiktionary_namespace_0_1.ndjson
enwiktionary_namespace_0_2.ndjson
...
...
- 2,858
1
vote
0
answers
92
views
Parent process unexpectedly exits during debugging when child receives SIGTERM (works fine outside debugger)
I'm experiencing an issue where the parent Python process terminates unexpectedly when debugging, but only when a child process is sent SIGTERM. The same code works perfectly when run normally (...
- 35
2
votes
1
answer
133
views
Brightway 'bw2io.import_ecoinvent_release()' failing unless I disable multiprocessing
I tried to run bw2io.import_ecoinvent_release() to import ecoinvent as I used to, but the function is now stuck in a loop and return error messages indefinitely (see below).
What I run:
import bw2data ...
0
votes
1
answer
100
views
Python gets stuck in an infinite loop restarting multiprocessing pool workers on error in initilization routine
I am trying to setup a multiprocessing Python task on Windows 10, Python 3.13. I have "main.py" module, containing the main entry, "orchestration.py" module with worker ...
- 1,895
0
votes
1
answer
43
views
Can I avoid setting-up and tearing down processes when using PyTorch DataLoader?
In my scenario I use multiple DataLoaders with multiple Datasets to evaluate models against each other (I want to test models with multiple resolutions, which means each dataset has a distinct ...
- 3,608
0
votes
1
answer
349
views
is jax really incompatible with python multiprocesses? [closed]
I have a simple app with a main controller process, and a child process that handles API calls. They communicate using Python queues.
The app looks (something) like this:
import multiprocessing as mp
...
- 29
0
votes
2
answers
119
views
Why does NumPy load() take longer in a larger script than in another?
I scan for a trigger and when I get it, load a .npy file and process it. It started to take almost 2 seconds to load the NumPy file from within the process but when I tried to load the same file from ...
- 11
3
votes
1
answer
135
views
Isolation of a custom multiprocessing manager and how to update internal state
I try to use a custom multiprocessing manager, mostly following the example from the docs. The main difference is that my class updates internal state. It looks like this:
class IdIndex:
def ...
- 15.7k
1
vote
1
answer
93
views
Python multiprocessing parallelization
I have a class with methods to simulate sources across 16 detectors using the Gelsa package. In my main script, I call the method generate.sources. I am trying to use multiprocessing to speed up the ...
- 11
4
votes
2
answers
306
views
Python multiprocessing shared memory seems to hang or crash when interacting with small numpy arrays
I've been trying to parallelize some code that I wrote in python. The actual work is embarrassingly parallel, but I don't have much experience with multiprocessing in Python.
The actual code I'm ...
- 116
0
votes
0
answers
122
views
XCB error when using Python with Matplotlib and Multiprocessing
I ran into a problem when I'm trying to plot figures with matplotlib in separate processes using multiprocessing. I tried it on OpenBSD 7.7 with Python 3.12.11 and on Debian 12 with Python 3.11.2 and ...
- 1
0
votes
1
answer
75
views
subprocess.run() eventually reaches python's global thread limit
I have a python script that pings another device every so often.
PING_SIZE = "1"
PING_RETRYS = "1"
while True:
try:
result = subprocess.run(
["ping&...
2
votes
2
answers
226
views
How to safely check a multiprocessing.Event in an asyncio loop without blocking?
I'm working on an asynchronous server using asyncio, but I need to monitor a multiprocessing.Event (used to signal termination from another process) inside my async event loop. Here's the simplified ...
3
votes
0
answers
144
views
What is the best/correct way to handle shared memory objects in a heterogeneous multi-CPU embedded microprocessor context [closed]
I'm working on several projects that are targeting bare-metal multi-core microprocessors, with hardware provisions for shared memory.
In other words, the device has several separate CPU cores (of ...
- 5,953
3
votes
1
answer
95
views
Accessing any object type from multiprocessing shared_memory?
Suppose I create a shared memory object:
from multiprocessing import shared_memory
shm_a = shared_memory.SharedMemory(create=True, size=1024)
buffer = shm_a.buf
and put a generic object of a generic ...
- 5,906
3
votes
4
answers
185
views
Where should I allocate the pool in Python multiprocessing?
I want to distribute a Python application over several kernels, and from the documentation, I understand that a pool is the way to do this.
My problem can be reproduced by the following code:
#!/usr/...
- 91
1
vote
1
answer
81
views
Сonnection timeout for function does not seem to work in Python script
for some reason the connection timeout does not seem to work when connecting to a Firebird database. For example, if an incorrect IP address is specified, the script hangs and waits for a long time.
#!...
- 11
2
votes
2
answers
185
views
Setting global variables for python multiprocessing
I have a large array and an object I'd like to call multiple times with multiprocessing. Neither the data nor the object internals get modified.
This works:
import numpy as np
from multiprocessing ...
- 995
2
votes
1
answer
235
views
Starting multiprocessing.Process() in a Pytest test/Python creates a Windows fatal exception: access violation
On windows, running the below code with Python or pytest makes it print out Windows fatal exception: access violation (but the script will continue with no issue)
Reproduction:
import multiprocessing ...
- 694
0
votes
1
answer
90
views
Cannot pickle weakref error in the multiprocessing in a class in Python
In the project I am currently working on, I am parsing a json file and according to client list, I need to start processes in parallel. However there might be the case where a inside a client list, I ...
- 13
0
votes
0
answers
112
views
FastAPI becomes unresponsive while running partition_pdf from unstructured in a background process
I have a FastAPI application where one of the API endpoints needs to parse PDF files using unstructured.partition_pdf. Since the parsing is CPU-heavy, I want to run it in the background so that the ...
1
vote
0
answers
141
views
Windows Python 3.12 Multiprocessing Deadlock
I'm having trouble getting this example working. In serial this function works just fine, but when I attempt to run it in a multiprocessing.Pool it locks up and will not return a simple random integer....
- 51
0
votes
0
answers
58
views
How do I sucessfully stop TCPDump recording?
I am very new to python. My current project is to create a basic Tkinter window to start and stop a TCPdump Ethernet recording using a couple of buttons. I have been able to start a recording using a ...
- 1
0
votes
1
answer
85
views
How should CPU time be computed for calculations parallelized with the multiprocessing module?
I am trying to measure the processing time, or CPU time, of a CPU-intensive computation that has been parallelized with multiprocessing. However, simply bookending the parallelization of the ...
- 437
2
votes
0
answers
72
views
Overthreading in Multiprocessing for ML models
The problem
I am running machine learning models in parallel using multiprocessing. When using models with parameters stating the number of threads used- num_threads, num_jobs, etc. - the code works ...
- 353
1
vote
0
answers
151
views
Python processes arent being evenly distributed over cpu nodes
I wasn't able to find anything regarding this on the internet: I am using multiprocessing (concurrent.futures. ProcessPoolExecutor(max_workers=(...)) as executor) to execute several DRL training ...
- 11
1
vote
2
answers
91
views
Is it possible to create numpy array in shared_memory using custom dtypes?
I can succesfully create numpy arrays with simple dtypes (int32 etc).
But when i try to use something like :
b_shared_memory = shared_memory.SharedMemory(create=True, name = "test235", size =...
- 19
0
votes
2
answers
69
views
Python threads killed when main thread times out with only one thread, but survive when more than one threads are created
I have a Python function decorated with a 1-minute timeout. Inside it I spawn one or more worker threads that each run a 2-minute sleep via a dynamically executed script (using exec() in a custom ...
2
votes
1
answer
53
views
Avoid PicklingError using multiprocessing in Python 3.8 libreoffice macro
I'm trying to use multiprocessing in a LibreOffice Macro in Python 3.8. When I create a Macro it is part of the 'ooo_script_framework' module. So, when the process is pickled it is in fact part of an ...
0
votes
2
answers
128
views
Pausing, unpausing, and stopping threads in python
I am trying to make a program that has equivalents for another module's code.
The module has it's own thread class, with the commands 'stop', and 'sleep_for'
For example, using this module, you could ...
2
votes
0
answers
84
views
Why is training multiple deep learning models (via multiple terminals) faster than training a single model? Is it because higher clock speeds?
I'm training deep learning models using TensorFlow (with GPU support) on my local machine. I noticed a surprising behavior:
When I train just one model (in a single terminal), it runs slower.
But ...
- 77
0
votes
1
answer
140
views
How to run multiple potentially multithreaded shell commands in python
I am trying to write a Python script to run another potentially multithreaded script in a way to further parallelise them.
Consider a (C++) executable which can be run as run_task <sample_number>...
0
votes
0
answers
84
views
How to speed up Keras Tuning on multi core machine?
This question is related to the Distributed hyperparameter tuning strategy of keras tuner to get the possible hyperparameters for my model. The search space is huge. The worker thread is as in the ...
- 173
0
votes
1
answer
212
views
How do I efficiently write results from python's ProcessPoolExecutor to a single parquet file?
I use python to read a large csv file, transform the data within (mostly string operations), and then write the results to a single parquet file. In the transformation process, rows are independent. 1 ...
- 132
0
votes
0
answers
69
views
Unknown DDP error when running multi-processing with pytorch on a Google GPU based colab kernel
I have created a colab (link: https://colab.research.google.com/drive/1gg57PS7KMLKvvx9wgDKLMDiyjhRICplp#scrollTo=zG2D7JO2OdEC) to play with the gpt2 fine-tuning.
And I was trying to practice the DDP ...
- 11