21,956 questions
1
vote
1
answer
128
views
Why when I place I/O task before CPU-bound task, runs faster than place I/O task after CPU-bound task?
using System.Diagnostics;
const int TASKS = 100;
var mainSw = Stopwatch.StartNew();
var tasks = Enumerable.Range(0, TASKS).Select(i =>
Task.Run(async () =>
{
await Task.Delay(...
- 29
Best practices
0
votes
5
replies
105
views
Parallel.ForEach Returning Values
I need to process a list of objects (not the same shown on the sample), which I thought could be greatly improved by running it in parallel.foreach loop. However, the result is not what I expected. ...
- 3
Advice
1
vote
0
replies
44
views
What is the best pattern for triggering N sub-workflows in parallel and resuming main workflow when all complete?
I need to trigger a dynamic number of sub-workflows in parallel (around 100)
and wait for ALL of them to complete before continuing the main workflow.
I’ve implemented a solution but I’m wondering if ...
- 121
-4
votes
0
answers
42
views
What is the Global Interpreter Lock (GIL) in Python and why does it prevent true multithreading? [duplicate]
I’ve been reading about Python’s Global Interpreter Lock (GIL), and I’m a bit confused about how it actually works behind the scenes.
From what I understand, the GIL allows only one thread to execute ...
Advice
2
votes
2
replies
59
views
Efficient MPI Parallelization Strategies for Localized PDE Subproblems within a Globally Decomposed Domain
I am working on a global PDE problem that is solved using a standard domain-decomposition strategy (e.g., Scotch, METIS). This part of the computation is well balanced across all MPI processes.
...
- 1
Tooling
1
vote
3
replies
72
views
using persistent-memory gawk how variables can created to be local and issolated from other execution instances?
The idea of Persistent-Memory gawk is fabulous because it improves the performance, size, and clarity of many scripts on static and reference data.
However, I have a significant problem in adopting ...
1
vote
1
answer
79
views
How to share a large CustomObject to workers in Python multiprocessing on Windows (spawn)?
I'm trying to run calculations using multiple cores in Python on multiple platforms (Linux, macOS, Windows). I need to pass a large CustomClass Object and a dict (both readonly) to all workers. So far ...
- 459
0
votes
0
answers
40
views
Attribution Error when using Huggingface transformers Trainer with FSDP
I am now trying to use FSDP in Huggingface transformers Trainer. The training script is something like
train_dataset = Mydataset(...)
args = TrainingArguments(...)
model = LlamaForCausalLM....
- 11
0
votes
0
answers
21
views
OptimisticLockingException when using multiInstanceLoopCharacteristics for parallel execution of subprocess
I have the following process definition I try to execute on Camunda 7.24 / CibSeven 2.1 which currently logs during execution many OptimisticLockingException. I could already trace it down that it ...
- 308
0
votes
1
answer
126
views
Why are items not written to console immediately after being processed?
I have the following C# code :
var rand = new Random(1);
var range = Enumerable.Range(1, 8);
var partition = Partitioner.Create(range, EnumerablePartitionerOptions.NoBuffering);
foreach (var x in ...
- 4,596
0
votes
1
answer
97
views
Taking advantage of memory contiguousness in HLSL
This is a bit of a slog so bare with me.
I'm currently writing a 3D S(moothed) P(article) H(ydrodynamics) simulation in Unity with a parallel HLSL backend. It's a Lagrangian method of fluid simulation,...
- 13
Tooling
0
votes
0
replies
32
views
ComfyUI + Flux 1 dev + limited RAM + same workflow: With 2 GPUs?
I am running Flux 1 dev text to image model through ComfyUI in Kaggle. Everything works but I noticed that Kaggle offers a second GPU inside the notebook. If I try to run two instances of the ComfyUI ...
- 113
1
vote
0
answers
81
views
Intuition over TBB parallel scan/parallel prefix requirements
I am reading a paragraph about the tbb::parallel_scan algorithm from the book Intel Threading Building Blocks, and I understood what the operation does serially, but I am not understanding what are ...
- 446
0
votes
0
answers
73
views
Simple TBB example where tbb::affinity_partitioner gives a measurable speedup
While looking at this TBB guide webpage: https://www.intel.com/content/www/us/en/docs/onetbb/developer-guide-api-reference/2021-9/bandwidth-and-cache-affinity.html, they mention this ...
- 446
Best practices
0
votes
2
replies
129
views
Looping Datasets in R
Essentially I am trying to create a dataset that is dependent on prior rows to generate values for any given row. I then would like to run this loop over many IDs for an entire dataset. Current set up ...
0
votes
0
answers
52
views
Parallel Equations Expansion in TFORM
TFORM is considered a great tool for manipulating large and symbolic equations. In this thread, I’d like to share my optimization problem, which concerns a very simple operation — equations expansion.
...
- 1
Advice
1
vote
1
replies
33
views
Is there a way to transfer MBTiles formatted maps faster?
I need to transfer a MBTiles map tiles from a disk to another, is there a faster way then just mv? It is huge and takes time.
0
votes
1
answer
97
views
C++ segmentation fault when throwing in ordered OMP parallel for
the code below crashes with
terminate called after throwing an instance of 'std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >'
Aborted ...
- 431
0
votes
1
answer
52
views
Parallel Processing in Sclang
I am doing some calculations on spectra, doing some maths on each partial. This is taking a lot of time, but since the partials are all calculated independently, I wonder how to do parallel processing ...
- 438
1
vote
0
answers
41
views
run single notebook with different objects lists dict parallel on spark, when called from master notebook in synapse
Anyone know how to call single notebook with different parameters parallel, and all notebooks should appear on spark UI to make the trouble shooting easier? I have one child notebook, calling from ...
- 11
0
votes
0
answers
25
views
Integrate socket.io namespaces with Node Cluster
I am trying to integrate socket.io with Node's HTTP alongside Node's Cluster Module. Consider the reproducible example:
index.js:
let cluster = require('cluster')
let fs = require('fs')
let http = ...
- 351
1
vote
3
answers
249
views
How to convert a batch to Powershell parallel processing to run over thousands of input files?
I have a large number of files (about 3800) that I want to run a program over. The program reads a WAV file and makes a short .TSV text file containing the WAV's lip-sync data (but that is by-the-by ...
0
votes
0
answers
31
views
How to prepare a set of equations for parallel expansion in TFORM?
I’m working with FORM/TFORM to automatically expand a large set of symbolic equations.
My goal is to make the expansion process run in parallel on multiple CPU cores using TFORM.
Here’s a simplified ...
- 1
2
votes
1
answer
86
views
Python multiprocessing parallelization
I have a class with methods to simulate sources across 16 detectors using the Gelsa package. In my main script, I call the method generate.sources. I am trying to use multiprocessing to speed up the ...
- 21
15
votes
0
answers
626
views
Why does my C++ N-body simulation have a pulsating performance slowdown?
I've been developing a 2D N-body gravity simulation in C++, and I've run into an interesting performance issue. Instead of a stable frame rate, the application's update time systematically pulsates ...
- 159
1
vote
1
answer
102
views
AMP deprecated code doesn't compile with lambdas
I am trying to compile some of my older applications, and it doesn't compile where it encounters lambdas. I am well aware that this api is deprecated, but it compiled ok some time ago on the same ...
- 2,417
0
votes
0
answers
66
views
How to decide the data size handled by each processor/core in SIMD?
I’m learning how to use SIMD (Single Instruction, Multiple Data) for parallel data processing.
Suppose I have a large dataset (e.g., an array of 1 million floats), and I want to process it efficiently ...
- 1
8
votes
1
answer
244
views
Does std::find still guarantee first element with std::execution::par?
Parallel policy states that order of iterator evaluation is not guaranteed. However, std::find*, std::search and std::mismatch all say that they return first iterator matching condition. How do those ...
- 3,192
1
vote
1
answer
108
views
Is CPU multithreading effected by divergence?
Building on this question here
The term thread divergence is used in CUDA; from my understanding it's a situation where different threads are assigned to do different tasks and this results in a big ...
- 960
0
votes
1
answer
80
views
Spawning multiple tasks but inner function is not executed without error message
I'm trying to spawn multiple parallel (not concurrent) tasks. Every task is running a PUT operation to a custom S3 storage using AWS SDK for Rust.
The function body looks the following (the different ...
- 195
0
votes
1
answer
207
views
For Xunit 3, when using [assembly: CaptureConsole], does that work with parallel tests
We have several tests which have been switched to using XUnit 3. We have an assembly-level
[assembly: CaptureConsole(CaptureOut = true, CaptureError = true)]
and we have several "more ...
- 1,779
-1
votes
0
answers
49
views
What is the correct way to use the Eval monad in haskell? [duplicate]
I am learning parallel programming in haskell.
My example code:
module Quicksort where
import Control.Parallel.Strategies
import Control.Parallel
qsort :: [Int] -> [Int]
qsort [] = []
qsort lst@(...
- 2,334
1
vote
0
answers
70
views
How to properly parallelise JFreeChart's PNG generation?
I use JFreeChart to create various kinds of charts en masse, based on huge amounts of data. Those charts are only meant to be written to PNG files on the hard drive; no JavaFX, Swing, AWT or other GUI ...
- 440
2
votes
0
answers
81
views
Why large number of sparks are GC’d? [duplicate]
I have function (RSA encryption) where end result m depend on m1 and m2 where m1 and m2 can be computed in parallel. I tried to use par and pseq but the result is weak.
Total time 20.062s ( 18....
- 427
0
votes
2
answers
103
views
Nested slurm jobs in R using future.batchtools
I am trying to test future.batchtools for parallelisation in R.
I have a small test job (run_futurebatchtools_job.R) as:
library(future)
library(future.batchtools)
# Set up the future plan to use ...
- 133
4
votes
1
answer
144
views
Setting seed for nested parallel simulation in R and storing the states of the random-number generator
In R, I am parallelising my simulation using packages foreach, doFuture, and doRNG.
I have two nested foreach loop: the inner loop generate and analyse data for each iteration, and the outer loop ...
- 43
0
votes
0
answers
111
views
How to make parallel within the parallel?
Right now I am running a model optimization to optimize one set of parameter for several sites (in total 47 sites, i.e. the cost function sum over these 47 results). Site computation is independent ...
- 325
2
votes
1
answer
192
views
Can `Stream.allMatch()` call the predicate multiple times for the same element?
I'm trying to implement a short-circuited processing for an external input of java.util.Stream (think Stream.forEach() but with short-circuiting). I do not care about order of the elements, but if ...
- 24.6k
1
vote
1
answer
117
views
How to remove intermediate results after execution of the chord without blocking execution?
Here’s the pattern I want:
Dispatch multiple tasks in parallel.
Aggregate all their results into a final result.
Remove the intermediate results right after the chord result is ready, without ...
- 97
-2
votes
1
answer
184
views
will C++ ever have keyword such as "for_parallel"? [closed]
While learning about parallelism, I learned that C++ support parallelism through functions such as std::for_each, std::transform and execution policy. So if, for example, we want to divide elements of ...
- 13
0
votes
0
answers
53
views
terra::app() error: 'Not compatible with requested type' when applying function returning numeric vector to SpatRaster in R
I'm working on a function to detect positive and negative events using a time series of cumulative anomalies. The function seems to work fine for vectors and produces the correct number of outputs as ...
0
votes
0
answers
54
views
How to parallelize R code that imports in a spark table so you can collect the data within the foreach loop?
I'm trying to parallelize my R code that pulls in data from a Snowflake table, but when I do, I get an error that I have an invalid connection. I don't receive this error when I do NOT have it ...
- 2,596
2
votes
3
answers
111
views
Open-MP Parallel for (three-dimensional array)
We are working with the following code:
int i, j, k;
for (i = 2; i < n; i++){ // S1
for (j = 3; j < n - 3; j++){ // S2
for (k = 4; k < n - 4; k++){ // S3
A[...
- 21
1
vote
0
answers
80
views
cache-efficient partitioning for multithreaded processing in arm
Suppose you are processing a large data set using several cores in parallel. I am looking for the most memory-efficient way to break up the data among the processors.
Specifically, this would be for ...
- 53
2
votes
0
answers
61
views
Persistent parallel threads in Panda3d, with args at runtime
I'm writing a physics game, and I'm trying to speed up my motion calculations. Every tick of the update cycle, I call an rk4 routine which calls an ODE function 4 times, passing updated values for dt/...
- 51
2
votes
2
answers
371
views
`purrr::in_parallel` extremely slow for rowwise operation on data.frames
I was excited when I read about the latest update of purrr (1.1.0) with its in_parallel capabilities.
I just happen to have a time-consuming data task that runs for several minutes, because:
my data ...
- 11.6k
1
vote
0
answers
42
views
Track number of in-use vs. idle workers in Matlab with fmincon
I'm currently in Matlab doing a set of fixed point iterations using fmincon. I use parfor multithreading to do so. However, one of my iterations in the parfor loop goes particularly slow just because ...
- 105
1
vote
1
answer
79
views
CoroutineTestExtension not working when parallel test execution is enabled in JUnit5
I have multi module android project where I am doing junit4 to junit5 migration
For CoroutineTestRule I have added CoroutineTestExtension but it's not working well when parallel test execution is ...
2
votes
1
answer
283
views
Why is std::execution::par_unseq slow compared with other parallel options
Sorry I was busy and made a few mistakes. First is that the logics for the various implementations were not the same, and I adjusted accordingly, second is that there is an overflow with float and I ...
- 810
1
vote
1
answer
407
views
Unable to set cpu device count for jax parallelisation?
I have been trying to generalise this jax program for solving on both CPU and GPU depending on the machine it's running on (essentially need cpu parallelisation to speed up testing versus gpu for ...
- 305