@knopers8 @sawenzel @matthiasrichter @shahor02 this now should optimise local DPL workflows to use (managed) shared memory.

@knopers8 could you try to repeat your QC benchmark?

I will run them. Just to make sure, this will work also for messages created with standard ctx.outputs().make<char>(...);, or I have to allocate them in some other way?

It should be completely transparent.

@ktf I see that some different workflows fail in each build, so the issue is most likely is with shm in a container. What are the settings for those currently?

Indeed. I use the defaults which I think allows for 64MB of shared memory.

It actually seems to work with a multinode setup as well!! To try it:

# On host1
stage/tests/o2-test-framework-ParallelPipeline --hostname host1 --resources host1:1:4000:22000:23000,host2:4:4000:24000:25000
# On host2
stage/tests/o2-test-framework-ParallelPipeline --hostname host2 --resources host1:1:4000:22000:23000,host2:4:4000:24000:25000

@aalkin ok, I think I understand the issue. There is a unique id "--session" which needs to be passed to FairMQ to allow different shared memory pools. We can simply put back uniqueWorkflowId as an argument of prepareArguments.

The gpu error seems unrelated. I will merge this to have the improved resource scheduling and address the actual usage of shmem as transport in a separate PR.

ok, but could we check what is going on with the GPU CI? It is just failling constantly at random recipes. Doesn't make much sense this way.

I am looking into it.

Some of the results for larger payloads and number of producers are missing, because something crashes inside FairMQ/shmem/boost. It helps a bit if I reduce the buffers size from standard 1000 to something smaller, but with rates high enough the problem appears anyway. I guess it is connected with https://alice.its.cern.ch/jira/browse/O2-879

By default FairMQ has 2GB shared memory segment. I suppose that is what is crashing the 1GB test. You should be able to change it with --shm-segment-size.

There seems to be a clear throughput maximum around 4M-16M payload size, which is far enough from 2GB limit. It would be interesting to look at a heatmap-like throughput plot with number of producers and payload sizes at the axes.

By default FairMQ has 2GB shared memory segment. I suppose that is what is crashing the 1GB test. You should be able to change it with --shm-segment-size.

I've been using that for these tests, it helped only to some extent.

There seems to be a clear throughput maximum around 4M-16M payload size, which is far enough from 2GB limit. It would be interesting to look at a heatmap-like throughput plot with number of producers and payload sizes at the axes.

Yes, that might be interesting to see, I will let it run later.

I have tested it on previous 'force-pushed' version (be1a810). Would the new one have something different regarding these crashes?

I would suggest to test with 4428d07 (has shmem transport) and 6fab7d9 (shmem transport removed, so the default zeromq is used).

@knopers8 could you try #2531 to see if numbers improve? This should remove the hash table lookup.