Skip to content

DPL: improve ability to forward messages to an output#4609

Merged
ktf merged 1 commit intoAliceO2Group:devfrom
ktf:improve-forwarding-policy
Nov 13, 2020
Merged

DPL: improve ability to forward messages to an output#4609
ktf merged 1 commit intoAliceO2Group:devfrom
ktf:improve-forwarding-policy

Conversation

@ktf
Copy link
Copy Markdown
Member

@ktf ktf commented Oct 15, 2020

  • Old dangling-outputs-policy renamed to forwarding-destination.
  • Introduced "forwarding-policy" to decide which kind of messages to forward.
  • Use enums rather than integers.

@ktf ktf requested a review from a team as a code owner October 15, 2020 22:09
@ktf
Copy link
Copy Markdown
Member Author

ktf commented Oct 15, 2020

@teo @ironMann initial attempt at making DPL forwarding abilities more suitable for your needs. In particular I guess you want to have:

--forwarding-policy all --forwarding-destination fairmq

@ironMann
Copy link
Copy Markdown
Contributor

ironMann commented Nov 2, 2020

@ktf can you please rebase. It is not building for me.
sw/SOURCES/O2/0_O2_DATAFLOW/0/Framework/Core/src/WorkflowHelpers.cxx:387:82: error: 'OutputsInputs' was not declared in this scope

@ktf ktf force-pushed the improve-forwarding-policy branch from 432e993 to df6d68b Compare November 3, 2020 14:06
@ktf
Copy link
Copy Markdown
Member Author

ktf commented Nov 3, 2020

@ironMann sorry, this is the correct PR. I will close the other one, as per @preghenella comment, it's not actually required to have consumed data forwarded.

@ironMann
Copy link
Copy Markdown
Contributor

ironMann commented Nov 3, 2020

@ktf With this I only get the following, even for o2-dpl-raw-proxy --help:

[ERROR] error while setting up workflow: Unexpected index
[INFO] Process 259035 is exiting.

@ktf
Copy link
Copy Markdown
Member Author

ktf commented Nov 3, 2020

Ok. I will check in a bit...

@ktf ktf force-pushed the improve-forwarding-policy branch from df6d68b to 7cb746d Compare November 12, 2020 13:57
@ktf
Copy link
Copy Markdown
Member Author

ktf commented Nov 12, 2020

@ironMann I have finally properly reviewed this and my synthetic test works. I also tried the following:

o2-dpl-raw-proxy --session default --forwarding-policy dangling --forwarding-destination fairmq --readout-proxy '--channel-config name=readout-proxy,type=pull,method=connect,address=ipc:///tmp/stfb-to-dpl,transport=shmem,rateLogging=1' --internal-dpl-injected-output-proxy '--channel-config name=downstream,type=push,method=bind,address=ipc:///tmp/dpl-to-stfs,rateLogging=1,transport=shmem' --severity debug

and it seems to connect properly, however I do not have readout.exe setup correctly at the moment, so I cannot fully guarantee. If you can try it it would be great I will try to do it as well, but not immediately. Notice that the device has been renamed internal-dpl-injected-output-proxy due to limitations in the current way I exclude injected devices and the associated option to push device specific parameters is --internal-dpl-injected-output-proxy.

@ktf
Copy link
Copy Markdown
Member Author

ktf commented Nov 12, 2020

Full setup with TOF compressor.

ninja stage/bin/o2-dpl-raw-proxy stage/bin/o2-testworkflows-simple-source stage/bin/o2-dpl-raw-proxy stage/bin/o2-tof-compressor stage/bin/o2-dpl-run

PROXY_SPEC="x:TOF/RAWDATA"
COMPR_CONF="x:TOF/RAWDATA"

VERBOSE=""
VERBOSE="--tof-compressor-verbose --tof-compressor-decoder-verbose"

stage/bin/o2-dpl-raw-proxy -b --session default                                                                                                 \
    --dataspec "$PROXY_SPEC"                                                                                                                    \
    --channel-config "name=readout-proxy,type=pull,method=connect,address=ipc:///tmp/stf-builder-dpl-pipe-0,transport=shmem,rateLogging=1"     |\
stage/bin/o2-tof-compressor -b --session default                                                                                                \
    --tof-compressor-rdh-version 6                                                                                                              \
    --tof-compressor-config "$COMPR_CONF"                                                                                                       \
    $VERBOSE  --dump                                                                                                                           |\
stage/bin/o2-dpl-run                                                                                                                            \
    --forwarding-policy dangling                                                                                                                \
    --forwarding-destination fairmq                                                                                                             \
     --internal-dpl-injected-output-proxy '--channel-config name=downstream,type=push,method=bind,address=ipc:///tmp/dpl-to-stfs,rateLogging=1,transport=shmem' \

@ironMann
Copy link
Copy Markdown
Contributor

ironMann commented Nov 12, 2020 

o2-dpl-raw-proxy --session default --forwarding-policy dangling --forwarding-destination fairmq --readout-proxy '--channel-config name=readout-proxy,type=pull,method=connect,address=ipc:///tmp/stfb-to-dpl,transport=shmem,rateLogging=1' --internal-dpl-injected-output-proxy '--channel-config name=downstream,type=push,method=bind,address=ipc:///tmp/dpl-to-stfs,rateLogging=1,transport=shmem' --severity debug

and it seems to connect properly

@ktf If I understand you, this would setup an pass-through DPL workflow. I gave it a test with some TOF data, and again there is nothing on the output:

[435668:readout-proxy]: [15:49:51][WARN] Some input data are not matched by filter rules
[435668:readout-proxy]:    TOF/RAWDATA/10
[435668:readout-proxy]:    TOF/RAWDATA/7
[435668:readout-proxy]:    TOF/RAWDATA/11
[435668:readout-proxy]:    TOF/RAWDATA/6
[435668:readout-proxy]: DROPPING OF THESE MESSAGES HAS BEEN ENABLED BY CONFIGURATION
[435668:readout-proxy]: [15:49:52][INFO]                                            readout-proxy[0]: in: 80 (68.1708 MB) out: 0 (0 MB)
[435669:internal-dpl-injected-output-proxy]: [15:49:52][INFO]                                               downstream[0]: in: 0 (0 MB) out: 0 (0 MB)
[435668:readout-proxy]: [15:49:53][INFO]                                            readout-proxy[0]: in: 123 (104.813 MB) out: 0 (0 MB)
[435669:internal-dpl-injected-output-proxy]: [15:49:53][INFO]                                               downstream[0]: in: 0 (0 MB) out: 0 (0 MB)
[435668:readout-proxy]: [15:49:54][INFO]                                            readout-proxy[0]: in: 134 (114.186 MB) out: 0 (0 MB)
[435669:internal-dpl-injected-output-proxy]: [15:49:54][INFO]                                               downstream[0]: in: 0 (0 MB) out: 0 (0 MB)

@ktf
Copy link
Copy Markdown
Member Author

ktf commented Nov 12, 2020 •
edited
Loading

Can you remind me your full setup? I am unable to run readout + stfbuilder. I do:

# test config to run readout-stbf out of the box with data emulator


cat << EOF >readout.cfg
# dummy data source
[equipment-emulator-1]
enabled=1
name=emulator-1
equipmentType=cruEmulator
memoryPoolNumberOfPages=1800
memoryPoolPageSize=1M
numberOfLinks=4
PayloadSize=8000

# monitor counters
[consumer-stats]
consumerType=stats
monitoringEnabled=0
monitoringUpdatePeriod=5
monitoringURI=no-op://

# send data to stfb
[consumer-StfBuilder]
enabled = 1
consumerType = FairMQChannel
sessionName = default
fmq-transport = shmem
fmq-name = readout
fmq-type = push
fmq-address = ipc://@flp-readout-pipe-0
unmanagedMemorySize = 2G
memoryPoolNumberOfPages = 200
memoryPoolPageSize = 1M
disableSending=0
EOF

readout.exe file:readout.cfg

and

StfBuilder --id stfb --session default --transport shmem \
  --detector TOF --detector-rdh 6                        \
  --dpl-channel-name dpl-chan                            \
  --input-channel-name readout                           \
  --output-channel-name dpl-chan                         \
  --data-source-enable 1 --data-source-rate=44           \
  --channel-config="name=readout,type=pull,method=connect,address=ipc://@flp-readout-pipe-0,transport=shmem,rateLogging=10" \
  --channel-config="name=dpl-chan,type=push,method=bind,address=ipc://@stfb-to-dpl,transport=shmem,rateLogging=10"

the two processes initialise, but then nothing else happens...

@ktf ktf force-pushed the improve-forwarding-policy branch from 7cb746d to 518d907 Compare November 13, 2020 00:42
@ktf
Copy link
Copy Markdown
Member Author

ktf commented Nov 13, 2020

Ok, I managed to run it and get something out of STF builder.

Screenshot 2020-11-13 at 01 20 34

The options for the internal devices need to be specified in the correct place and in general the --channel-config options needs to be specified as part of the device specific flags (i.e. prefixing them with --id-of-the-device). @teo, I suspect this is also why you see an extra channel in your configuration.

This is annoying, but at least can be worked around. I will open a JIRA ticket on finally reworking the whole thing.

With the following:

ninja stage/bin/o2-dpl-raw-proxy stage/bin/o2-testworkflows-simple-source stage/bin/o2-dpl-raw-proxy stage/bin/o2-tof-compressor stage/bin/o2-dpl-run

PROXY_SPEC="x:TOF/RAWDATA"
COMPR_CONF="x:TOF/RAWDATA"

VERBOSE=""
VERBOSE="--tof-compressor-verbose --tof-compressor-decoder-verbose"

fairmq-shmmonitor --cleanup
stage/bin/o2-dpl-raw-proxy -b --session default                                                                                                 \
    --dataspec "$PROXY_SPEC"                                                                                                                    \
    --forwarding-policy dangling                                                                                                                \
    --forwarding-destination fairmq                                                                                                             \
    --severity debug                                                                                                                            \
    --internal-dpl-injected-output-proxy '--channel-config="name=downstream,type=push,method=bind,address=ipc:///tmp/dpl-to-stfs,rateLogging=1,transport=shmem"' \
    --readout-proxy '--channel-config "name=readout-proxy,type=pull,method=connect,address=ipc:///tmp/stfb-to-dpl,transport=shmem,rateLogging=1"' |\
stage/bin/o2-tof-compressor -b --session default                                                                                                \
    --tof-compressor-rdh-version 6                                                                                                              \
    --tof-compressor-config "$COMPR_CONF"                                                                                                       \
    --forwarding-policy dangling                                                                                                                \
    --severity debug \
    --forwarding-destination fairmq                                                                                                             \
    $VERBOSE

I am able to run stf-builder + DPL + stf-sender with the sender actually receiving something, before the compressor crashes (not sure why). There is a few issues:

  • One needs to start the DPL workflow first, I suspect because DPL cleans shared memory by default. I will open a separate PR which forces DPL not to clean up shared memory.
  • DataDistribution leaves the shared memory in a weird state when exiting (at least on mac) forcing me to clean up the memory in any case.
  • The above mentioned crash in compressor. Stacktrace:
[10029:tof-compressor-0]: [/Users/ktf/src/sw/BUILD/bba0d7fd2f84ed6d18c59f067737cc1eaff0f089/O2/stage/lib/libO2TOFCompression.dylib] o2::tof::Compressor<o2::header::RAWDataHe
aderV6, false>::processHBF() /Users/ktf/src/sw/SOURCES/O2/v1.3.0/0/Detectors/TOF/compression/src/Compressor.cxx:132
[10029:tof-compressor-0]: [/Users/ktf/src/sw/BUILD/bba0d7fd2f84ed6d18c59f067737cc1eaff0f089/O2/stage/lib/libO2TOFCompression.dylib] o2::tof::CompressorTask<o2::header::RAWDa
taHeaderV6, false>::run(o2::framework::ProcessingContext&) /Users/ktf/src/sw/SOURCES/O2/v1.3.0/0/Detectors/TOF/compression/src/CompressorTask.cxx:87
[10029:tof-compressor-0]: [/Users/ktf/src/sw/BUILD/bba0d7fd2f84ed6d18c59f067737cc1eaff0f089/O2/stage/lib/libO2Framework.dylib] o2::framework::DataProcessingDevice::tryDispat
chComputation(o2::framework::DataProcessorContext&, std::__1::vector<o2::framework::DataRelayer::RecordAction, std::__1::allocator<o2::framework::DataRelayer::RecordAction>
>&) /Users/ktf/src/sw/SOURCES/O2/v1.3.0/0/Framework/Core/src/DataProcessingDevice.cxx:1025
[10029:tof-compressor-0]: [/Users/ktf/src/sw/BUILD/bba0d7fd2f84ed6d18c59f067737cc1eaff0f089/O2/stage/lib/libO2Framework.dylib] o2::framework::DataProcessingDevice::doRun(o2:
:framework::DataProcessorContext&) /Users/ktf/src/sw/SOURCES/O2/v1.3.0/0/Framework/Core/src/DataProcessingDevice.cxx:561
[10029:tof-compressor-0]: [/Users/ktf/src/sw/BUILD/bba0d7fd2f84ed6d18c59f067737cc1eaff0f089/O2/stage/lib/libO2Framework.dylib] o2::framework::DataProcessingDevice::Condition

@ktf ktf merged commit bdcc6a4 into AliceO2Group:dev Nov 13, 2020
@ktf
Copy link
Copy Markdown
Member Author

ktf commented Nov 13, 2020

Regarding the dropped messages, shouldn't you simply give --dataspec TOF/RAWDATA to it?

@ktf ktf deleted the improve-forwarding-policy branch November 13, 2020 13:44
@ironMann
Copy link
Copy Markdown
Contributor

ironMann commented Nov 13, 2020

This has nothing to do with TOF, or RAWDATA. One should simply not need to know the type or source of the individual O2 messages in the STF. Can the input proxy simply forward everything not matched to an DPL consumer to the output proxy? Eg, messages encoding end-of-stream also get blocked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ktf @ironMann