192 questions
0
votes
0
answers
98
views
Is it possible to set SO_REUSEADDR and SO_REUSEPORT in libfabric (verbs provider) to allow multiple server instances on the same port?
I'm using the libfabric library (version 1.7) with the verbs provider to build a high-performance server application.
I want to run multiple instances of the same server on the same machine, using the ...
- 137
1
vote
0
answers
136
views
GPU to GPU direct data transfer with connectX and RDMA
we are trying to connect two gpus located on two servers via RDMA and infinibands. The GPUs are Nvidia RTX 6000 Ada and the infinbands are NVIDIA ConnectX-6.
Server configuration
Our server has the ...
- 11
0
votes
1
answer
48
views
Is there a way to add specific data to a pcap file that the program writes manually (using C)?
I am capturing the packets sent on an Infiniband network and manually writing them to a pcap file.
The user is parsing the pcap file with pyshark.
I'd like for the user to be able to know what ...
0
votes
1
answer
170
views
Unable to Receive Messages with InfiniBand UD (Unreliable Datagram) Communication
I am currently writing a sample program for RDMA communication using InfiniBand. The setup involves two servers, each running a single process. The goal is to send data from one server (Sender) to the ...
0
votes
1
answer
66
views
How to transfer large message(>MTU) using RDMA UD mode?
RDMA UD (Unreliable Datagram) mode support Send/Recv operation only, and with the limit that only one packet can be sent with a send wr, which causes that the transfered message's size should less ...
- 127
0
votes
0
answers
41
views
In libibverbs, when does completion entry get pushed to CQ?
After a wr has been posted through ibv_post_send to a RC, does completion entry get pushed to sender's cq when data has been written to destination buffer and the sender receive an acknowledgement?
...
1
vote
0
answers
146
views
What is the minimal setup needed to start to develop for InfiniBand networking on Windows and C++?
I want to create a project that uses InfiniBand for RDMA between the machines, however currently I have to develop my code in windows, as I have to be able to call code created by other teams who only ...
- 6,663
0
votes
1
answer
90
views
Is system call always involved in issuing a RDMA operation?
I read the following from ibverbs' website:
"InfiniBand host channel adapters (HCAs) and iWARP NICs commonly support direct hardware access from userspace (kernel bypass), and libibverbs supports ...
2
votes
2
answers
363
views
verbs: Difference between a Shared Completion Queue and a Shared Receive Queue?
I am trying to manage multiple Queue Pairs connections in the same thread. To do that, I can share the same Completion Queue with multiple Queue Pairs and poll it in a single thread.
I don't see what ...
- 1,708
0
votes
1
answer
172
views
RDMA access flags validation
If we receive Send or Send with Immediate packets but don't have local write access right in RC, UC, or UD, what behavior should we expect?
I think that there’s a local protection error in the target, ...
- 11
0
votes
0
answers
119
views
Singularity container with OpenMPI and InfiniBand (UCX)
I'm working as an intern currently, and I was asked to build a Singularity container for OpenMPI to make distributed programming possible on multiple machines of our HPC cluster using containers.
...
-1
votes
1
answer
52
views
The bandwidth used to test the code is not the same as that tested by perftest
In order to learn RDMA, I found an example on the Internet, which is similar to the one provided by MELLANOX , but when I used two machines to run, I found the following problems:
1.There is a big gap ...
- 1
1
vote
0
answers
75
views
Unable to Monitor Sending Speed on Mellanox NIC in Loopback Configuration
I’m using a connectx4 NIC with 2 ports, connected in a loopback configuration. I have two RDMA programs running: one sends data from one port, and the other receives it on the second port. While ...
1
vote
0
answers
33
views
Any method to limit a cross-node RDMA-based program's maximum usable IB bandwidth?
I want to test a RDMA-based program's inter-node performance under a limited IB bandwidth. But I don't have sudo permission, no container software. Is there any alternative method can achieve this ...
- 11
0
votes
1
answer
70
views
Is it possible to disable “Coalesced Acknowledge Messages” for RDMA Write?
According to the IB specification, "Coalesced Acknowledge Messages" means "It is not required that there be a unique, discrete response for each request packet. Instead, the responder ...
- 1
0
votes
1
answer
81
views
OpenMPI over Infiniband Pkey
I'm trying to use use openmpi over InfiniBand Pkey networks.
I can't seem to find any documentation on how to do that.
I'm using openmpi 4.1.2 and just a simple ring_c.
mpiexec -n 8 -host cn0001-pkey,...
- 43
0
votes
1
answer
842
views
How to know which RDMA device/port/gid to use?
I have two hosts that are connected through RDMA (one is a SmartNIC, the other is the server). How can I know which pair of device/port/gid to use, if for example I want to run ib_send_bw -d <...
- 1,708
0
votes
0
answers
77
views
Should I use volatile with RDMA buffers?
I am learning RDMA verbs in C (libibverbs).
One question I have is after having polled a Receive Work Completion with a Receive RDMA write with immediate (IBV_WC_RECV_RDMA_WITH_IMM), the compiler ...
- 1,708
0
votes
1
answer
198
views
`libibverbs`: Is it possible to know the size of a receiving data buffer?
Is it possible to call ibv_post_send() with a size of a buffer (ibv_sge.length) smaller than the the size of the buffer in the receiver side (ibv_post_recv())? So that we only specify on the receiver ...
- 1,708
-1
votes
1
answer
290
views
Confusion regarding RDMA read behavior during retransmission
The Spec says:
The PSN of the retried RDMA READ request need not be the same as the PSN of the original RDMA READ request. Any retried request must correspond exactly to a subset of the original RDMA ...
0
votes
1
answer
224
views
User-mode application that performs RDMA directly to nvme drive on Linux
We've been developing HPC applications that take advantage of the infiniband infrastructure. One of our applications exchange data that is stored in an nvme cache with other nodes, and for that it ...
- 491
3
votes
1
answer
2k
views
Perf stat counts context-switches in what way?
perf stat displays some interesting statistics that can be gathered from examining hardware and software counters.
In my research, I couldn't find any reliable information about what counts as a ...
- 397
2
votes
1
answer
5k
views
OpenMPI 4.1.1 There was an error initializing an OpenFabrics device Infinband Mellanox MT28908
Similar to the discussion at MPI hello_world to test infiniband, we are using OpenMPI 4.1.1 on RHEL 8 with 5e:00.0 Infiniband controller [0207]: Mellanox Technologies MT28908 Family [ConnectX-6] [15b3:...
- 198
1
vote
1
answer
996
views
Multithreaded use of a single QP vs multiple QPs to improve throughput
I am using RDMA writes in my application and want to improve throughput.
Currently, I have a single thread using my queue pair. I was wondering, what is a more standard way (or what are the advantages ...
- 29
1
vote
1
answer
551
views
How to dynamically modify QP rate limit in ConnectX-6 Infiniband/VPI adapter
I have a ConnectX-6 Infiniband/VPI Adapter. I can setup the hardware rate limit when creating a qp like this:
...
ibv_qp_attr.ah_attr.static_rate = 7; // set qp rate limit to 40Gbps
...
ibv_modify_qp(...
- 361
0
votes
1
answer
1k
views
Rdma infiniband cannot open hosts (iberror: discovery failed) Port state: Down
I am facing an issue while configuring rdma and Infiniband on my two nodes. Both of these two nodes are connected and I have installed the recommended software libraries and packages required.
But my ...
- 11
0
votes
1
answer
107
views
How does SEND bandwidth improve when the registered memory is aligned to system page size? (In Mellanox IBD)
Operating System: RHEL Centos 7.9 Latest
Operation:
Sending 500MB chunks 21 times from one System to another connected via Mellanox Cables.
(Ethernet controller: Mellanox Technologies MT28908 Family [...
- 79
2
votes
1
answer
492
views
RDMA Read protection for local memory operations
I have the following scenario: My server allocates a buffer of 1MB, which is periodically updated and written to (about every 50ms). The client is connected to the the server via Infiniband and ...
- 43
1
vote
0
answers
6k
views
MPI hello_world to test infiniband
I have virtual machine which has passthrough infiniband nic. I am testing inifinband functionality using hello world program. I am new in this world so may need help to understand following error
I ...
- 17.7k
0
votes
0
answers
109
views
ibv_post_send performance optimization in KMDF windows driver
I want to use RDMA in a KMDF driver where the buffer received in an EvtIoWrite will be written directly to remote memory but this operation is performance critical so I'm wondering if there is a way ...
1
vote
1
answer
923
views
Infiniband RDMA completion queues with multithreading
I'm learning how to use RDMA via Inifniband and one problem I'm having is using a connection with more than 1 thread because I cant figure out how to create another completion queue so the work ...
0
votes
1
answer
261
views
what to change in ibverbs when switching from UD to RC connections
I'm looking at a ibverbs code from Mellanox.
With a send/recv operation via ibverbs.
The code is using UD connections.
But it didnt work when I change qp_type = IBV_QPT_UD to IBV_QPT_RC
What do I need ...
1
vote
0
answers
199
views
FPGA to DMA to RDMA
I am trying to send data generated from my FPGA card out to an IB device. I want the latency to be as low as possible, so I am thinking this may be the data path.
FPGA --> DMA via scatter/gather ...
- 29
-1
votes
1
answer
1k
views
Rsync over Infiniband/RDMA
Does rsync currently supports datta transfer over RDMA/Infiniband? I have to send some data to another server, but it's taking long to transfer, after searching for some time, i found there is ...
- 73
1
vote
1
answer
255
views
What is the logic behind setting valid region half the size of PSN range?
In order to make it possible for the responder to distinguish duplicate
packets from out of order packets, a given send queue shall have a
series of PSNs no greater than 8,388,608 outstanding at any ...
- 43
2
votes
1
answer
1k
views
What is the difference b/w RDMA_CREATE_QP/RDMA_CREATE_EP and IBV_CREATE_QP?
As far as I understand, IBV_CREATE_QP() and RDMA_CREATE_QP literally do the same thing. If I have called ibv_create_qp() first, do I still need to call the other one when I'm establishing a ...
- 43
0
votes
1
answer
2k
views
RoCE connection problem with MLNX_OFED (RDMA over Converged Ethernet)
I am trying to get RoCe (RDMA over converged ethernet) to work on two workstations. I have installed MLNX_OFED on both computers which are equiped with Mellanox ConnectX-5 EN 100GbE adapters and ...
- 11
0
votes
1
answer
302
views
Using tensorflow with VERBS support without infiniband device
I am interested in the RDMA support in tensorflow 1.15 for workers and parameter servers to communicate directly without going through CPU. I do not have infiniband VERBS devices but can build ...
- 985
1
vote
1
answer
777
views
What's the difference between Doorbell and Doorbell Record?
I'm new to RDMA technology and I can't figure out what's the difference between doorbell and doorbell record. In my opinion, when Software posts its work request, it rings the doorbell register to ...
- 41
0
votes
0
answers
2k
views
How to force open mpi 3 to use TCP
I run a small cluster for mpi computing, and recently we acquired some EDR Infiniband Equipment. I am testing it with two computers, connected through an unmanaged switch, and I am able to run a test ...
- 1
2
votes
0
answers
394
views
Relationship among different RDMA libraries
I'm new to RDMA technology, and I'm confused by several rdma libraries. I cannot figure out the relationship among them. The libraries and kernel headers I'm confused with are as follows:
1.rdma-...
- 41
0
votes
3
answers
2k
views
How to test RDMA code without actual hardware?
I have C++ code which makes use of infiniband verbs for RDMA communication. I need to unit test this code, and thus, the function calls related to RDMA such as ibv_get_device_list() need to succeed ...
- 1
0
votes
2
answers
878
views
MSN (message sequence number) in response for a retransmitted RDMA Read
While running ib_read_bw test for 64K message sizes from Mellanox CX-4 (request initiator) to another RNIC, re-transmissions are happening from Mellanox for the 5th RDMA-READ on-wards for 50KB of data ...
- 11
0
votes
2
answers
2k
views
Setting max outstanding work requests to be put on a Send Queue of a Queue Pair in RDMA
I am trying to create a QueuePair with ibv_create_qp() and I have to describe the size of the Queue Pair by setting the fields of the struct ibv_qp_cap and providing it to the create function. My ...
- 137
5
votes
1
answer
12k
views
What is the difference between OFED, MLNX OFED and the inbox driver
I'm setting up Infiniband networks, and I do not fully get the difference between the different software stacks.
OFED https://www.openfabrics.org/ofed-for-linux/
MLNX OFED https://www.mellanox.com/...
- 932
0
votes
1
answer
1k
views
Does gRPC+MPI require RDMA?
Tensorflow allows for the options "gRPC", "gRPC+verbs" and "gRPC+mpi" when specifying a communication protocol. In the gRPC+verbs documentation, it clearly states that this protocol is based on RDMA. ...
- 31
2
votes
1
answer
837
views
RDMA over native Infiniband
Is it possible to use RDMA over native InfiniBand without IPoIB (using just guid or lid)?
I have checked Infiniband addressing - host names to IB address without IBoIP and in RDMA Connection Manager ...
- 23
4
votes
1
answer
244
views
How to Configure Chapel/GASNet for running multilocale codes on MXM Infiniband network with Partition Key?
I'm trying to run a multilocale Chapel code on a cluster that has an MXM Infiniband network(40 Gbps, model: Mellanox Technologies MT26428).
I followed both Chapel and GASNet documentations, and I ...
- 307
1
vote
2
answers
2k
views
How can I receive Ethernet frames with ibverbs?
I want to write a simple test program to receive Ethernet frames using the ibverbs API.
The code below compiles and runs but never receives any packets. I'm using Mellanox ConnectX-3 hardware on ...
- 4,828
0
votes
1
answer
220
views
Will the RDMA enabled NIC do endian conversion?
Is it possible to get an RDMA adapter (e.g. Mellanox NIC) to do an endian conversion during data transfer? Specifically, we're doing an RDMA transfer from a big-endian to a little-endian system and ...
- 452