2

I'm trying to reading PCI CSR (Configuration Space Register) on my system via open,mmap /dev/mem.

I met some problems when using 8 byte length reading

Here is the minimal working example of my code

#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

#define FATAL                                                                 \
    do {                                                                      \
        fprintf(stderr,                                                       \
                "Error at line %d, file %s (%d) [%s]\n",                      \
                __LINE__,                                                     \
                __FILE__,                                                     \
                errno,                                                        \
                strerror(errno));                                             \
        exit(1);                                                              \
    } while(0)
#define PAGE_SIZE 4096UL
#define PAGE_MASK (PAGE_SIZE - 1)

typedef struct rw_config rw_config;

struct rw_config {
    uint64_t address;
    uint64_t data;
};

static uint64_t _mmio_read_worker(uint64_t address) {
    int fd;
    void *map_base = NULL;
    void *map_address = NULL;
    uint64_t result = 0UL;

    if((fd = open("/dev/mem", O_RDONLY | O_SYNC)) < 0) FATAL;
    // PAGE_SIZE = 4096UL
    // PAGE_MASK = (PAGE_SIZE - 1) = 4095UL
    if((map_base = mmap(NULL,
                        PAGE_SIZE,
                        PROT_READ,
                        MAP_SHARED,
                        fd,
                        (address & ~PAGE_MASK)))
       == MAP_FAILED)
        FATAL;
    map_address = map_base + (address & PAGE_MASK);

    result = *(uint64_t *)map_address;
    printf("uint32_t 0x%016x, uint64_t 0x%016lx\n",
           (*(uint32_t *)map_address),
           (*(uint64_t *)map_address));

    close(fd);
    return result;
}

void rw_worker(rw_config *cfg) {
    cfg->data = _mmio_read_worker(cfg->address);
    return;
}

int main(int argc, char *argv[]) {
    rw_config *cfg = malloc(sizeof(rw_config));

    cfg->address = 0x80000000;
    cfg->data = 0x0;

    rw_worker(cfg);

    return 0;
}

Reading the address = 0x80000000 which is pci mmio base address.

The output of my code is as follows:

uint32_t 0x0000000009a28086, uint64_t 0xffffffffffffffff

And I try to using gdb to get some information.

(gdb) printf "0x%llx\n",(*(uint64_t *)map_address)
0x10000009a28086
# before assigning 'result'
(gdb) printf "0x%llx\n",result
0x0
(gdb) next
# after assigning 'result'
(gdb) printf "0x%llx\n",result
0xffffffffffffffff
(gdb) print map_address
$2 = (void *) 0x7ffff7ffb000
(gdb) x/1xg 0x7ffff7ffb000
0x7ffff7ffb000: 0x0010000009a28086

I guess I fail to casting (void*) to *(uint64_t *), but why?

The value storage in map_address is correct, am I using the wrong way to get the value?

11
  • Edit the question to provide a minimal reproducible example, including a call to _mmio_read_worker showing how the address is obtained. Commented Jan 6, 2023 at 8:30
  • What is sizeof(long unsigned) on your system? Not sure that's the bug but printing uint64_t (and friends) using the "normal" %-specifier is dangerous. Ude PRIu64 or PRIx64 Commented Jan 6, 2023 at 8:30
  • Which value did address have in your test? Commented Jan 6, 2023 at 8:34
  • 2
    In gdb you use %llx for printing 64 bit values but in the code you do %lx. Why? Commented Jan 6, 2023 at 8:36
  • 1
    I'm not too familiar with the details of PCI, but I suppose there is some memory-mapped hardware register at address 0x80000000. When you read an I/O register, you have to use the proper size. If it's documented as a 32-bit register, you have to do a 32-bit read. It is quite possible for a read of a different size to return incorrect and inconsistent results. Although the hardware is mapped into the memory address space, it isn't actually memory and you shouldn't expect it to behave like memory. Commented Jan 6, 2023 at 17:23

2 Answers 2

0

After reading the replies from other members, I read some documents that may be related to this bug, and the following are some of my insights:

  • I tried testing with another address which NOT in PCI CSR(Configuration Space Register), and got the correct value. So I think this bug is related to hardware registers rather than software implementation

  • In EDK II UEFI Writer Guide link, using 64bits read on PCI BAR(Base Address Register, which is a part of PCI CSR) may cause an alignment fault, you should use 2x of 32bits read to achieve 64bits read. Although in the example it is not enforced that the whole CSR has this limitation, but I think there is already a good reason for this bug.

Sign up to request clarification or add additional context in comments.

Comments

0

PCIe config space must be read using 1, 2, or 4-byte accesses. 8-byte accesses are not permitted.

This is specified in the PCIe spec, section 2.2.7.1: "For Configuration Requests, Length[9:0] must be 00 0000 01b." (Length is specified in DW.)

My experience is that 8-byte accesses always return FF in all bytes.

8 Comments

Are talking about Type 1 accesses or Type 2? I believe Type 2 can be read in 8 bytes if CPU supports that.
@0andriy, I think the actual read to the device cannot be 8 bytes because the PCIe message definition for config space accesses doesn't have a way to represent an 8-byte access. The root complex is allowed to break up an 8-byte access into two 4-byte accesses, but I don't know if any actually do that.
Maybe I was not clear. Under Type 2 I, probably mistakenly, meant ECAM access. I'm not sure how this can't be supported of reading 8-byte at a time (on 64-bit CPU, of course).
@0andriy, yes, I understood what you meant. I assure you it isn't supported. It will return FFFFFFFFFFFFFFFF, because the PCIe protocol doesn't define a way to perform an 8-byte access to the device's config space and the root complex doesn't break it into two PCIe transactions.
@0andriy, this is specified in PCIe spec 2.2.7.1: "For Configuration Requests, Length[9:0] must be 00 0000 01b." (Length is specified in DW.)
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.