Hello,
Thank you for the response. I did read that section before I posted this question.

For performance reasons, the simdjson library requires that the JSON input contain at least simdjson::SIMDJSON_PADDING bytes at the end of the stream. The value simdjson::SIMDJSON_PADDING is small (e.g., 64 bytes). On modern systems, you can safely read beyond an allocated buffers, as long as you remain within an allocated page. Pages on modern systems span at least 4 kilobytes, but can be significantly larger. E.g., Apple systems favour pages spanning 16 kilobytes.

The part I felt confused is that it gave me a mixed information.

For performance reasons, the simdjson library requires that the JSON input contain at least simdjson::SIMDJSON_PADDING bytes at the end of the stream.

So this means that I have to allocate a buffer that can store the original json data + 64 bytes in order to make it work for ondemand parsing.

On modern systems, you can safely read beyond an allocated buffers, as long as you remain within an allocated page.

What does this mean for the parser users?
Can I cheat and do the following even though the raw_data has no padding in the end?

const char* raw_data = ...;
const size_t buffer_size = ...;
simdjson::ondemand::document simdjson_msg = parser.iterate(raw_data, buffer_size, buffer_size + 64);

Also, with this extra copy when I use ondemand, will it still offer better performance comparing to the simdjson::dom::parser?

Thank you

@Jack012a

I recommend you share a full piece of code so that I can understand what you are doing. Currently, it is too abstract to be useful.

The raw_data is received through network

The raw_data is very likely to be part of your memory subsystem (i.e., in RAM). It is likely accessible as virtual memory part of memory pages. Either your process or the operating system allocated the memory.

Can you be more specific as to where raw_data comes from?

A typical network access might look as follows:

    int server_fd, new_socket;
    struct sockaddr_in address;
    int addrlen = sizeof(address);
    char buffer[BUFFER_SIZE] = {0};
    
    // Creating socket file descriptor
    if ((server_fd = socket(AF_INET, SOCK_STREAM, 0)) == 0) {
        perror("socket failed");
        exit(EXIT_FAILURE);
    }

    // Bind socket to address and port
    address.sin_family = AF_INET;
    address.sin_addr.s_addr = INADDR_ANY;
    address.sin_port = htons(PORT);

    if (bind(server_fd, (struct sockaddr *)&address, sizeof(address)) < 0) {
        perror("bind failed");
        close(server_fd);
        exit(EXIT_FAILURE);
    }

    // Listen for incoming connections
    if (listen(server_fd, 3) < 0) {
        perror("listen");
        close(server_fd);
        exit(EXIT_FAILURE);
    }

    printf("Server listening on port %d\n", PORT);

    // Accept a connection
    if ((new_socket = accept(server_fd, (struct sockaddr *)&address, (socklen_t*)&addrlen)) < 0) {
        perror("accept");
        close(server_fd);
        exit(EXIT_FAILURE);
    }

    // Receive data from the client
    int valread = read(new_socket, buffer, BUFFER_SIZE);
    printf("Received message: %s\n", buffer);

    // Closing the connected socket
    close(new_socket);
    // Closing the listening socket
    close(server_fd);

Notice how the data is received in memory we allocated (here buffer).

Can I cheat and do the following even though the raw_data has no padding in the end?

The explanation is here: "On modern systems, you can safely read beyond an allocated buffers, as long as you remain within an allocated page."

This discussion is outside our basic documentation is meant for advanced users who know about heap memory, pages, virtual memory and so forth.

The explanation is accompanied by a source code example, here is part of it:

If you are not an advanced programmer with in-depth knowledge of how modern operating systems work, I recommend you just create a copy. Copies are very cheap.

Also, with this extra copy when I use ondemand, will it still offer better performance comparing to the simdjson::dom::parser?

Whether you use On-Demand or DOM, we expect padding.

I understand that you have performance concerns. Have you run benchmarks? Can you share your results?

If you are getting poor performance, can you produce a reproducible test case?