Add support for custom allocators - #2697

Sorry for the delay.

1. I don't think we want to break the noexcept. It would be ok to do noexcept(myallocator.except) with a default allocator that is noexcept. The general rule should be to avoid breaking our public interface.
2. I am not too worried about performance and we will verify this anyhow.

Sorry for the delay.

No worries, thanks for the feedback!

I don't think we want to break the noexcept. [...] The general rule should be to avoid breaking our public interface.

Sounds good.

Just so I understand the guidance: Is simdjson aiming for source-code compatibility or ABI compatibility? Afaict, source-code compatibility would be retained even if we remove the noexcept? (At least for most "just call the functions" usages)

It would be ok to do noexcept(myallocator.except) with a default allocator that is noexcept

That would only work if we would pass in the allocator type as a template parameter - that's not what this commit is doing, though. This commit doesn't pass the allocator as a template parameter but as a type-erased runtime parameter.

With (1) being a no-go, I see two remaining options

1. Drop noexcept unconditionally (current PR).
2. Drop noexcept only when SIMDJSON_EXCEPTIONS is set.
3. Gate behind a new SIMDJSON_SUPPORT_THROWING_ALLOCATOR macro,
   defaulting to off.

For my use case, both would be fine. I would go with (3) since it means less code churn.

@vogelsgesang Can't we require the allocator to be noexcept ?

T* allocate(std::size_t n) noexcept {
        try {
 // blabla
        } catch (...) {
            return nullptr;
        }
 }

it's not perfect for my particular downstream use case, but works for me, too.

In my use case, the exceptions thrown by the allocator have additional payload information.

More details (feel free to skip)
We have two memory limits: one process-wide limit and a per-request limit.
We over-commit memory, i.e. we allow more requests to be in the system at the same time than we could serve if every request actually uses its full memory budget. We bet on the fact that most requests are simple and don't actually use the memory.
Our allocator instances track how much memory was already allocated per query and then throw the corresponding "per-request limit exceeded" or "global limit exceeded". Depending on which limit we reach, this counts differently against our SLA: If we hit the per-request limit, it's the users fault since he gave us a too complex query. This doesn't count against our SLA. On the other hand, if the query failed due to us over-comitting memory, it does count against our SLA, since this is an optimization done by us under the covers, outside the control of the user.

I see how this might be an edge case, and for the majority of users of simdjson, keeping the allocators noexcept might indeed be the better choice.

If we keep allocate as noexcept as you propose, I would simply do the following in my downstream code:

threadlocal std::exception_ptr threadlocal_exception_ptr;

// My custom allocator, implemented in my downstream code
struct tracking_allocator : allocator {
  // ... <other functions> ...
  simdjson::allocation_result allocate(size_t size) override {
    try {
      return backing_allocator->allocate(size_t size);
    } catch(...) {
      threadlocal_exception_ptr = std::current_exception();
      return nullptr;
    }
  }
  // ... <other functions> ...
};

auto someCaller() {
  tracking_allocator alloc(1);
  simdjson::dom::parser parser(alloc);
  simdjson::dom::element elem;
  auto result = parser.parse(jsonStr, jsonStr.size()).get();
  if (result == simdjson::MEMALLOC) {
    std::rethrow_exception(threadlocal_exception_ptr);
  }
  // ...<more code>...
}

@vogelsgesang We can also do the macro thing. (We have lots of little macros for specific use cases.)

But let me go to something I feel is important. The way the simdjson lib is built, you can effectively avoid allocations. That's the point of the parser, we have it have a capacity and then from there, it can basically be reused. That's really how we hope people use it on long running services.

So I am a little worried that you might be doing lots of allocation and de-allocation. Of course, you do what you want... but I just want to make sure that there is no confusion as to what we expect.

@vogelsgesang So, here for example...

auto someCaller() {
  tracking_allocator alloc(1);
  simdjson::dom::parser parser(alloc);
  simdjson::dom::element elem;
  auto result = parser.parse(jsonStr, jsonStr.size()).get();
  if (result == simdjson::MEMALLOC) {
    std::rethrow_exception(threadlocal_exception_ptr);
  }
...

We prefer that users do not allocate a parser each time they parse a document. The reason we have distinct type parser is precisely to control and limit allocations.

So let us say that each thread only ever parses one JSON document at any one time, then what we would suggest is a thread local parser.

And you can even put a limit on it (so that it throws or return an error if the input is too large). So you can create a parser, say that the parser can't exceed that much capacity and then reuse the parser.

Your memory usage will then be flat as far as the parser is concerned. If your purpose is to have bounded capacity, there is no need for a custom allocator at all. The functionality is built in!!!

But let me go to something I feel is important. The way the simdjson lib is built, you can effectively avoid allocations. That's the point of the parser, we have it have a capacity and then from there, it can basically be reused. That's really how we hope people use it on long running services.

Understood - Thank you for pointing out the potential performance pitfalls and that I should reuse the allocations!

Long story short: Don't worry, (I think?) I know what I am doing

My example regarding auto someCaller() was oversimplified and only focused on the the aspect of "how could I propagate the exception payload via a threadlocal" while ignoring the reuse of allocations. In practice, I will be reusing the parser.

(Also, I will be using an ondemand::parser instead of a dom::parser. In fact, I would even prefer a parser which pre-parses the document into a "tape" similar to the DOM parser, but into a single instead of two tapes as the DOM parser currently does - but I am digressing...)

And the long version: I am planning to integrate simdjson into a database system. This system will receive queries like

SELECT
  json_value(my_table.details, '$.stacktrace[0].symbol_name') AS crash_symbol,
  COUNT(*)
FROM log_events
WHERE json_value(my_table.metadata, '$.event.type') = 'crash'
GROUP BY crash_symbol

Note how:

1. I have an input table log_events. Let's assume it contains 1 million records
2. I have 2 json_value calls accessing JSON documents from 2 different columns (metadata and details)

-> I will be processing 2 million JSON documents for this query.

When starting to process this query, I would set up two simdjson::ondemand::parser instances in the thread-local storage of each thread, one for each of the json_value call sites.
For each new tuple from the log_events, I would reuse those two parsers.
After the query is done, I would tear down the parser instances.
I think the setup / teardown cost of two parsers per thread is acceptable for processing 2 million tuples.

And you can even put a limit on it (so that it throws or return an error if the input is too large). So you can create a parser, say that the parser can't exceed that much capacity and then reuse the parser.

Afaict, I can only set a static limit via this mechanism and there is no callback mechanism "I am about to allocate X MB - is that ok right now?".

In my case, whether an allocation should be successful also depends on what the other parts of the query are doing. I want to limit the overall usage for the complete query, not only of each individual component. JSON-parsing is just one potential user of memory.

E.g., if the hash table for the GROUP BY crash_symbol is already taking up 99MB, then the simdjson parser would have only 1MB of the overall query budget of 100MB left. However, if the hash table happens to only take 5 MB, the simdjson parser would be allowed to use up to 95MB.

The easiest way I could find to make such memory budgeting work across a wide set of data structures (tuple buffers, hash tables, regular expression engines, ...) was to pass each of those component the same (query-local) allocator. As soon as the memory allocated via that allocator goes over the budget, I throw an exception and thereby fail the query

@vogelsgesang

We could also keep the style of the current PR and remove the noexcept if we can establish that it does not affect the performance. That would require a major release, of course.

Afaict, I can only set a static limit via this mechanism and there is no callback mechanism "I am about to allocate X MB - is that ok right now?".

If the parser has max capacity X and you try to parse something larger, it will fail right away, with some kind of memory error. It is designed so that you can forbid the parser from allocating (at all).

We could also keep the style of the current PR and remove the noexcept if we can establish that it does not affect the performance. That would require a major release, of course.

The longer I think about it, the more I realize that removing the noexcept is actually the least important part of this change. I think I would just keep the noexcept.

From the earlier options

1. Drop noexcept unconditionally (current PR).
2. Drop noexcept only when SIMDJSON_EXCEPTIONS is set.
3. Gate behind a new SIMDJSON_SUPPORT_THROWING_ALLOCATOR macro, defaulting to off.

(2) and (3) also aren't nice, because throwing allocators would be a rarely used feature, and new code would probably add new unconditional noexcepts, which has a risk of inadvertently breaking the SIMDJSON_SUPPORT_THROWING_ALLOCATOR config. The threadlocal approach is good enough for me.

I will update the PR soon(-ish) to keep the noexcept in place