some metrics, will try to tune them, I think we can just maintain a percentage of $things-served-by-slab as a single metric per-class

[("pipeline_name", "pipeline"), ("size_class_bytes", "1024")]
storage_fbuf_slab_size_class_alloc_requests_total:         9715.00
storage_fbuf_slab_size_class_free_fallbacks_total:            0.00
storage_fbuf_slab_size_class_frees_saved_total:        11637.00
storage_fbuf_slab_size_class_mallocs_saved_total:         9689.00

[("pipeline_name", "pipeline"), ("size_class_bytes", "16384")]
storage_fbuf_slab_size_class_alloc_requests_total:       801012.00
storage_fbuf_slab_size_class_free_fallbacks_total:        22801.00
storage_fbuf_slab_size_class_frees_saved_total:       704663.00
storage_fbuf_slab_size_class_mallocs_saved_total:       703974.00

[("pipeline_name", "pipeline"), ("size_class_bytes", "2048")]
storage_fbuf_slab_size_class_alloc_requests_total:        10986.00
storage_fbuf_slab_size_class_free_fallbacks_total:            0.00
storage_fbuf_slab_size_class_frees_saved_total:        11982.00
storage_fbuf_slab_size_class_mallocs_saved_total:        10849.00

[("pipeline_name", "pipeline"), ("size_class_bytes", "32768")]
storage_fbuf_slab_size_class_alloc_requests_total:           49.00
storage_fbuf_slab_size_class_free_fallbacks_total:            0.00
storage_fbuf_slab_size_class_frees_saved_total:            2.00
storage_fbuf_slab_size_class_mallocs_saved_total:            1.00

[("pipeline_name", "pipeline"), ("size_class_bytes", "4096")]
storage_fbuf_slab_size_class_alloc_requests_total:     37303172.00
storage_fbuf_slab_size_class_free_fallbacks_total:      3445829.00
storage_fbuf_slab_size_class_frees_saved_total:     37272596.00
storage_fbuf_slab_size_class_mallocs_saved_total:     37268507.00

[("pipeline_name", "pipeline"), ("size_class_bytes", "512")]
storage_fbuf_slab_size_class_alloc_requests_total:         9398.00
storage_fbuf_slab_size_class_free_fallbacks_total:            0.00
storage_fbuf_slab_size_class_frees_saved_total:        11077.00
storage_fbuf_slab_size_class_mallocs_saved_total:         9379.00

[("pipeline_name", "pipeline"), ("size_class_bytes", "65536")]
storage_fbuf_slab_size_class_alloc_requests_total:           48.00
storage_fbuf_slab_size_class_free_fallbacks_total:            0.00
storage_fbuf_slab_size_class_frees_saved_total:            0.00
storage_fbuf_slab_size_class_mallocs_saved_total:            0.00

[("pipeline_name", "pipeline"), ("size_class_bytes", "8192")]
storage_fbuf_slab_size_class_alloc_requests_total:     70890244.00
storage_fbuf_slab_size_class_free_fallbacks_total:      3773457.00
storage_fbuf_slab_size_class_frees_saved_total:     63522642.00
storage_fbuf_slab_size_class_mallocs_saved_total:     63522136.00

@gz is there a measurable impact in terms of fewer allocations, cpu usage, pipeline throughput?

This is seemingly a lot of mechanism. I hope there is commensurate benefit.

My first thought on how to do this is to make the cache thread-local with std::thread_local!. Then every thread would have its own cache and there would be no need to have anything special in the runtime. (Stats maybe would be harder.) That could be simpler than what is here.

This is seemingly a lot of mechanism. I hope there is commensurate benefit.

we see jemalloc using a significant amount of CPU cycles using madvise.
we can configure jemalloc to use background threads at which points it offloads this work and things get faster (in the workload where this problem is prominent we're seeing reduction of ~1h backfill time)
however, it then uses 4 threads that do nothing else but call madvise

all of this seems stupid and really jemalloc's problem, so the goal here is to alleviate malloc pressure. I don't expect performance gain form this compared to jemalloc with background threads, but I expect us not having to waste 4 cores doing nothing but madvise

My first thought on how to do this is to make the cache thread-local with std::thread_local

It might be a better strategy to be per-thread, I'll experiment and look at if it's better/worse in terms of avoiding mallocs/frees. I'm not sure how to add metrics and use dev-tweaks to override defaults without having the Runtime be aware of the slabs so will probably need to keep that part.

This is seemingly a lot of mechanism. I hope there is commensurate benefit.

we see jemalloc using a significant amount of CPU cycles using madvise. we can configure jemalloc to use background threads at which points it offloads this work and things get faster (in the workload where this problem is prominent we're seeing reduction of ~1h backfill time) however, it then uses 4 threads that do nothing else but call madvise

There's a (disputed) claim on hackernews that jemalloc wastes a lot of cpu time with madvise because the kernel wastes cpu time zeroing those pages before they circulate right back to the process: https://news.ycombinator.com/item?id=47404107. I don't know whether that helps at all.

There's a (disputed) claim on hackernews that jemalloc wastes a lot of cpu time with madvise because the kernel wastes cpu time zeroing those pages before they circulate right back to the process

I saw that too, from whatever we see the fb person might be right :)

we did experiment with mimalloc and this problem doesn't really show up anymore then; but the downside of switching allocators is we don't get heap profiles which can be very valuable when something goes wrong