We could put the method in a helper static class and save the result in an private static. So we will create the delegate only once.
I found the pattern in https://www.stevejgordon.co.uk/high-performance-logging-in-net-core.

Does the compiler not cache the delegate?

@iSazonov The delegate body for MatchEvaluator requires 2 closed-over variables from its lexically-enclosing method sb and context -- they are not passed to the delegate as paramters, and thus a generated helper class is needed.

The article you referred to is a good read, but the key perf improvement there is avoid processing the message template for every call to each of the log methods.

The key improvement here, vs. Standard logging, is that the template for the log message now only needs to be parsed once when this message is defined. In standard logging, that occurs for each call.

@xtqqczze The delegate here cannot be cached becuase it depends on the scirpt block sb and also the execution context context. These 2 closed-over variables make caching not useful in most cases.

The delegate body for MatchEvaluator requires 2 closed-over variables from its lexically-enclosing method sb and context -- they are not passed to the delegate as paramters, and thus a generated helper class is needed.

Can't we pass them as parameters?
I guess the pattern I mentioned excludes delegate allocations at all.

We seem to be talking about different things. :-)
One code path where the ReplaceOperatorImpl() is called is a while cycle. There the same context and the same scriptblock but we create the delegate again and again - what if the lval is a large list?

I see what you were talking about now :)
I'm not trying to optimize for all scenarios, but the most common scenario.

To optimize for the scenario you are talking about, you will need a delegate cache with both the scirpt block and the context together as the key. But as I said previously, the cache will not be useful for most cases. You need to think about how often would $ABigList -replace "pattern", $AScriptBlock be used. Holding the cache indefinitely would be a waste of memory; adding logic to clear the cache smartly would be extra complexity. Either way, I don't think the value would be worth the effort.

Either way, I don't think the value would be worth the effort.

After bad experience, I avoid such scripts. I suppose others do the same. But this does not mean that it cannot be improved. For example, the original script from the question could be rewritten using this pattern and if optimized, it could possibly run even faster than the original script.

I will open a tracking issue for the optimization.

For example, the original script from the question could be rewritten using this pattern and if optimized could possibly run even faster than the original script.

No, the original repro scirpt cannot be rewritten with $ABigList -replace "pattern", $AScriptBlock, because

1. the script uses many different patterns
2. the substitute strings are known, not requiring calculation based on $match. (be noted on the $AScriptBlock part)

The repro script uses -replace in a way that doesn't allocate delegate at all.

I am definitely in another dimension :-)

I implemented my thoughts in #14221.