Skip to content

Include a caching document provider implementation by default #4107

Open

Description

@bbakerman

Caching the parsing and validation of a document can save significant latency.

For very large operations, and tools like Relay produce very large operations, the parse and validate latency can be upwards of 15-20ms (p99)

By caching this step you can amortize the cost over many requests and hence cut nearly all of this latency out.

The graphql.execution.preparsed.PreparsedDocumentProvider is the interface that allows you to build a caching document mechanism but out of the box graphql-java does not provide a working implementation

I propose that we

  • Create a public class CachingDocumentProvider implements PreparsedDocumentProvider
  • That is used Caffeine as the cache
  • Provide a GraphQLUnusualConfiguration mechanism to to tune the cache
    • time base
    • size based
    • etc
  • Ship with this TURN on by default (in 2 versions)
  • Shade in https://github.com/ben-manes/caffeine as the cache implementation

The implication here is that https://github.com/ben-manes/caffeine would become a dependency (shaded) and we have mostly aimed for low to zero dependencies but I think the juice is worth the squeeze here. All operations will run faster with a cache in place.

I dont know right now of the implications for library frameworks such as Spring GraphQL. Do they have one themselves, is it always on? Hence this issue to stand as a discussion mechanism

Activity

  1. bbakerman commented on Sep 12, 2025

    @bbakerman
    MemberAuthor
    com.github.ben-manes.caffeine:caffeine:3.1.8
       com.google.errorprone:error_prone_annotations:2.21.1
       org.checkerframework:checker-qual:3.37.0
    

    Caffeine has the following dependencies should we choose to include it as a dep

  2. andimarek commented on Sep 12, 2025

    @andimarek
    Member

    What is it specifically from caffeine that we would leverage?

    Could we not have a default implementation that ist just a hashmap and people could adapt caffeine to it themself?

    Other alternatives:

    • Release it as extra maven artifact

    • multi module jar (no idea how/if that would work)

    • have a runtime reflection detection like spring that leverages caffeine if in the classpath

  3. bclozel commented on Sep 12, 2025

    @bclozel

    What are the typical challenges for implementing such an interface?

    I noticed one implementation in DGS and it doesn't look too extensive. I'm wondering if shading (extra jar size, extra maintenance work for the team) is the best solution, since applications might want to use their custom version of caffeine, with custom options and maybe share the allocated memory with other bits of the application?

    Maybe a sample implementation in the reference docs and more guidance around persisted queries would be more valuable to the community.

  4. bbakerman commented on Sep 14, 2025

    @bbakerman
    MemberAuthor

    What is it specifically from caffeine that we would leverage?
    Could we not have a default implementation that ist just a hashmap and people could adapt caffeine to it themself?

    The reason for Caffeine is that we can then release graphql-java (eventually) with this caching turned on. This will make ALL graphql-java implementations faster - a lot laster - 5 / 10/ 20 ms faster

    We cant ship with a HashMap because of its unbounded nature. There must be a LRU type cache in place so we dont by default blow memory limits. And this is what leads us to Caffeine. Its a world class cache, it can efficiently evict entries and hence means we can have this on by default but be sensible in memory usage

    Other alternatives:
    Release it as extra maven artifact

    Yes considered that but then its can be on by default. Hence the 5 / 10/ 20 ms faster savings are not on by default.

    multi module jar (no idea how/if that would work)

    No idea either

    have a runtime reflection detection like spring that leverages caffeine if in the classpath

    This is not crazy however I would like to put the configuration of the cache into the graphql.GraphQLUnusualConfiguration and hence expose direct API level for configuring the cache

    LoadingCache<Key, Graph> graphs = Caffeine.newBuilder()
        .maximumSize(10_000)
        .expireAfterWrite(Duration.ofMinutes(5))
        .refreshAfterWrite(Duration.ofMinutes(1))
        .build(key -> createExpensiveGraph(key));
    

    but perhaps we can look more info this. eg becomes a NO op if there if Caffeine is not present.

  5. bbakerman commented on Sep 21, 2025

    @bbakerman
    MemberAuthor
    Benchmark                                            (queryCount)  (querySize)  Mode  Cnt     Score     Error  Units
    CachingDocumentBenchmark.benchMarkCachingOffAvgTime            10           50  avgt    6     0.922 ±   0.024  ms/op
    CachingDocumentBenchmark.benchMarkCachingOffAvgTime            10          500  avgt    6     8.414 ±   0.525  ms/op
    CachingDocumentBenchmark.benchMarkCachingOffAvgTime            10         5000  avgt    6    31.682 ±   0.284  ms/op
    CachingDocumentBenchmark.benchMarkCachingOffAvgTime            50           50  avgt    6     4.662 ±   0.357  ms/op
    CachingDocumentBenchmark.benchMarkCachingOffAvgTime            50          500  avgt    6    41.018 ±   0.626  ms/op
    CachingDocumentBenchmark.benchMarkCachingOffAvgTime            50         5000  avgt    6   154.708 ±   6.785  ms/op
    CachingDocumentBenchmark.benchMarkCachingOffAvgTime           500           50  avgt    6    47.500 ±   3.699  ms/op
    CachingDocumentBenchmark.benchMarkCachingOffAvgTime           500          500  avgt    6   414.916 ±   9.643  ms/op
    CachingDocumentBenchmark.benchMarkCachingOffAvgTime           500         5000  avgt    6  1623.578 ± 113.649  ms/op
    CachingDocumentBenchmark.benchMarkCachingOnAvgTime             10           50  avgt    6     0.090 ±   0.005  ms/op
    CachingDocumentBenchmark.benchMarkCachingOnAvgTime             10          500  avgt    6     0.832 ±   0.060  ms/op
    CachingDocumentBenchmark.benchMarkCachingOnAvgTime             10         5000  avgt    6     0.206 ±   0.001  ms/op
    CachingDocumentBenchmark.benchMarkCachingOnAvgTime             50           50  avgt    6     0.451 ±   0.021  ms/op
    CachingDocumentBenchmark.benchMarkCachingOnAvgTime             50          500  avgt    6     4.075 ±   0.038  ms/op
    CachingDocumentBenchmark.benchMarkCachingOnAvgTime             50         5000  avgt    6     0.390 ±   0.008  ms/op
    CachingDocumentBenchmark.benchMarkCachingOnAvgTime            500           50  avgt    6     4.420 ±   0.021  ms/op
    CachingDocumentBenchmark.benchMarkCachingOnAvgTime            500          500  avgt    6    41.472 ±   2.652  ms/op
    CachingDocumentBenchmark.benchMarkCachingOnAvgTime            500         5000  avgt    6     2.505 ±   0.200  ms/op
    
    Image

    This is some early benchmarking from having the caching in place. As you can see the larger the query the great the payback. This is expected since the time for parsing text is directly correlated with the size of the text.

    So caching query parsing and validation pays off as queries become larger and with tools like Relay "computer generating queries" they can get very large easily.

  6. paulbakker commented on Oct 6, 2025

    @paulbakker

    Hey folks, I think frameworks (e.g., Spring/DGS) should provide this instead.

    It's easy to provide for a framework, and that way graphql-java doesn't need to pull in a new external dependency.
    One of the strengths of GraphQL-Java is that it's a foundational library without many dependencies, configurations, or opinions. Adding caching by default gets into a gray area without much benefit because frameworks can already do this anyway.

    That being said, I agree on the benefits of having a cache; we're seeing this in practice as well.

  7. github-actions commented on Dec 6, 2025

    @github-actions
    Contributor

    Hello, this issue has been inactive for 60 days, so we're marking it as stale. If you would like to continue this discussion, please comment within the next 30 days or we'll close the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stalekeep-openTells Stale Bot to keep PRs and issues open

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions