Skip to content

Massive GCs with failing upstreams #1788

Closed

Description

@jhaals

Hi,

We've noticed huge spikes in garbage collection time when an upstream to the graphql service starts timing out. JFR recordings show 41GB byte[] allocations followed by 5.54GB StringBuilder and another 3.41GB in String allocations. This is during a 60s sample when the service have upstream issues.

Most/all of this come from the error reporting doing toString() on the client responses stating something similar to the example below. These will repeat for each item missing which is generating a ton of garbage. I have two questions;

  • Can I modify the "stacktrace" like error reporting for each field to something simpler just saying error?
  • Is there a way to stop the execution early on if a certain condition is met? For example if this field fail to resolve, stop and return an error.
      "message": "Exception while fetching data (/dogShelter/dogs/items[9]/name) : Request timed out",
      "locations": [
        {
          "line": 5,
          "column": 9
        }
      ],
      "path": [
        "dogShelter",
        "dogs",
        "items",
        9,
        "name"
      ],

Screenshot 2020-02-12 at 09 22 26
Screenshot 2020-02-12 at 09 22 50

Activity

  1. changed the title [-]Massive GCs when with failing upstreams[/-] [+]Massive GCs with failing upstreams[/+] on Feb 12, 2020
  2. bbakerman commented on Feb 17, 2020

    @bbakerman
    Member

    So looking at the toString() method is defintely has a design fault. It references its parent ExecutionStepInfo which will reference its parent and so on.

    So it creates more "strings" than is strictly needed. We will fix this

    The reason it happens via NonNullableFieldExceptioon is this

        java.lang.Throwable#Throwable(java.lang.Throwable)
    
        public Throwable(Throwable cause) {
            fillInStackTrace();
            detailMessage = (cause==null ? null : cause.toString());
            this.cause = cause;
        }
    
    

    It causes a call to .toString() on the throwable as a message (when there is no message)

    which calls

    graphql.execution.NonNullableFieldWasNullException#toString
    
    public String toString() {
            return "NonNullableFieldWasNullException{" +
                    " path=" + path +
                    " executionStepInfo=" + executionStepInfo +
                    '}';
        }
    

    So the extra strings are coming from that.

    You must have 10s of 1000s of erroring requests here to be generated so many strings (GBs of them) however. Out of interest how many requests are we talking and how many are in errors.

  3. jhaals commented on Feb 17, 2020

    @jhaals
    Author

    Yeah it's definitely toString() casing the issues and your PR will be an improvement to that.
    Are the pretty toString's needed if there's nothing in the library reading them? Feels like an unnecessary overhead with strings that get thrown away.

    Currently a few hundred RPS but we expect that to increase a lot. The problem that occur is that some of the responses contain lists with 10-50 items and the toString is triggered for each field

  4. jhaals commented on Aug 27, 2020

    @jhaals
    Author

    Closing since this has been addressed in #1792

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions