Skip to content

[JsonStreamer] High Memory Usage #61559

@natewiebe13

Description

@natewiebe13

Symfony version(s) affected

7.x

Description

I'd like to use the JsonStreamer component to work with an API that returns a large dataset in json (~200k items) where the array of items is a nested property. Attempting to do so results is very high memory usage. After some digging, it also appears to happen even if it's a flat array.

I've created a minimal repo to reproduce the issue for both a 1D array, as well as a nested array: https://github.com/natewiebe13/symfony-json-streamer-memory

In my tests, I'm getting about 200MB of memory usage if I load the entire JSON structure into memory and iterating. If I use halaxa/json-machine it uses about 25MB of memory, and using the JsonStreamer component, it uses 2.1GB

At this point, I'm not sure if my usage is incorrect, or if there's something that can be improved in the component/documentation.

How to reproduce

Reproducer: https://github.com/natewiebe13/symfony-json-streamer-memory

Possible Solution

No response

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions