Skip to content

[BUG] exec --output json writes one array per 100-row batch for a provider SELECT (concatenated arrays, invalid JSON) #925

Open

Description

@jeffreyaven

Problem

stackql exec --output json writes a provider SELECT result of more than 100 rows as several JSON arrays back to back, one per 100-row batch, with nothing between them. The output is not valid JSON, so every consumer that parses stdout fails on a result that large. pystackql reports it as Invalid JSON format: Extra data.

Reproduction

stackql v1.0.888 (Linux, f7b4116), the ovh provider (stackql-registry/stackql-provider-ovh), a list that returns 182 rows in one API response (GET /cloud/project/{serviceName}/flavor?region=BHS5, a bare JSON array):

stackql exec "SELECT id, name, vcpus, ram, disk FROM ovh.cloud.flavors WHERE region = 'BHS5'" --output json > out.json
  • exit status 0, stderr empty, stdout 18,507 bytes
  • json.loads fails with Extra data at offset 9978
  • the bytes around the offset: ..."name":"win-b2-15","ram":"15","vcpus":"4"}][{"disk":"50","id":"0062dad0-...
  • decoding array by array gives 2 arrays of 100 and 82 rows (182 in total, the right rows)

Through pystackql (StackQL(output="dict").execute(...)) the same statement returns:

[{"error": "Invalid JSON format: Extra data: line 1 column 9979 (char 9978)", "position": 9978, "line": 1, "column": 9979}]

What does not show it:

  • SHOW RESOURCES IN ovh.cloud_database (157 rows) is one array.
  • A provider SELECT whose final result is 100 rows or fewer is one array, including an aggregate over the same 182 rows (SELECT region, COUNT(*) FROM ovh.cloud.flavors GROUP BY region).

So it is the row batching of a provider read reaching the JSON writer: each batch is serialised as its own array.

Expected

One JSON document per statement on stdout: a single array holding every row, however many batches the rows arrived in. If a streaming form is wanted it should be an explicit output mode (newline-delimited JSON), not the default json.

Impact

Any exec --output json consumer and pystackql in its default (non-server) mode fail on results over 100 rows; smaller results work, so it shows up only once an estate grows. Found by the ovh provider's live smoke suite, which now keeps its exec reads under 100 rows as a workaround.

Test

A mocked list of 250 rows through stackql exec --output json: stdout parses as one array of 250. Worth the same check for --output csv (a single header row) and for a result that is exactly 100 and exactly 101 rows.

Activity

  1. pcbeingused333 commented on Nov 15, 2026

    @pcbeingused333
    Contributor

    Cause is JSONWriter.writeRowsFromResult marshalling each 100-row Read() batch as its own array (internal/stackql/output/output.go ~L159), so large results come out as [...][...] back to back. Since --output jsonl/ndjson already covers the streaming case, the proposal is to buffer all batches and emit a single array for json, with a unit test for 100/101/250 rows. Happy to open a PR if that approach works for you.

    Disclosure: drafted with AI assistance (Claude Code).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions