You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] exec --output json writes one array per 100-row batch for a provider SELECT (concatenated arrays, invalid JSON) #925
stackql exec --output json writes a provider SELECT result of more than 100 rows as several JSON arrays back to back, one per 100-row batch, with nothing between them. The output is not valid JSON, so every consumer that parses stdout fails on a result that large. pystackql reports it as Invalid JSON format: Extra data.
Reproduction
stackql v1.0.888 (Linux, f7b4116), the ovh provider (stackql-registry/stackql-provider-ovh), a list that returns 182 rows in one API response (GET /cloud/project/{serviceName}/flavor?region=BHS5, a bare JSON array):
stackql exec"SELECT id, name, vcpus, ram, disk FROM ovh.cloud.flavors WHERE region = 'BHS5'" --output json > out.json
exit status 0, stderr empty, stdout 18,507 bytes
json.loads fails with Extra data at offset 9978
the bytes around the offset: ..."name":"win-b2-15","ram":"15","vcpus":"4"}][{"disk":"50","id":"0062dad0-...
decoding array by array gives 2 arrays of 100 and 82 rows (182 in total, the right rows)
Through pystackql (StackQL(output="dict").execute(...)) the same statement returns:
[{"error": "Invalid JSON format: Extra data: line 1 column 9979 (char 9978)", "position": 9978, "line": 1, "column": 9979}]
What does not show it:
SHOW RESOURCES IN ovh.cloud_database (157 rows) is one array.
A provider SELECT whose final result is 100 rows or fewer is one array, including an aggregate over the same 182 rows (SELECT region, COUNT(*) FROM ovh.cloud.flavors GROUP BY region).
So it is the row batching of a provider read reaching the JSON writer: each batch is serialised as its own array.
Expected
One JSON document per statement on stdout: a single array holding every row, however many batches the rows arrived in. If a streaming form is wanted it should be an explicit output mode (newline-delimited JSON), not the default json.
Impact
Any exec --output json consumer and pystackql in its default (non-server) mode fail on results over 100 rows; smaller results work, so it shows up only once an estate grows. Found by the ovh provider's live smoke suite, which now keeps its exec reads under 100 rows as a workaround.
Test
A mocked list of 250 rows through stackql exec --output json: stdout parses as one array of 250. Worth the same check for --output csv (a single header row) and for a result that is exactly 100 and exactly 101 rows.
Cause is JSONWriter.writeRowsFromResult marshalling each 100-row Read() batch as its own array (internal/stackql/output/output.go ~L159), so large results come out as [...][...] back to back. Since --output jsonl/ndjson already covers the streaming case, the proposal is to buffer all batches and emit a single array for json, with a unit test for 100/101/250 rows. Happy to open a PR if that approach works for you.
Disclosure: drafted with AI assistance (Claude Code).
Activity
pcbeingused333 commentedon Nov 15, 2026
Cause is
JSONWriter.writeRowsFromResultmarshalling each 100-rowRead()batch as its own array (internal/stackql/output/output.go~L159), so large results come out as[...][...]back to back. Since--output jsonl/ndjsonalready covers the streaming case, the proposal is to buffer all batches and emit a single array forjson, with a unit test for 100/101/250 rows. Happy to open a PR if that approach works for you.Disclosure: drafted with AI assistance (Claude Code).