Summary

Implements #19356 - adds a new JsonStreamResponse class for memory-efficient streaming of large JSON datasets using generators.

Key Features

• True streaming: Uses echo + flush() to send data to the client as each item is encoded, keeping memory usage constant regardless of dataset size
• Dual format support: Standard JSON arrays and NDJSON (newline-delimited JSON)
• Envelope/wrapper structures: Support for root key wrapping and full envelope with static metadata
• Transform callbacks: Custom transformation of items before encoding
• Graceful error handling: Three-layer strategy with pre-validation, mid-stream error markers, and server-side logging
• Debug mode: Pretty-printed output when Configure::read('debug') is true
• PSR-7 immutability: withStreamOptions() returns new instance

Memory Profile

With true streaming, memory usage stays constant regardless of dataset size:

The implementation uses echo + flush() to send data to the client immediately as each item is encoded, rather than building a complete string in memory first.

Architecture

Class Location

Cake\Http\Response\JsonStreamResponse extending Cake\Http\Response

Options

Usage Examples

// Simple array streaming
return new JsonStreamResponse($query);
// Output: [{...}, {...}, ...]

// With root wrapper
return new JsonStreamResponse($query, ['root' => 'articles']);
// Output: {"articles": [{...}, {...}]}

// With envelope
return new JsonStreamResponse($query, [
    'envelope' => ['meta' => ['total' => 5000, 'page' => 1]],
    'dataKey' => 'articles',
]);
// Output: {"meta": {"total": 5000, "page": 1}, "articles": [{...}, {...}]}

// NDJSON format
return new JsonStreamResponse($query, ['format' => 'ndjson']);
// Output: {"id": 1, ...}\n{"id": 2, ...}\n...

// With transform
return new JsonStreamResponse($query, [
    'transform' => fn($article) => ['id' => $article->id, 'title' => $article->title],
]);

Error Handling Strategy

1. Pre-validation: First item encoded before output starts - if it fails, proper 500 response possible
2. Mid-stream error marker: Later failures output {"__streamError": {"message": "...", "index": N}} to maintain valid JSON
3. Server-side logging: All encoding failures logged via Log::error()

Content-Type Headers

Additionally, X-Accel-Buffering: no is set to prevent nginx proxy buffering.

ORM Integration

For true streaming benefits, use unbuffered queries and avoid result formatters:

// Good - streams one row at a time
$query = $this->Articles->find()->bufferResults(false);
return new JsonStreamResponse($query);

// Avoid - formatters like map(), combine() buffer results internally
$query = $this->Articles->find()->map(fn($row) => $row); // Breaks streaming

Note: Result formatters (map(), combine(), etc.) buffer results internally, which defeats the memory-efficient streaming purpose.