Skip to content

feat: add task API for bidirectional background worker communication#2319

Open
nicolas-grekas wants to merge 2 commits intophp:mainfrom
nicolas-grekas:sidekicks-tasks
Open

feat: add task API for bidirectional background worker communication#2319
nicolas-grekas wants to merge 2 commits intophp:mainfrom
nicolas-grekas:sidekicks-tasks

Conversation

@nicolas-grekas
Copy link
Copy Markdown
Contributor

This PR builds on top of #2287 (background workers), which is the first commit.

The following description is only about what's currently the second commit.

An overview of both features combined together can be reviewed by reading the doc attached to this PR.

Summary

Adds a task API for bidirectional communication between HTTP workers and background workers. While set_vars/get_vars pushes config from background workers to HTTP workers, tasks enable the reverse: HTTP workers dispatch work to background workers and stream results back.

PHP API

Sender-side:

  • frankenphp_worker_task_send(string $name, array $payload, float $timeout = 30.0): resource - sends a task to a named background worker, returns a readable stream for results
  • frankenphp_worker_task_read(resource $stream): ?array - reads the next update, returns null on clean completion

Receiver-side:

  • frankenphp_worker_task_receive(): ?array - dequeues a pending task to process, returns [$stream, $payload] or null
  • frankenphp_worker_task_update(resource $stream, array $data): void - sends a progress update or result back to the sender

All payloads and updates follow the same type constraints as set_vars: null, bool, int, float, string, array (nested), and enums. Objects and resources are rejected.

How it works

  1. HTTP worker calls task_send('worker-name', $payload) - blocks until a background worker picks up the task
  2. Background worker receives "task\n" on the signaling stream, calls task_receive() to get [$stream, $payload]
  3. Background worker processes the payload, sends results via task_update($stream, $data)
  4. HTTP worker reads results via task_read($stream) - wakes up on each update
  5. Background worker calls fclose($stream) when done - sender gets null from task_read()
  6. Sender calls fclose($stream) to acknowledge completion

Blocking behavior

  • task_send blocks until the background worker picks up the task (with timeout). Uses a buffered channel (size 1) so the sender can enqueue before the "task\n" signal reaches the receiver.
  • task_read blocks until the next update arrives or the stream is closed. The task stream from task_send is stream_select-compatible, so callers can check readability before calling task_read.
  • Sender can cancel a task by calling fclose() before the background worker completes. Cancelled tasks are detected at task_receive() time and skipped.

Crash detection

  • If the background worker exits without calling fclose($stream) (crash, exit(), fatal error), task_read() throws RuntimeException
  • Clean completion (fclose by the background worker) returns null from task_read()
  • Detection uses EG(flags) & EG_FLAGS_IN_SHUTDOWN to distinguish explicit fclose() from resource cleanup during script exit

Pooling

Named background workers support num > 1 to run multiple threads. All threads share the same task channel - tasks are distributed automatically across the pool. The signaling stream fans out "task\n" to all threads.

Example

// Background worker: process tasks
$signaling = frankenphp_worker_get_signaling_stream();

while (true) {
    $r = [$signaling];
    $w = $e = [];
    if (!stream_select($r, $w, $e, 30)) { continue; }

    $signal = fgets($signaling);
    if ("stop\n" === $signal) { break; }

    if ("task\n" === $signal && [$stream, $payload] = frankenphp_worker_task_receive()) {
        frankenphp_worker_task_update($stream, ['result' => process($payload)]);
        fclose($stream);
    }
}
// HTTP worker: send a task and read the result
$stream = frankenphp_worker_task_send('image-resizer', ['file' => 'photo.jpg']);
$result = frankenphp_worker_task_read($stream);
fclose($stream);

Architecture

  • taskRequest struct coordinates sender and receiver via closedSides atomic counter - task slot is freed when both sides close
  • taskFIFO provides bounded backpressure for updates (max 16 items, blocks on push when full)
  • Pipe-based signaling: each task gets a pipe pair. Background worker writes nudge bytes via task_update, sender detects them via stream_select
  • Custom PHP stream ops for both sender (read-only, wraps pipe read fd) and receiver (write-only, wraps task id)
  • Task signal fan-out uses backgroundFdList stored on backgroundWorkerState for O(1) lookup (no worker search)

Test coverage

10 tests covering: basic task send/receive, progress streaming, non-worker sender, crash before receive, crash mid-task, cancellation, cancel-then-send recovery, cancel-then-crash, pooling (num=2).

Documentation

Task API section added to docs/background-workers.md.

@nicolas-grekas nicolas-grekas force-pushed the sidekicks-tasks branch 2 times, most recently from cf660f6 to bb3febd Compare March 28, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant