Skip to content

feat(site): add smooth streaming text engine for LLM responses#22503

Merged
kylecarbs merged 5 commits intomainfrom
smooth-streaming-text
Mar 2, 2026
Merged

feat(site): add smooth streaming text engine for LLM responses#22503
kylecarbs merged 5 commits intomainfrom
smooth-streaming-text

Conversation

@kylecarbs
Copy link
Copy Markdown
Member

Problem

LLM responses currently stream in bulk chunks — multiple message_part events arrive per WebSocket frame, get batched into a single startTransition state update, and render as a visual jump. This looks janky compared to smooth character-by-character reveal.

Solution

Port the jitter-buffer approach from coder/mux into a single self-contained file: SmoothText.ts.

What's in the file

Component Purpose
STREAM_SMOOTHING constants Tuning knobs (72–420 cps adaptive rate, 120 char max visual lag, 48 char frame cap)
SmoothTextEngine class Pure state machine — two-clock model (ingestion vs presentation) with budget-gated adaptive reveal
useSmoothStreamingText hook React bridge via requestAnimationFrame loop, single useState<number>, grapheme-safe slicing

How the engine works

  • Adaptive rate: Linear interpolation from 72 → 420 chars/sec based on backlog pressure (how far behind the display is from ingested text)
  • Budget accumulation: Fractional character budget accrues per RAF tick. Only reveals when ≥1 whole character is ready. This makes it frame-rate invariant — 60Hz and 240Hz displays reveal the same amount over wall-clock time (tested to ≤2 char deviation)
  • Max visual lag: Hard cap of 120 chars. If the gap exceeds this, the visible pointer jumps forward immediately
  • Clean flush: When streaming ends, remaining buffer appears instantly — no trailing animation
  • Grapheme safety: Uses Intl.Segmenter (with codepoint fallback) to never split emoji mid-animation

Integration

To wire this up, wrap the <Response> component in ConversationTimeline.tsx with the hook:

const SmoothedResponse: FC<{text: string; isStreaming: boolean; streamKey: string}> =
    ({ text, isStreaming, streamKey }) => {
        const { visibleText } = useSmoothStreamingText({
            fullText: text,
            isStreaming,
            bypassSmoothing: false,
            streamKey,
        });
        return <Response>{visibleText}</Response>;
    };

Tests

8 engine tests covering: steady reveal, adaptive acceleration, max lag cap, immediate flush on stream end, bypass mode, content shrink, sub-char budget gating, and frame-rate invariance.

Ports the jitter-buffer approach from coder/mux to smooth out bursty
LLM token delivery into steady character-by-character reveal.

SmoothText.ts contains:
- SmoothTextEngine: pure state machine with adaptive-rate budget model
  (72-420 cps, frame-rate invariant, max 120 char visual lag)
- useSmoothStreamingText: React hook bridging the engine via RAF loop
  with single-integer state, grapheme-safe slicing, and self-terminating
  animation
- STREAM_SMOOTHING constants for tuning

SmoothText.test.ts covers all 8 engine behaviors including frame-rate
invariance across 60Hz/240Hz displays.
Adds SmoothedResponse component that wraps <Response> with the
useSmoothStreamingText hook during live streaming. Historical
messages bypass smoothing and render through <Response> directly.

The integration is minimal: renderBlockList's 'response' case now
branches on isStreaming to pick SmoothedResponse vs plain Response.
- biome import ordering (alphabetical)
- biome-ignore for intentional useEffect deps (fullText, streamKey
  trigger RAF re-arm and stream reset)
- minor formatting adjustments
UseSmoothStreamingTextOptions and UseSmoothStreamingTextResult are
only consumed internally by useSmoothStreamingText. They can be
re-exported later if external consumers need them.
… SmoothText (#22509)

## Summary

Move RAF loop ownership and subscriber management into
`SmoothTextEngine`. The hook now uses `useMemo` (keyed on `streamKey`)
and `useSyncExternalStore` instead of two `useEffect`s, `useState`, and
six `useRef`s.

## What changed

### `SmoothTextEngine` gained:
- `listeners: Set<() => void>` — subscriber tracking
- `rafId` / `previousTimestamp` — moved from the hook's refs into the
engine
- `subscribe(listener)` — adds a listener, returns unsubscribe
(`useSyncExternalStore` contract)
- `dispose()` — stops loop, clears listeners
- `startLoop()` / `stopLoop()` — private RAF lifecycle
- `frame` — the RAF callback, previously a ref-stored closure in the
hook
- `notify()` — calls all listeners
- `update()` now calls `startLoop()`/`stopLoop()`/`notify()` as side
effects of state changes
- `reset()` now calls `stopLoop()`

### `useSmoothStreamingText` removed:
- `useState` (visibleLength)
- Both `useEffect`s (re-arm + lifecycle)
- 6 refs: `engineRef`, `previousStreamKeyRef`, `visibleLengthRef`,
`rafIdRef`, `previousTimestampRef`, `frameRef`
- Both `biome-ignore` suppressions

### `useSmoothStreamingText` replaced with:
- `useMemo(() => new SmoothTextEngine(), [streamKey])` — engine creation
+ reset on key change
- `useSyncExternalStore(engine.subscribe, () => engine.visibleLength)` —
reactive snapshot

Everything else (constants, `clamp`, `getAdaptiveRate`, `tick()`
internals, grapheme segmenter, `sliceAtGraphemeBoundary`, types, return
value shape) is identical.

## Testing

Existing `SmoothTextEngine` unit tests pass without modification.
@kylecarbs kylecarbs enabled auto-merge (squash) March 2, 2026 20:13
@kylecarbs kylecarbs disabled auto-merge March 2, 2026 20:13
@kylecarbs kylecarbs merged commit c4a4ad6 into main Mar 2, 2026
26 checks passed
@kylecarbs kylecarbs deleted the smooth-streaming-text branch March 2, 2026 20:18
@github-actions github-actions bot locked and limited conversation to collaborators Mar 2, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants