Skip to content

Latest commit

 

History

History
604 lines (468 loc) · 17 KB

VideoFrame.mdx

File metadata and controls

604 lines (468 loc) · 17 KB
title VideoFrame
description Represents a single video frame with pixel data and metadata
icon image

VideoFrame

The VideoFrame class represents a single frame of video, containing raw pixel data along with timing and layout metadata. VideoFrames can be created from raw pixel buffers or by cloning existing frames.

This class follows the [W3C WebCodecs VideoFrame specification](https://www.w3.org/TR/webcodecs/#videoframe-interface). VideoFrames hold significant memory resources. You **must** call `close()` when done with a frame to release its resources. Failing to do so will cause memory leaks.

Quick Example

import { VideoFrame } from 'node-webcodecs';

// Create a frame from raw I420 pixel data
const width = 1920;
const height = 1080;
const i420Data = new Uint8Array(width * height * 1.5);  // I420 is 12 bits per pixel

const frame = new VideoFrame(i420Data, {
  format: 'I420',
  codedWidth: width,
  codedHeight: height,
  timestamp: 0,           // Presentation time in microseconds
  duration: 33333         // Frame duration (30fps = ~33333us)
});

// Access frame properties
console.log(frame.format);        // 'I420'
console.log(frame.codedWidth);    // 1920
console.log(frame.codedHeight);   // 1080
console.log(frame.timestamp);     // 0

// Copy pixel data to a buffer
const buffer = new Uint8Array(frame.allocationSize());
await frame.copyTo(buffer);

// Always close when done!
frame.close();

Constructors

VideoFrame has two constructor overloads: one for creating frames from raw pixel buffers, and one for creating frames from existing VideoFrames.

From Buffer

Creates a new VideoFrame from raw pixel data.

new VideoFrame(data: BufferSource, init: VideoFrameBufferInit)
The raw pixel data as an `ArrayBuffer` or `TypedArray` (e.g., `Uint8Array`). The data layout must match the specified `format`. Configuration object for the video frame. The pixel format of the data. See [Pixel Formats](#pixel-formats) below.
<ParamField body="codedWidth" type="number" required>
  Width of the frame in pixels, including any padding.
</ParamField>

<ParamField body="codedHeight" type="number" required>
  Height of the frame in pixels, including any padding.
</ParamField>

<ParamField body="timestamp" type="number" required>
  Presentation timestamp in **microseconds**. Determines when the frame should be displayed.
</ParamField>

<ParamField body="duration" type="number">
  Duration of the frame in **microseconds**. For 30fps video, this is typically `33333` (1/30 second).
</ParamField>

<ParamField body="displayWidth" type="number">
  Display width in pixels. Defaults to `codedWidth`. Use for non-square pixel aspect ratios.
</ParamField>

<ParamField body="displayHeight" type="number">
  Display height in pixels. Defaults to `codedHeight`. Use for non-square pixel aspect ratios.
</ParamField>

<ParamField body="visibleRect" type="object">
  The visible rectangle within the coded frame. Useful for cropping.
  <Expandable title="visibleRect properties">
    <ParamField body="x" type="number" required>
      X offset of the visible region.
    </ParamField>
    <ParamField body="y" type="number" required>
      Y offset of the visible region.
    </ParamField>
    <ParamField body="width" type="number" required>
      Width of the visible region.
    </ParamField>
    <ParamField body="height" type="number" required>
      Height of the visible region.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="colorSpace" type="VideoColorSpaceInit">
  Color space information for the frame. See [VideoColorSpace](/api-reference/VideoColorSpace).
</ParamField>

From Existing Frame

Creates a new VideoFrame by cloning an existing frame with optional modifications.

new VideoFrame(image: VideoFrame, init?: VideoFrameInit)
An existing `VideoFrame` to copy. The source frame's data and metadata are copied. Optional configuration to override properties from the source frame. Presentation timestamp in **microseconds**.
<ParamField body="duration" type="number">
  Duration of the frame in **microseconds**.
</ParamField>

<ParamField body="format" type="VideoPixelFormat">
  Override the pixel format.
</ParamField>

<ParamField body="codedWidth" type="number">
  Override the coded width.
</ParamField>

<ParamField body="codedHeight" type="number">
  Override the coded height.
</ParamField>

<ParamField body="displayWidth" type="number">
  Override the display width.
</ParamField>

<ParamField body="displayHeight" type="number">
  Override the display height.
</ParamField>

<ParamField body="visibleRect" type="object">
  Override the visible rectangle.
  <Expandable title="visibleRect properties">
    <ParamField body="x" type="number" required>
      X offset of the visible region.
    </ParamField>
    <ParamField body="y" type="number" required>
      Y offset of the visible region.
    </ParamField>
    <ParamField body="width" type="number" required>
      Width of the visible region.
    </ParamField>
    <ParamField body="height" type="number" required>
      Height of the visible region.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField body="colorSpace" type="VideoColorSpaceInit">
  Override the color space information.
</ParamField>

Properties

All properties are readonly after construction.

The pixel format of the frame data (e.g., `'I420'`, `'NV12'`, `'RGBA'`). See [Pixel Formats](#pixel-formats) for all supported formats. Width of the frame in pixels, including any codec-required padding. May be larger than `displayWidth`. Height of the frame in pixels, including any codec-required padding. May be larger than `displayHeight`. Width the frame should be displayed at, accounting for pixel aspect ratio. Use this for rendering. Height the frame should be displayed at, accounting for pixel aspect ratio. Use this for rendering. Presentation timestamp in microseconds. Determines when this frame should be displayed relative to the start of the video. Duration of this frame in microseconds, or `null` if not specified during construction. Color space information for the frame, including primaries, transfer characteristics, and matrix coefficients. See [VideoColorSpace](/api-reference/VideoColorSpace). The visible portion of the frame. This defines the crop region that should be displayed. Contains `x`, `y`, `width`, and `height` properties.

Methods

allocationSize()

Calculates the number of bytes needed to hold the frame's pixel data.

frame.allocationSize(options?: VideoFrameCopyToOptions): number
Optional configuration for the allocation calculation. Copy only a specific rectangular region. X offset Y offset Width Height
<ParamField body="layout" type="PlaneLayout[]">
  Custom layout for the output planes.
</ParamField>

<ParamField body="format" type="VideoPixelFormat">
  Convert to a different pixel format when copying.
</ParamField>

Returns: number - The size in bytes needed for the buffer.

```typescript const frame = new VideoFrame(data, { format: 'I420', codedWidth: 1920, codedHeight: 1080, timestamp: 0 });

// Full frame size const fullSize = frame.allocationSize(); console.log(Full frame: ${fullSize} bytes); // ~3,110,400 bytes for I420

// Size for a cropped region const cropSize = frame.allocationSize({ rect: { x: 0, y: 0, width: 640, height: 480 } }); console.log(Cropped: ${cropSize} bytes);

frame.close();

</Accordion>

---

### copyTo()

Copies the frame's pixel data to a destination buffer.

```typescript
frame.copyTo(destination: BufferSource, options?: VideoFrameCopyToOptions): Promise<PlaneLayout[]>
An `ArrayBuffer` or `TypedArray` to copy the data into. Must have at least `allocationSize()` bytes available. Optional configuration for the copy operation. See `allocationSize()` for options.

Returns: Promise<PlaneLayout[]> - Layout information for each plane in the output buffer.

```typescript const frame = new VideoFrame(data, { format: 'I420', codedWidth: 1920, codedHeight: 1080, timestamp: 0 });

// Allocate buffer with exact size needed const buffer = new Uint8Array(frame.allocationSize()); const layout = await frame.copyTo(buffer);

// Layout describes where each plane is in the buffer console.log('Y plane:', layout[0]); // { offset: 0, stride: 1920 } console.log('U plane:', layout[1]); // { offset: 2073600, stride: 960 } console.log('V plane:', layout[2]); // { offset: 2592000, stride: 960 }

frame.close();

</Accordion>

---

### clone()

Creates an independent copy of this frame.

```typescript
frame.clone(): VideoFrame

Returns: VideoFrame - A new VideoFrame with the same data and metadata.

Both the original and cloned frames must be closed independently. Cloning does not share resources. ```typescript const original = new VideoFrame(data, { format: 'RGBA', codedWidth: 1920, codedHeight: 1080, timestamp: 0 });

// Create an independent copy const cloned = original.clone();

// Close original - cloned frame is still valid original.close();

// Use the cloned frame console.log(cloned.codedWidth); // 1920

// Don't forget to close the clone too! cloned.close();

</Accordion>

---

### close()

Releases the resources held by this frame. After calling `close()`, the frame becomes unusable.

```typescript
frame.close(): void
Always call `close()` when you're done with a VideoFrame. Failure to do so will cause memory leaks, as video frames can hold significant amounts of pixel data. ```typescript const frame = new VideoFrame(data, { format: 'I420', codedWidth: 1920, codedHeight: 1080, timestamp: 0 });

try { // Use the frame await processFrame(frame); } finally { // Always close in a finally block frame.close(); }

</Accordion>

---

## Pixel Formats

VideoFrame supports the following pixel formats:

| Format | Description |
|--------|-------------|
| `I420` | YUV 4:2:0 planar (most common) |
| `I420A` | I420 with alpha plane |
| `I422` | YUV 4:2:2 planar |
| `I444` | YUV 4:4:4 planar |
| `NV12` | YUV 4:2:0 semi-planar |
| `RGBA` | 32-bit RGBA interleaved |
| `RGBX` | 32-bit RGB (alpha ignored) |
| `BGRA` | 32-bit BGRA interleaved |
| `BGRX` | 32-bit BGR (alpha ignored) |

```typescript
type VideoPixelFormat =
  | 'I420'   // Y, U, V planes (most efficient for encoding)
  | 'I420A'  // Y, U, V, A planes
  | 'I422'   // Y, U, V planes (higher chroma resolution)
  | 'I444'   // Y, U, V planes (full chroma resolution)
  | 'NV12'   // Y plane, interleaved UV plane
  | 'RGBA'   // Red, Green, Blue, Alpha interleaved
  | 'RGBX'   // Red, Green, Blue, padding interleaved
  | 'BGRA'   // Blue, Green, Red, Alpha interleaved
  | 'BGRX';  // Blue, Green, Red, padding interleaved

Type Definitions

PlaneLayout

Describes the layout of a single plane in a pixel buffer.

interface PlaneLayout {
  offset: number;  // Byte offset of the plane in the buffer
  stride: number;  // Number of bytes per row
}

VideoFrameInit

Configuration for creating a VideoFrame from an existing frame.

interface VideoFrameInit {
  timestamp: number;                    // Required: microseconds
  duration?: number;                    // Optional: microseconds
  format?: VideoPixelFormat;            // Optional: pixel format
  codedWidth?: number;                  // Optional: coded width
  codedHeight?: number;                 // Optional: coded height
  displayWidth?: number;                // Optional: display width
  displayHeight?: number;               // Optional: display height
  visibleRect?: {                       // Optional: visible region
    x: number;
    y: number;
    width: number;
    height: number;
  };
  colorSpace?: VideoColorSpaceInit;     // Optional: color space
}

VideoFrameBufferInit

Configuration for creating a VideoFrame from raw pixel data.

interface VideoFrameBufferInit extends VideoFrameInit {
  format: VideoPixelFormat;   // Required: pixel format
  codedWidth: number;         // Required: coded width
  codedHeight: number;        // Required: coded height
}

VideoFrameCopyToOptions

Options for allocationSize() and copyTo() methods.

interface VideoFrameCopyToOptions {
  rect?: {                    // Optional: copy only this region
    x: number;
    y: number;
    width: number;
    height: number;
  };
  layout?: PlaneLayout[];     // Optional: custom output layout
  format?: VideoPixelFormat;  // Optional: convert to this format
}

Usage with VideoEncoder

VideoFrames are typically passed to VideoEncoder.encode() for compression:

import { VideoEncoder, VideoFrame } from 'node-webcodecs';

const encoder = new VideoEncoder({
  output: (chunk, metadata) => {
    console.log(`Encoded ${chunk.type} frame: ${chunk.byteLength} bytes`);
  },
  error: (e) => console.error('Encoding error:', e)
});

encoder.configure({
  codec: 'avc1.42001f',  // H.264 Baseline
  width: 1920,
  height: 1080,
  bitrate: 5_000_000
});

// Create and encode frames
for (let i = 0; i < 60; i++) {
  const frame = new VideoFrame(pixelData, {
    format: 'I420',
    codedWidth: 1920,
    codedHeight: 1080,
    timestamp: i * 33333  // 30fps
  });

  encoder.encode(frame);
  frame.close();  // Close after encoding
}

await encoder.flush();

Usage with VideoDecoder

VideoFrames are produced by VideoDecoder when decoding encoded chunks:

import { VideoDecoder, EncodedVideoChunk } from 'node-webcodecs';

const decoder = new VideoDecoder({
  output: (frame) => {
    console.log(`Decoded: ${frame.codedWidth}x${frame.codedHeight} ${frame.format}`);

    // Copy pixel data for processing
    const buffer = new Uint8Array(frame.allocationSize());
    await frame.copyTo(buffer);

    // Always close the frame when done!
    frame.close();
  },
  error: (e) => console.error('Decoding error:', e)
});

decoder.configure({
  codec: 'avc1.42001f',
  codedWidth: 1920,
  codedHeight: 1080
});

decoder.decode(encodedChunk);
await decoder.flush();

See Also

Encodes VideoFrames into compressed video chunks Decodes compressed video chunks into VideoFrames Color space metadata for video frames Represents compressed video data