Skip to content

fix: pass pagination limit to log callables to prevent OOM (#806)#1

Open
JeremiahChurch wants to merge 9 commits intomasterfrom
fix/log-pagination-memory-exhaustion
Open

fix: pass pagination limit to log callables to prevent OOM (#806)#1
JeremiahChurch wants to merge 9 commits intomasterfrom
fix/log-pagination-memory-exhaustion

Conversation

@JeremiahChurch
Copy link
Copy Markdown
Owner

Summary

Fixes the PHP memory exhaustion issue described in pfrest#806.

Log endpoints (/api/v2/status/logs/*) load all log files - including every rotated/compressed copy - into memory before Model::read_all() applies pagination via array_slice(). Even ?limit=1 triggers loading the entire log history, causing OOM at 512MB on systems with moderate log volumes.

This PR passes the pagination limit from Model::read_all() down through get_internal_objects() to the log-reading callables, so they can stop reading early once enough lines are collected.

Changes

LogFileModelTraits.inc - the core fix:

  • read_log() gains an optional $limit parameter
  • When $limit > 0: reads newest files first, stops when enough lines are collected
  • Plain text files: reverse-reads from EOF via fseek() - O(limit) memory
  • Gzip files: streams line-by-line with a ring buffer - O(limit) memory
  • Bzip2/xz: full decompression (no streaming API), but truncated immediately and skipped entirely if newer files already satisfy the limit
  • gather_log_filepaths() now returns newest-first ordering with explicit sort
  • When $limit = 0 (default): preserves existing behavior exactly

Model.inc - plumbing:

  • get_internal_objects() gains optional int $limit parameter
  • Uses ReflectionMethod to check if the callable accepts a $limit param before passing it (backward-compatible with callables that don't)
  • read_all() passes $limit + $offset as the max needed hint

All 6 log models - trivial forwarding:

  • FirewallLog, SystemLog, DHCPLog, AuthLog, OpenVPNLog, RESTAPILog
  • Each callable gains int $limit = 0 and forwards it to read_log()

Performance impact

Request Before After
?limit=50 (no filter) Loads ALL files into memory Reads last 50 lines from current log via fseek; skips rotated files
?limit=50&text__contains=X Loads ALL files Same (query() path - unchanged)
No limit (unbounded) Loads all files Same (backward compat)

For ?limit=50 - the common case - this typically reads only the tail of the current log file and never touches the compressed rotated copies, reducing memory from O(total_log_size) to O(limit).

Backward compatibility

  • API response format unchanged (same {'text': ...} entries)
  • Unbounded reads follow the exact same code path
  • Callables without a $limit parameter continue to work (reflection check)
  • query() with filter parameters still does full reads (required for correctness)

Test plan

  • All 5 existing LogFileModelTraits test cases pass unchanged (unbounded reads)
  • 5 new test cases for bounded reads:
    • Bounded read returns correct newest entries across multiple files
    • Bounded read from current file only when limit < current file lines
    • Bounded read with gzip compressed rotated logs
    • Unbounded read (limit=0) returns all entries (backward compat)
    • Bounded read with limit > total lines returns everything without error
  • Manual test: GET /api/v2/status/logs/firewall?limit=50 no longer OOMs on systems with large log files
  • Manual test: GET /api/v2/status/logs/firewall (no limit) still returns all entries

Ref: pfrest#806

🤖 Generated with Claude Code

@JeremiahChurch JeremiahChurch force-pushed the fix/log-pagination-memory-exhaustion branch from f583b89 to 4c26a44 Compare March 22, 2026 23:31
jaredhendrickson13 and others added 9 commits March 22, 2026 20:00
A conflict error is a more accurate choice when someone tries to delete a CA that is currently in use.
Log endpoints load ALL log files (current + every rotated/compressed
copy) into memory before Model::read_all() applies pagination via
array_slice(). Even ?limit=1 loads the entire dataset, causing PHP
memory exhaustion (512MB) on systems with moderate log volumes.

This passes the pagination limit from Model::read_all() through
get_internal_objects() to log-reading callables so they stop early.

LogFileModelTraits changes:
- read_log() accepts optional $limit parameter
- When limit > 0, reads newest files first and stops when satisfied
- Plain files: reverse-reads from EOF via fseek - O(limit) memory
- Compressed files: streams via PHP stream wrappers
  (compress.zlib://, compress.bzip2://) and popen (xz) with a ring
  buffer - O(limit) memory for all formats
- gather_log_filepaths() uses glob($base.'.*') to avoid matching
  unrelated files, sorts newest-first by rotation number
- Unbounded reads (limit=0) preserve existing behavior exactly

Model.inc changes:
- get_internal_objects() accepts optional $limit hint
- Forwards limit to callables via cached reflection check
  (backward-compatible with callables that don't accept it)
- read_all() passes limit+offset down; skips optimization when
  reverse=true since that needs the full dataset

All 6 log model callables updated to accept and forward limit.
10 new test cases covering bounded reads, edge cases, mixed
compression, and unrelated file filtering.

Tested on live pfSense (FreeBSD 15.0, 28MB filter.log + 31 rotated
bz2 files): memory dropped from 512MB+ (OOM) to 22MB, response
time 1ms for ?limit=10.

Fixes pfrest#806
@jaredhendrickson13 jaredhendrickson13 force-pushed the fix/log-pagination-memory-exhaustion branch from 4c26a44 to ec5f91b Compare March 28, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants