Skip to content

Commit c3d7c9e

Browse files
CopilotCopilot
authored
Support wildcard cached logs files (#60702)
1 parent 329f454 commit c3d7c9e

10 files changed

Lines changed: 441 additions & 32 deletions

‎docs/adr/60702-support-wildcard-cached-logs-files.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# ADR-60702: Support Wildcard Cached Logs Files
2+
3+
**Date**: 2026-09-13
4+
**Status**: Draft
5+
**Deciders**: gh-aw maintainers
6+
7+
---
8+
9+
### Context
10+
11+
This pull request changes the `gh aw logs` cache behavior in `pkg/cli/` so callers can pass a trailing wildcard cache prefix such as `logs-*` instead of a single JSONL file. The implementation now merges multiple matching cache shards, chooses a collision-resistant output shard name, and prunes wildcard source files that contain only out-of-range dated run records when a date filter is applied. The PR also adds a `--cached-logs` CLI alias, updates user-facing help text, and extends tests around wildcard resolution, shard ordering, and pruning. The architectural question is how the logs command should represent reusable cached run data when repeated collections produce multiple partial cache files over time.
12+
13+
### Decision
14+
15+
We will treat cached logs JSONL inputs as either a single file or a trailing-wildcard shard prefix, and in wildcard mode we will load all matching `.jsonl` shards as the starting cache while writing fresh results to a new uniquely named shard. We decided to sort matching shards deterministically, let newer shards override duplicate cached run and workflow-list records, and prune fully out-of-range dated shards after collection when a date range is requested. This approach was chosen because the PR evidence shows a need to reuse accumulated cache history safely without overwriting existing shards or keeping obviously stale wildcard cache files forever.
16+
17+
### Alternatives Considered
18+
19+
#### Alternative 1: Keep a single append-only cached JSONL file
20+
21+
The logs command could continue requiring one explicit cache file and append every new record into that same file. This was considered because it is the simplest mental model and avoids wildcard resolution, duplicate handling, and shard cleanup logic. It was not chosen because the PR adds unique shard output names and wildcard loading specifically to avoid collisions and let multiple cache fragments be reused together.
22+
23+
#### Alternative 2: Support arbitrary glob patterns for cache discovery
24+
25+
Another option would be to accept any glob expression for cache inputs rather than only a trailing prefix wildcard. This was considered because it would give users more flexibility in how they organize cache files. It was not chosen because the implementation intentionally rejects non-trailing wildcard patterns, which keeps discovery rules predictable and allows the writer to derive a safe output prefix for new shard creation.
26+
27+
#### Alternative 3: Merge wildcard sources and rewrite one consolidated cache file
28+
29+
The command could read several cache shards, combine them in memory, and then rewrite a single consolidated JSONL file as the new cache state. This was considered because it would leave users with one canonical cache artifact after each run. It was not chosen because the diff explicitly preserves existing shards, writes only newly downloaded data to a fresh file, and prunes only shards proven irrelevant to the requested date range.
30+
31+
### Consequences
32+
33+
#### Positive
34+
- Users can reuse multiple cached logs shards in one invocation, which makes incremental log collection more resilient across repeated runs.
35+
- New cache output files avoid name collisions and preserve prior cache artifacts instead of overwriting them.
36+
- Date-range pruning removes wildcard shards that contain only out-of-range dated run records, reducing stale cache buildup.
37+
38+
#### Negative
39+
- Cache handling becomes more complex because the command now resolves wildcard prefixes, merges shards, sorts them, and warns on duplicate records.
40+
- Duplicate cached runs or workflow-list payloads are resolved by last-wins behavior, which can hide older conflicting data behind warning messages.
41+
- Wildcard pruning relies on record structure and timestamps, so unusual or metadata-only files are intentionally preserved and may still accumulate.
42+
43+
#### Neutral
44+
- The CLI surface grows by one alias, `--cached-logs`, while preserving `--cached-jsonl` compatibility.
45+
- The implementation extends existing JSONL cache mechanisms rather than introducing a new cache format or storage backend.
46+
- Additional tests now codify shard naming, wildcard validation, deterministic ordering, and date-range cleanup behavior.
47+
48+
---
49+
50+
*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*

‎docs/src/content/docs/setup/cli.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -510,12 +510,14 @@ cat run-ids.txt | gh aw logs --stdin --repo owner/repo # required for bare num
510510
gh aw logs --runtime gvisor # Filter to runs using a specific sandbox agent runtime
511511
```
512512

513-
**Options:** `--after-run-id`, `--artifacts`, `--before-run-id`, `--cache-before`, `--cached-jsonl`, `--count/-c`, `--end-date`, `--engine/-e`, `--evals`, `--exclude-staged`, `--filtered-integrity`, `--firewall`, `--format`, `--json/-j`, `--last`, `--no-firewall`, `--output/-o`, `--parse`, `--ref`, `--report-file`, `--repo/-r`, `--runtime`, `--safe-output`, `--start-date`, `--stdin`, `--summary-file`, `--timeout`, `--tool-graph`, `--train`
513+
**Options:** `--after-run-id`, `--artifacts`, `--before-run-id`, `--cache-before`, `--cached-jsonl`, `--cached-logs`, `--count/-c`, `--end-date`, `--engine/-e`, `--evals`, `--exclude-staged`, `--filtered-integrity`, `--firewall`, `--format`, `--json/-j`, `--last`, `--no-firewall`, `--output/-o`, `--parse`, `--ref`, `--report-file`, `--repo/-r`, `--runtime`, `--safe-output`, `--start-date`, `--stdin`, `--summary-file`, `--timeout`, `--tool-graph`, `--train`
514514

515515
`logs` defaults `--artifacts` to `usage` for faster, compact downloads. The `--last` flag is an alias for `--count/-c`.
516516
When multiple targets run concurrently, `--count` limits the combined number of workflow runs and `--timeout` limits the total wall-clock download time across all targets.
517517

518-
`--cached-jsonl` reuses compatible, schema-versioned run records and workflow-run discovery responses. It writes exactly one JSON value per line, appending every complete `gh run list` payload before downloading artifacts and available GitHub API rate-limit reports after collection. Each enriched `run` record includes job execution data, sanitized MCP tool-call metadata, and available engine, model, runtime, and component versions for downstream dashboards. Raw tool errors, arguments, responses, and artifact bodies are excluded. Discovered runs therefore remain available when a timeout or API limit interrupts processing. Records from incompatible schema versions are ignored. Use `gh aw json-schema logs-jsonl` to generate the schema for each JSON Lines item.
518+
`--cached-jsonl` and its `--cached-logs` alias reuse compatible, schema-versioned run records and workflow-run discovery responses. They write exactly one JSON value per line, appending every complete `gh run list` payload before downloading artifacts and available GitHub API rate-limit reports after collection. Each enriched `run` record includes job execution data, sanitized MCP tool-call metadata, and available engine, model, runtime, and component versions for downstream dashboards. Raw tool errors, arguments, responses, and artifact bodies are excluded. Discovered runs therefore remain available when a timeout or API limit interrupts processing. Records from incompatible schema versions are ignored. Use `gh aw json-schema logs-jsonl` to generate the schema for each JSON Lines item.
519+
520+
Pass a trailing wildcard prefix such as `--cached-logs 'logs-*'` to load all matching `logs-*.jsonl` files as the starting cache and write newly downloaded data to a unique `logs-<unix-time>-<random>.jsonl` file. With `--start-date` or `--end-date`, wildcard cache files containing exclusively dated run records outside the requested range are deleted.
519521

520522
#### `audit`
521523

‎pkg/cli/logs_cached_json.go‎

Lines changed: 247 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,14 @@ package cli
22

33
import (
44
"bytes"
5+
"crypto/rand"
6+
"encoding/hex"
57
"encoding/json"
68
"errors"
79
"fmt"
810
"os"
911
"path/filepath"
12+
"slices"
1013
"strconv"
1114
"strings"
1215
"sync"
@@ -93,23 +96,45 @@ type cachedLogsJSONLCache struct {
9396
workflowRunLists map[string]json.RawMessage
9497
}
9598

99+
type preparedCachedLogsJSONL struct {
100+
cache *cachedLogsJSONLCache
101+
writer *cachedLogsJSONLWriter
102+
sourcePaths []string
103+
wildcard bool
104+
}
105+
96106
func loadCachedLogsJSONL(path string) (*cachedLogsJSONLCache, error) {
97107
if path == "" {
98108
return nil, nil
99109
}
100-
data, err := os.ReadFile(path)
110+
cache := &cachedLogsJSONLCache{
111+
runs: make(cachedLogsRuns),
112+
workflowRunLists: make(map[string]json.RawMessage),
113+
}
114+
recordCount, err := visitCachedLogsJSONLRecords(path, func(record cachedLogsJSONLRecord, recordNumber int) error {
115+
return cache.addRecord(record, recordNumber)
116+
})
101117
if errors.Is(err, os.ErrNotExist) {
102118
fmt.Fprintln(os.Stderr, console.FormatInfoMessage("Cached logs JSONL file not found: "+path))
103119
return nil, nil
104120
}
105121
if err != nil {
106-
return nil, fmt.Errorf("failed to read cached logs JSONL: %w", err)
122+
return nil, err
107123
}
108-
lines := bytes.Split(data, []byte{'\n'})
109-
cache := &cachedLogsJSONLCache{
110-
runs: make(cachedLogsRuns, len(lines)),
111-
workflowRunLists: make(map[string]json.RawMessage),
124+
fmt.Fprintf(os.Stderr, "%s\n", console.FormatInfoMessage(fmt.Sprintf(
125+
"Found cached logs JSONL file: %s (lines=%d, runs=%d, workflow_run_lists=%d)",
126+
path, recordCount, len(cache.runs), len(cache.workflowRunLists),
127+
)))
128+
logsCacheLog.Printf("Loaded %d run records and %d workflow run lists from cached logs JSONL", len(cache.runs), len(cache.workflowRunLists))
129+
return cache, nil
130+
}
131+
132+
func visitCachedLogsJSONLRecords(path string, visit func(cachedLogsJSONLRecord, int) error) (int, error) {
133+
data, err := os.ReadFile(path)
134+
if err != nil {
135+
return 0, fmt.Errorf("failed to read cached logs JSONL: %w", err)
112136
}
137+
lines := bytes.Split(data, []byte{'\n'})
113138
recordCount := 0
114139
for index, line := range lines {
115140
if len(bytes.TrimSpace(line)) == 0 {
@@ -122,18 +147,50 @@ func loadCachedLogsJSONL(path string) (*cachedLogsJSONLCache, error) {
122147
logsCacheLog.Printf("Ignoring incomplete final cached logs JSONL record: %v", err)
123148
break
124149
}
125-
return nil, fmt.Errorf("failed to parse cached logs JSONL record %d: %w", index+1, err)
150+
return recordCount, fmt.Errorf("failed to parse cached logs JSONL record %d: %w", index+1, err)
151+
}
152+
if err := visit(record, index+1); err != nil {
153+
return recordCount, err
126154
}
127-
if err := cache.addRecord(record, index+1); err != nil {
155+
}
156+
return recordCount, nil
157+
}
158+
159+
func loadCachedLogsJSONLFiles(paths []string) (*cachedLogsJSONLCache, error) {
160+
if len(paths) == 0 {
161+
return nil, nil
162+
}
163+
merged := &cachedLogsJSONLCache{
164+
runs: make(cachedLogsRuns),
165+
workflowRunLists: make(map[string]json.RawMessage),
166+
}
167+
for _, path := range paths {
168+
cache, err := loadCachedLogsJSONL(path)
169+
if err != nil {
128170
return nil, err
129171
}
172+
if cache == nil {
173+
continue
174+
}
175+
for id, run := range cache.runs {
176+
if _, exists := merged.runs[id]; exists {
177+
warnDuplicateCachedLogsJSONLRecord(fmt.Sprintf("Duplicate cached logs JSONL run record for run %d in %s; newer cache file wins", id, path))
178+
}
179+
merged.runs[id] = run
180+
}
181+
for key, payload := range cache.workflowRunLists {
182+
if _, exists := merged.workflowRunLists[key]; exists {
183+
warnDuplicateCachedLogsJSONLRecord(fmt.Sprintf("Duplicate cached workflow runs JSONL record in %s; newer cache file wins", path))
184+
}
185+
merged.workflowRunLists[key] = append(json.RawMessage(nil), payload...)
186+
}
130187
}
131-
fmt.Fprintf(os.Stderr, "%s\n", console.FormatInfoMessage(fmt.Sprintf(
132-
"Found cached logs JSONL file: %s (lines=%d, runs=%d, workflow_run_lists=%d)",
133-
path, recordCount, len(cache.runs), len(cache.workflowRunLists),
134-
)))
135-
logsCacheLog.Printf("Loaded %d run records and %d workflow run lists from cached logs JSONL", len(cache.runs), len(cache.workflowRunLists))
136-
return cache, nil
188+
return merged, nil
189+
}
190+
191+
func warnDuplicateCachedLogsJSONLRecord(message string) {
192+
logsCacheLog.Print(message)
193+
fmt.Fprintln(os.Stderr, console.FormatWarningMessage(message))
137194
}
138195

139196
func prepareCachedLogsJSONL(opts *LogsDownloadOptions) error {
@@ -146,15 +203,188 @@ func prepareCachedLogsJSONL(opts *LogsDownloadOptions) error {
146203
if opts.cachedJSONLWriter != nil {
147204
return nil
148205
}
149-
cache, err := loadCachedLogsJSONL(opts.CachedJSONL)
206+
prepared, err := prepareCachedLogsJSONLPath(opts.CachedJSONL)
150207
if err != nil {
151208
return err
152209
}
153-
opts.cachedJSONLCache = cache
154-
opts.cachedJSONLWriter = newCachedLogsJSONLWriter(opts.CachedJSONL)
210+
opts.CachedJSONL = prepared.writer.path
211+
opts.cachedJSONLCache = prepared.cache
212+
opts.cachedJSONLWriter = prepared.writer
213+
opts.cachedJSONLSourcePaths = prepared.sourcePaths
214+
opts.cachedJSONLWildcard = prepared.wildcard
155215
return nil
156216
}
157217

218+
func prepareCachedLogsJSONLPath(path string) (preparedCachedLogsJSONL, error) {
219+
sourcePaths, writerPath, wildcard, err := resolveCachedLogsJSONLPaths(path)
220+
if err != nil {
221+
return preparedCachedLogsJSONL{}, err
222+
}
223+
cache, err := loadCachedLogsJSONLFiles(sourcePaths)
224+
if err != nil {
225+
return preparedCachedLogsJSONL{}, err
226+
}
227+
return preparedCachedLogsJSONL{
228+
cache: cache,
229+
writer: newCachedLogsJSONLWriter(writerPath),
230+
sourcePaths: sourcePaths,
231+
wildcard: wildcard,
232+
}, nil
233+
}
234+
235+
func resolveCachedLogsJSONLPaths(path string) ([]string, string, bool, error) {
236+
if path == "" {
237+
return nil, "", false, nil
238+
}
239+
if !strings.Contains(path, "*") {
240+
return []string{path}, path, false, nil
241+
}
242+
if !strings.HasSuffix(path, "*") || strings.Count(path, "*") != 1 {
243+
return nil, "", false, fmt.Errorf("cached logs wildcard must be a trailing prefix match, such as %q", "foo-bar-*")
244+
}
245+
dir := filepath.Dir(path)
246+
prefix := strings.TrimSuffix(filepath.Base(path), "*")
247+
entries, err := os.ReadDir(dir)
248+
if errors.Is(err, os.ErrNotExist) {
249+
writerPath, err := uniqueCachedLogsJSONLPath(dir, prefix)
250+
if err != nil {
251+
return nil, "", false, err
252+
}
253+
return nil, writerPath, true, nil
254+
}
255+
if err != nil {
256+
return nil, "", false, fmt.Errorf("failed to list cached logs JSONL directory: %w", err)
257+
}
258+
sourcePaths := make([]string, 0, len(entries))
259+
for _, entry := range entries {
260+
if entry.IsDir() {
261+
continue
262+
}
263+
name := entry.Name()
264+
if strings.HasPrefix(name, prefix) && strings.HasSuffix(name, ".jsonl") {
265+
sourcePaths = append(sourcePaths, filepath.Join(dir, name))
266+
}
267+
}
268+
sortCachedLogsJSONLSourcePaths(sourcePaths, prefix)
269+
writerPath, err := uniqueCachedLogsJSONLPath(dir, prefix)
270+
if err != nil {
271+
return nil, "", false, err
272+
}
273+
return sourcePaths, writerPath, true, nil
274+
}
275+
276+
func sortCachedLogsJSONLSourcePaths(paths []string, prefix string) {
277+
slices.SortStableFunc(paths, func(leftPath, rightPath string) int {
278+
left, leftErr := os.Stat(leftPath)
279+
right, rightErr := os.Stat(rightPath)
280+
if leftErr == nil && rightErr == nil && !left.ModTime().Equal(right.ModTime()) {
281+
if left.ModTime().Before(right.ModTime()) {
282+
return -1
283+
}
284+
return 1
285+
}
286+
leftUnix, leftOK := cachedLogsJSONLUnixSuffix(leftPath, prefix)
287+
rightUnix, rightOK := cachedLogsJSONLUnixSuffix(rightPath, prefix)
288+
if leftOK && rightOK && leftUnix != rightUnix {
289+
if leftUnix < rightUnix {
290+
return -1
291+
}
292+
return 1
293+
}
294+
return strings.Compare(leftPath, rightPath)
295+
})
296+
}
297+
298+
func cachedLogsJSONLUnixSuffix(path, prefix string) (int64, bool) {
299+
name := strings.TrimSuffix(filepath.Base(path), ".jsonl")
300+
suffix, ok := strings.CutPrefix(name, prefix)
301+
if !ok {
302+
return 0, false
303+
}
304+
digitCount := 0
305+
for digitCount < len(suffix) && suffix[digitCount] >= '0' && suffix[digitCount] <= '9' {
306+
digitCount++
307+
}
308+
if digitCount == 0 {
309+
return 0, false
310+
}
311+
value, err := strconv.ParseInt(suffix[:digitCount], 10, 64)
312+
return value, err == nil
313+
}
314+
315+
func uniqueCachedLogsJSONLPath(dir, prefix string) (string, error) {
316+
for range 16 {
317+
var random [8]byte
318+
if _, err := rand.Read(random[:]); err != nil {
319+
return "", fmt.Errorf("failed to generate cached logs JSONL file name: %w", err)
320+
}
321+
name := fmt.Sprintf("%s%d-%s.jsonl", prefix, time.Now().Unix(), hex.EncodeToString(random[:]))
322+
path := filepath.Join(dir, name)
323+
if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) {
324+
return path, nil
325+
} else if err != nil {
326+
return "", fmt.Errorf("failed to check cached logs JSONL file name: %w", err)
327+
}
328+
}
329+
return "", errors.New("failed to generate a unique cached logs JSONL file name")
330+
}
331+
332+
func finalizeCachedLogsJSONL(writer *cachedLogsJSONLWriter, sourcePaths []string, wildcard bool, startDate, endDate string) error {
333+
return errors.Join(
334+
writer.filterDateRange(startDate, endDate),
335+
pruneCachedLogsJSONLWildcardSources(sourcePaths, wildcard, startDate, endDate),
336+
)
337+
}
338+
339+
func pruneCachedLogsJSONLWildcardSources(sourcePaths []string, wildcard bool, startDate, endDate string) error {
340+
if !wildcard || len(sourcePaths) == 0 || (startDate == "" && endDate == "") {
341+
return nil
342+
}
343+
dateRange, err := newCachedLogsJSONLDateRange(startDate, endDate)
344+
if err != nil {
345+
return err
346+
}
347+
var result error
348+
for _, path := range sourcePaths {
349+
hasMatch, canDelete, err := cachedLogsJSONLFileDateRangeStatus(path, dateRange)
350+
if err != nil {
351+
result = errors.Join(result, err)
352+
continue
353+
}
354+
if hasMatch || !canDelete {
355+
continue
356+
}
357+
if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) {
358+
result = errors.Join(result, fmt.Errorf("failed to delete out-of-range cached logs JSONL file %s: %w", path, err))
359+
}
360+
}
361+
return result
362+
}
363+
364+
func cachedLogsJSONLFileDateRangeStatus(path string, dateRange cachedLogsJSONLDateRange) (bool, bool, error) {
365+
hasMatchingRun := false
366+
hasDatedRun := false
367+
hasPreservedRecord := false
368+
_, err := visitCachedLogsJSONLRecords(path, func(record cachedLogsJSONLRecord, _ int) error {
369+
if record.Kind != cachedLogsJSONLKindRun ||
370+
record.SchemaVersion != cachedLogsJSONLSchemaVersion ||
371+
record.Run == nil ||
372+
record.Run.CreatedAt.IsZero() {
373+
hasPreservedRecord = true
374+
return nil
375+
}
376+
hasDatedRun = true
377+
if dateRange.includes(record.Run.CreatedAt) {
378+
hasMatchingRun = true
379+
}
380+
return nil
381+
})
382+
if err != nil {
383+
return false, false, err
384+
}
385+
return hasMatchingRun, hasDatedRun && !hasPreservedRecord, nil
386+
}
387+
158388
func (cache *cachedLogsJSONLCache) addRecord(record cachedLogsJSONLRecord, recordNumber int) error {
159389
if record.SchemaVersion != cachedLogsJSONLSchemaVersion {
160390
logsCacheLog.Printf("Ignoring incompatible cached logs JSONL record: record=%d, schema_version=%d", recordNumber, record.SchemaVersion)

0 commit comments

Comments
 (0)