Repository navigation
fix(core): keep locations with running sessions or background jobs - #57416
Merged
Merged
Conversation
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
This looks correct to me. The eviction sweep now skips any location that owns a running background shell or subagent job, using the in-memory job registry. A stale running marker left after a restart therefore can't pin a location. I found no blocking problems.
I checked it by running the two new eviction tests against the base: both fail there and pass with this change. The 4 related test files (41 pass) and the packages/core type check also pass. I didn't run it end to end, because that needs the real 60-minute TTL to expire.
Public API is unchanged: Job.runningBackground is an internal core service method.
One point on scope, which doesn't block this PR: shells started directly through POST /api/shell and desktop PTY terminals don't go through Job, so their locations can still be evicted. That is the reproduction path of #51828 and the symptom in #48691, so I've listed those issues as related rather than fixed.
jlongster
changed the title
jlongster
force-pushed
the
evict-skip-background
branch
from
October 20, 2026 14:54
92f6234 to
9d562d9
Compare
This was referenced Oct 20, 2026
Closed
Contributor
yangyaofei
added a commit
to yangyaofei/opencode-buraunkan-kobo
that referenced
this pull request
Oct 20, 2026
* fix(quota-retry): 事件归属按 session 反查 + 续命标记幂等键, 修 10 次耗尽后不续命 ## 问题/错误 原生 10 次重试耗尽 → session.execution.failed → 续命应注入 synthetic 开新一轮, 但从未发生(生产 DB 0 条 msg_quota-retry-round-*, 失败会话只有 user/assistant/idle 三条)。 ## 发现过程 隔离环境(fake HOME + mock 429)复现, 给插件加事件日志后抓到: session.step.failed loc=/mnt/.../note ← 带 location session.execution.failed loc= ← 空 宿主 2.0.32 的 execution.*/usage.updated 由服务器级执行队列发布, 发布点在 location 作用域之外, 信封为空。原过滤 `evLoc !== selfDir` 把它们全部静默丢弃 → 续命/轮次清零/心跳 disarm 全部变死代码, 且无任何报错。 (生产事故: 2026-10-20 ses_ea9e725d5ffeno0tUqw9JbYGmE 429 后 90 秒 failed) ## 实现方案 1. 归属判据改为双路(core.resolveEventOwner, 纯函数+单测): - 有信封(location 级事件, 高频) → 信封是权威, 快路径不查库; - 无信封(execution.*) → 按 sessionID 反查 ctx.session.get().directory 再比; - 反查失败/不匹配必须 dbg 记录, 不允许静默 return。 2. 续命标记幂等键: assistantMessageID(拿不到退回轮次号)替代时间戳 —— 同一次失败多实例并发注入时, 第二个得到 synthetic 冲突, 视为已续过, 幂等跳过。 不用纯轮次号: 实例重建后 rounds 归零会与旧标记撞 id, 导致"以为续过"而不再唤醒。 ## 实现过程 - core.ts: +resolveEventOwner(带信封规则注释); core.test.ts: +4 用例(含回归)。 - server.ts: handleEventInner 归属双路 + [owner] 日志; step.failed 记 lastFailureMessage; resetSession 清理; redoRound 用确定性 id + 冲突幂等跳过。 ## 总结 修复"10 次耗尽后不续命"; 事件形状再变化时最坏是日志里出现 [owner] 跳过记录, 不会再静默失效。心跳/保活不在本 PR(上游 anomalyco/opencode#57416 已合并等 release)。 * debug: 归属反查失败时记录具体错误 * fix: Session.Info 无顶层 directory, 目录在 info.location.directory * fix(quota-retry): 按 PR #18 review 修三处: 幂等兜底键/残留键清理/归属反查缓存 ## 问题/错误 review 指出三处: ① 幂等兜底键 r{round} 与注释自相矛盾(纯轮次号撞旧标记→静默丢续命); ② lastFailureMessage 只在 succeeded 清, 未经 step.failed 的失败会复用上一 turn 旧 id; ③ 无信封事件逐条 session.get, 全 location 广播下是热路径。 ## 发现过程 外部 review 逐条核过 2.0.32 源码(execution.failed 无 assistantMessageID / Session.Info 无顶层 directory / Execution.Failed 发布点无 location), 判定成立。 ## 实现方案 ① 兜底键改 轮次号+时间戳: 宁可多续一轮, 不可能撞旧标记; ② execution.started 时清 lastFailure + lastFailureMessage(新执行不该继承旧键); ③ sid→dir 缓存 60s TTL(顺带兜住 session.move 陈旧性); 冲突判定 tag 优先、正则兜底。 ## 实现过程 server.ts: handleEventInner 加 dirCache; 新增 execution.started 清理分支; redoRound 兜底键与注释改写; synthetic 冲突匹配加 _tag。 ## 总结 三处均闭环; [owner] 日志保持 logFile 开关门控(默认关); PR 描述补 #8 回归溯源。 * fix(quota-retry): dirCache 只缓存成功结果 + 4096 上限(#18 review r2) 查询失败(undefined)若被缓存 60s, 期间到达的 execution.failed(一次性事件)会被 误判非本 location 而静默丢续命。改为: 失败不写缓存(下一条无信封事件自然重试), 成功结果带 4096 条粗上限防无限增长; 卸载时随插件实例闭包一起回收。
yangyaofei
added a commit
to yangyaofei/opencode-buraunkan-kobo
that referenced
this pull request
Oct 20, 2026
* fix(quota-retry): 事件归属按 session 反查 + 续命标记幂等键, 修 10 次耗尽后不续命 ## 问题/错误 原生 10 次重试耗尽 → session.execution.failed → 续命应注入 synthetic 开新一轮, 但从未发生(生产 DB 0 条 msg_quota-retry-round-*, 失败会话只有 user/assistant/idle 三条)。 ## 发现过程 隔离环境(fake HOME + mock 429)复现, 给插件加事件日志后抓到: session.step.failed loc=/mnt/.../note ← 带 location session.execution.failed loc= ← 空 宿主 2.0.32 的 execution.*/usage.updated 由服务器级执行队列发布, 发布点在 location 作用域之外, 信封为空。原过滤 `evLoc !== selfDir` 把它们全部静默丢弃 → 续命/轮次清零/心跳 disarm 全部变死代码, 且无任何报错。 (生产事故: 2026-10-20 ses_ea9e725d5ffeno0tUqw9JbYGmE 429 后 90 秒 failed) ## 实现方案 1. 归属判据改为双路(core.resolveEventOwner, 纯函数+单测): - 有信封(location 级事件, 高频) → 信封是权威, 快路径不查库; - 无信封(execution.*) → 按 sessionID 反查 ctx.session.get().directory 再比; - 反查失败/不匹配必须 dbg 记录, 不允许静默 return。 2. 续命标记幂等键: assistantMessageID(拿不到退回轮次号)替代时间戳 —— 同一次失败多实例并发注入时, 第二个得到 synthetic 冲突, 视为已续过, 幂等跳过。 不用纯轮次号: 实例重建后 rounds 归零会与旧标记撞 id, 导致"以为续过"而不再唤醒。 ## 实现过程 - core.ts: +resolveEventOwner(带信封规则注释); core.test.ts: +4 用例(含回归)。 - server.ts: handleEventInner 归属双路 + [owner] 日志; step.failed 记 lastFailureMessage; resetSession 清理; redoRound 用确定性 id + 冲突幂等跳过。 ## 总结 修复"10 次耗尽后不续命"; 事件形状再变化时最坏是日志里出现 [owner] 跳过记录, 不会再静默失效。心跳/保活不在本 PR(上游 anomalyco/opencode#57416 已合并等 release)。 * debug: 归属反查失败时记录具体错误 * fix: Session.Info 无顶层 directory, 目录在 info.location.directory * feat(quota-retry): location 保活 sweeper 对齐上游 #57416 保护集(keepalive 开关) * fix(quota-retry): 按 PR #18 review 修三处: 幂等兜底键/残留键清理/归属反查缓存 ## 问题/错误 review 指出三处: ① 幂等兜底键 r{round} 与注释自相矛盾(纯轮次号撞旧标记→静默丢续命); ② lastFailureMessage 只在 succeeded 清, 未经 step.failed 的失败会复用上一 turn 旧 id; ③ 无信封事件逐条 session.get, 全 location 广播下是热路径。 ## 发现过程 外部 review 逐条核过 2.0.32 源码(execution.failed 无 assistantMessageID / Session.Info 无顶层 directory / Execution.Failed 发布点无 location), 判定成立。 ## 实现方案 ① 兜底键改 轮次号+时间戳: 宁可多续一轮, 不可能撞旧标记; ② execution.started 时清 lastFailure + lastFailureMessage(新执行不该继承旧键); ③ sid→dir 缓存 60s TTL(顺带兜住 session.move 陈旧性); 冲突判定 tag 优先、正则兜底。 ## 实现过程 server.ts: handleEventInner 加 dirCache; 新增 execution.started 清理分支; redoRound 兜底键与注释改写; synthetic 冲突匹配加 _tag。 ## 总结 三处均闭环; [owner] 日志保持 logFile 开关门控(默认关); PR 描述补 #8 回归溯源。 * fix: 合并重复的 execution.started handler(merge 后第二个永远不可达) * fix(quota-retry): dirCache 只缓存成功结果 + 4096 上限(#18 review r2) 查询失败(undefined)若被缓存 60s, 期间到达的 execution.failed(一次性事件)会被 误判非本 location 而静默丢续命。改为: 失败不写缓存(下一条无信封事件自然重试), 成功结果带 4096 条粗上限防无限增长; 卸载时随插件实例闭包一起回收。 * feat(quota-retry): pending 条目超龄剔除 + 目标存活校验(#19 review) ## 问题/错误 review 指出: activeExecutions/runningShells 只在终态事件删除, 终态事件一旦丢失 (断连/会话被删/插件重载错位)条目永久残留 → 永久保活 + 每 45 分钟一条标记持续膨胀。 事件流推导没有宿主内存注册表的天然自清理, 必须自己兜底。 ## 实现方案 采纳建议 1 + 目标校验组合: ① 条目带时间戳, 超 MAX_PENDING_AGE_MS(12h)在 sweep 时 剔除(续命轮每轮新 execution.started 自然刷新, 不受影响); ② 每 sweep 对选中目标做 一次 session.get, 会话已删 → 剔除全部相关条目并跳过(不给死会话发标记)。 ## 实现过程 core.ts: +MAX_PENDING_AGE_MS(带取舍注释: 12h+ 真实后台 shell 失去保护, dbg 可见, 两害取其轻)。server.ts: 两个集合改带时间戳的 Map; +pruneStale/dropSession; sweep 循环先剔除超龄, 再校验目标存活。 ## 总结 陈旧条目最坏存活 12h 后自动清出; 噪音抑制建议(仅静默时发)被否: 插件侧无法区分 durable/ephemeral 事件, 用"任意事件"抑制心跳会让 60min+ 无 durable 事件的长流 (慢流场景)被误杀。
This was referenced Oct 20, 2026
Contributor
|
Superstar! This saves me so much headache, was running into this multiple times a day. Thank you James! |
12 tasks
3 tasks done
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #57411, which was auto-closed when
v2was promoted todev.Problem
LocationActivityevicts a cached Location after 60 minutes without durable Session activity. Two kinds of live work emit no durable activity while they run:inactivity, which cancelled the pending question, and then evicted the Location.execution.tscallsjobs.cancel), invalidating the graph killed background shell processes (Shell finalizer), and the completion notifications were lost.Change
When a Location's timer runs out, the sweep builds one set of protected Sessions:
SessionExecution.active,Job.runningBackground, traced asJob.runningBackground): the shell'ssessionID, or bothchildSessionIDandparentSessionIDfor a subagent.If any protected Session lives in the Location, the sweep refreshes the timer and checks again after the next TTL. Otherwise it evicts the Location as before. The sweep no longer interrupts anything.
Running background state comes from the in-memory job registry, not the durable
pendingBackgroundmarkers. After a restart, a stalerunningmarker therefore cannot pin a Location forever.Simplification
With owners no longer interrupted, the interrupt step and the post-interrupt re-check are gone. The re-check existed because the interrupt waited for slow tool cleanup (
awaitSettlement), a long window in which new work could be admitted. Now only a fewSessionStorereads separate the snapshot frominvalidate, the same shape as the earlier prior art (ff09490bcc). A Session admitted in that window still runs on the graph it borrowed, as invalidation only detaches the cache entry.Code that only eviction used is removed as well: the
awaitSettlementoption onSessionExecution.interruptandSessionRunCoordinator.interrupt, and the internalreasonoption and"inactivity"interrupt reason inexecution.ts. The"inactivity"literal stays in the publicSessionEvent.Execution.Interruptedschema and the generated client, because existing durable events may contain it.Trade-off
A hung Session (for example a question nobody answers, or a tool that never returns) now keeps its Location loaded until it is stopped or goes idle, instead of being reclaimed after 60 minutes. The same applies to a background job that never finishes.
Tests (
packages/core/test/location-activity.test.ts)interrupted == []check that always passed.bun typecheckpasses, andbun test test/location-activity.test.ts test/location-layer.test.ts test/job.test.ts test/browser-idle.test.ts test/session-execution.test.tsgives 75 pass, 2 skip, 0 fail.