Repository navigation
conhost leaks a reference to the client's primary token on NotifyConsoleApplication when an MSYS2/Cygwin process exec()s: one kernel Token object per exec, until reboot #20781
Description
Windows Terminal version
1.24.11911.0 is installed, but the leak was traced in the inbox conhost.exe (10.0.26100.1, C:\Windows\System32). I have not checked whether OpenConsole.exe behaves the same way.
Windows build number
10.0.26200.9457 (Windows 11 Home 25H2, x64). win32kfull.sys is 10.0.26100.9444.
Other Software
Git for Windows 2.55.0.windows.3 (MSYS2 runtime 3.6.9, a Cygwin fork). Any Cygwin/MSYS2 exec() triggers it. AI coding agents (Claude Code, Codex) run every shell command through Git Bash, so a developer machine hits this path hundreds of thousands of times.
Steps to reproduce
The script below needs only Git for Windows and no admin rights. It counts kernel Token objects (NtQueryObject(ObjectTypesInformation)) before and after a Git Bash loop that fork+execs 300 times. A cmd.exe loop is the control.
# powershell -NoProfile -ExecutionPolicy Bypass -File repro.ps1
param([string]$Bash = "$env:ProgramFiles\Git\usr\bin\bash.exe", [int]$N = 300)
Add-Type @'
using System; using System.Runtime.InteropServices;
public static class ObjCount {
[DllImport("ntdll.dll")] static extern int NtQueryObject(IntPtr h, int cls, IntPtr buf, int len, out int ret);
public static long Of(string typeName) {
int len = 1 << 18, ret; IntPtr b = Marshal.AllocHGlobal(len);
try {
if (NtQueryObject(IntPtr.Zero, 3, b, len, out ret) != 0) return -1;
int n = Marshal.ReadInt32(b, 0); long off = IntPtr.Size;
for (int i = 0; i < n; i++) {
IntPtr e = new IntPtr(b.ToInt64() + off);
short nameLen = Marshal.ReadInt16(e, 0), maxLen = Marshal.ReadInt16(e, 2);
string name = Marshal.PtrToStringUni(Marshal.ReadIntPtr(e, IntPtr.Size), nameLen / 2);
if (name == typeName) return (uint)Marshal.ReadInt32(e, IntPtr.Size * 2);
off += 0x68 + ((maxLen + IntPtr.Size - 1) & ~(IntPtr.Size - 1));
}
return -1;
} finally { Marshal.FreeHGlobal(b); }
}
}
'@
function Run([string]$exe, [string]$argLine) { $psi = New-Object Diagnostics.ProcessStartInfo $exe, $argLine; $psi.UseShellExecute = $false; [Diagnostics.Process]::Start($psi).WaitForExit() }
function Measure-Case([string]$label, [scriptblock]$body) {
Start-Sleep 2; $before = [ObjCount]::Of('Token'); & $body; Start-Sleep 5; $after = [ObjCount]::Of('Token')
'{0,-50} Token objects {1:N0} -> {2:N0} ({3:+#;-#;0})' -f $label, $before, $after, ($after - $before) }
Measure-Case "control: cmd.exe runs $N x where.exe" { Run "$env:SystemRoot\System32\cmd.exe" "/c for /l %i in (1,1,$N) do @`"%SystemRoot%\System32\where.exe`" /q cmd" }
Measure-Case "bash runs $N x /usr/bin/true (fork + exec)" { Run $Bash "-c 'for i in `$(/usr/bin/seq $N); do /usr/bin/true; done'" }
Measure-Case "bash runs $N x ( : ) (fork only, no exec)" { Run $Bash "-c 'for i in `$(/usr/bin/seq $N); do ( : ); done'" }Expected Behavior
Token objects return to baseline once the processes exit, as they do in the cmd.exe control.
Actual Behavior
Each exec() from an MSYS2 process leaves one Token object behind. The leaked objects never go away; only a reboot frees them.
| 300 iterations | Token objects |
|---|---|
control: cmd.exe → where.exe |
−7 to +2 |
Git Bash fork + exec (/usr/bin/true) |
+238 to +345 (7 of 8 runs; one run showed −2) |
exec only (bash -c 'exec /usr/bin/true', started from PowerShell) |
+54 to +354 |
| fork only (subshell, no exec) | 0 to +29 |
After 22 days of uptime this machine had ~660,000 Token objects but only ~2,650 token handles system-wide, which is ~1.3 GB of paged pool (Toke tag). The leaked tokens also keep their logon sessions alive: 277 stale sessions of a sandbox account that runs commands through MSYS. Plain CreateProcess in every variant I tried (suspended, inherited handles, DETACHED_PROCESS, new process group) does not leak, and neither do console apps that load user32 (powershell.exe, tasklist.exe).
Where the reference comes from. I recorded an xperf trace (OB_OBJECT + OB_HANDLE, stackwalk ObjectReference/ObjectDeReference) of 100 fork+exec iterations and followed every Token object created by bash.exe from create to delete. The 101 that were never deleted are the primary tokens of the forked bash.exe that execs. Grouping their references by process context:
| References taken in | 101 leaked tokens | 101 freed process tokens |
|---|---|---|
conhost.exe |
net +101: one per token, never released | net 0 |
| csrss.exe, MsMpEng.exe, others | net 0 | net 0 |
The unreleased reference has this stack (one representative; the leaked token's process has already been deleted, and ObjectDelete for the process object fired):
ObjectReference conhost.exe Object=<token> Tag=Dflt Count=1
nt!PsReferencePrimaryTokenWithTag
nt!PsReferencePrimaryToken
win32kfull.sys!0xfffff800365c53ed
win32kfull.sys!0xfffff80036428cf1
win32kfull.sys!0xfffff80036428af3
win32kfull.sys!0xfffff8003642861c
win32kfull.sys!0xfffff800365d9690
win32kfull.sys!0xfffff800365d9149
win32k.sys!0xfffff8003cbb9253
nt!KiSystemServiceCopyEnd
win32u.dll!0x00007ffbb7b42c44 (presumably NtUserConsoleControl; no symbols for win32u)
user32.dll!0x00007ffbb8431e1a
conhost.exe!Microsoft::Console::Interactivity::Win32::ConsoleControl::NotifyConsoleApplication
conhost.exe!IoDispatchers::ConsoleHandleConnectionRequest
conhost.exe!ConsoleIoThread
For the program that the fork then execs (true.exe), the same conhost-context reference is released. My reading, without win32kfull symbols: the ConsoleNotifyConsoleApplication registration made when the exec-ing MSYS process connects is never undone. Under Cygwin's exec the process hands its console over to the new program and exits, so the matching cleanup never runs for it. I can share the .etl (2.8 GB) and the reduced dumps.
Activity
nonacc commentedon Oct 8, 2026
Cross-reference: also reported to Git for Windows (the MSYS2 runtime's exec triggers it): git-for-windows/git#6471
handoffandhope commentedon Oct 9, 2026
Independent repro on a second machine, with a kernel reference trace that pins the leaking call. It also shows the leak is not specific to MSYS2.
Build: Windows 11 26300.9550 (release channel, not Insider), win32kfull.sys 10.0.26100.9549, conhost.exe 10.0.26100.1.
Impact: after 8 days of uptime with dev tools running (git/gh/shell polling): about 1.8M Token objects held against about 1.7k open Token handles. Paged pool was 10.1 GB and nonpaged pool 4.8 GB on a 32 GB machine. At normal load it grows about 14k tokens/hour. A cheap way to watch it without admin:
NtQueryObject(NULL, ObjectTypesInformation), then compare TokenTotalNumberOfObjectswithTotalNumberOfHandles.Reference trace: WPR with the kernel
ObjectandHandlekeywords, stacks onObjectReferenceandObjectDeReference. Each token created during the trace was netted, reference by reference, per tag and per stack:Dflt.Not MSYS2-only. Chains that leaked in the trace:
cmd.exe → tasklist.exepwsh.exepython.exe → git.exe(thecmd\git.exewrapper)→ mingw64\bin\git.exegit → git-remote-https → sh → ghLaunch-style matrix: a Python worker ran
git --versionN times. These numbers come from one machine with background load, so treat them as indicative:CREATE_NO_WINDOW)DETACHED_PROCESS), child's console created through the default-terminal handoff to Windows TerminalRuled out on this machine with A/B runs: the GPU driver (integrated GPU disabled), credential-manager paths, zombie or orphaned parents, and stdout/stderr redirection style.
The pattern looks like a token reference taken in the
ConsoleControlpath with no matching dereference on some path. Given that two independent reports appeared in the same week, it may be a recent regression in win32kfull.Investigated and written up with help from Claude Opus 5.5. The traces, counts and test runs are real measurements from my machine; happy to share the analysis scripts or rerun anything.
Asromer-biology commentedon Oct 10, 2026
Same leak here, on a third machine.
Build: Windows 11 Home 10.0.26200.9457, uptime 10 days (booted 2026-09-30).
Pool tags after 10 days (read with NtQuerySystemInformation / SystemPoolTagInformation):
Toke: 2,687,877 live allocations, 5.59 GB pagedSeAt: 0.98 GB paged,SeTd: 0.37 GB paged,SeTl: 0.33 GB nonpaged (same 2,687,877 live count)Total handle count system-wide is about 285k, so the tokens are not held by handles. That matches the unreleased kernel reference described above.
Workload: several Claude Code agent sessions running shell commands through Git Bash and PowerShell, plus Docker Desktop/WSL2. The live
Tokecount was flat over a 60 s idle sample, so growth tracks process spawning.Impact: available memory fell to about 7.7 GB of 31.6 GB. That blocked an 8 GB container workload until apps were closed by hand, and the only way to reclaim it is a reboot. For anyone running agent tooling on 25H2, this effectively forces a scheduled reboot every few days. Please triage.
DHowett commentedon Oct 12, 2026
OKAY THANK YOU SEVEN ROBOTS IN A TRENCH COAT
No more reports of this issue are required.
lhecker commentedon Oct 16, 2026
Before I unlock the conversation, I'd like to remind the bots again:
We allow AI bug reports. We DO NOT allow AI-authored comments. The above is a small example why that is.
Avi Vaulin (@nonacc) I come bearing good and bad news. The bad news is that I cannot repro it:
bash.exetrue.exewhere.execmd.exeThe good news is that there's some difference between our two systems:
You're on build 26200.9457 and I'm on 26300.9711. Do you mind updating Windows?
Edit: And otherwise, I think we may want to close it anyway? cygwingitgadget/cygwin#15 looks like the right solution.
lhecker commentedon Oct 21, 2026
The missing ingredient here was that you need to switch focus to another application while the repro is running. The leak occurs inside the foreground-stealing permission logic of win32kfull.sys.
lhecker commentedon Oct 21, 2026
Coincidentally this was recently reported internally as MSFT-64668009.