Every exec() from Git Bash leaks one kernel Token object (paged pool) until reboot: ~660k tokens / 1.3 GB after 3 weeks of agent use #6471

Activity

1. dscho commented on Oct 20, 2026

   

   dscho

   on Oct 20, 2026

   Member

   More actions

   I have a tentative fix in hand: cygwingitgadget/cygwin#15. If you have the time, please feel free to build an MSYS2 runtime with that patch and to test whether this fixes your issue.
2. dscho commented on Oct 20, 2026

   

   dscho

   on Oct 20, 2026

   Member

   More actions

   @nonacc actually, I forgot that I had already ran a tentative build of the MSYS2 runtime with this fix: https://github.com/dscho/Cygwin-msys2-fork/actions/runs/38449886036 has an install artifact that contains a fixed msys-2.0.dll. Could you please (temporarily) exchange the copy of that DLL in your setup with that fixed version and verify that the token leak is gone?
3. dscho

   added 2 commits that reference this issue on Oct 20, 2026

   Cygwin: testsuite: Add regression test for token leak on exec

   ...

   52260dd

   Cygwin: spawn: detach console on exec to avoid leaking token

   ...

   bee392f
4. martindroneharmony

   marked Each process spawned from Git Bash leaks a kernel token object (paged pool Toke) on Windows 11 26H2 #6499 as a duplicate of this issue on Oct 20, 2026
5. martindroneharmony commented on Oct 20, 2026

   

   martindroneharmony

   on Oct 20, 2026

   More actions

   Tested the fixed msys-2.0.dll from https://github.com/dscho/Cygwin-msys2-fork/actions/runs/38449886036 (3.6.11-426a8d2290bd668c0de8b57b3dfea51fdc4b5fc3) on Windows 11 26H2, build 10.0.26300, in a copy of Git for Windows 2.55.0.windows.5 (verified that bash.exe loads the fixed DLL). Unfortunately it does not stop the leak here.

   Method: kernel Token object count via NtQueryObject(ObjectTypesInformation), 1000 runs of /usr/bin/true per measurement, each run in its own console window, 3 rounds alternating:

   • runtime 3.6.10 (installed): +1003, +1144, +496
   • runtime 3.6.11 with the fix: +1273, +1096, +1603
   • control, cmd.exe starting cmd /c exit 1000 times: +11, -12, +207
   • idle 60 s (background noise from other processes): -43, +650, +227

   Note: without a console attached (bash started from a process that has none), neither runtime leaks, which matches the conhost root cause described above.

   Happy to test another build.
6. dscho commented on Oct 21, 2026

   

   dscho

   on Oct 21, 2026

   Member

   More actions

   @martindroneharmony ah, I missed something important. You ran /usr/bin/true 1,000 times directly in separate console windows. That does not exercise exec() (_P_OVERLAY), so our fix in spawn.cc never executed.

   Even more importantly, with the help of GPT-6.1 Sol I found https://learn.microsoft.com/en-gb/answers/questions/5952454/windows-11-25h2-26200-8655-unbounded-paged-pool-le, which seems to talk precisely about the same problem. And it suggests a work-around: Setting SPI_SETFOREGROUNDLOCKTIMEOUT to 0 via SystemParametersInfoW(), to bypass _CheckAllowForeground () and thereby stopping the leak. Here is a PowerShell command to do that, and according to my research you'd only need to run this once per logon session:

   Add-Type @'
   using System; using System.Runtime.InteropServices;
   public static class SPIHelper {
     [DllImport("user32.dll", SetLastError = true)]
     public static extern bool SystemParametersInfoW(uint uiAction, uint uiParam, IntPtr pvParam, uint fWinIni);
     public static uint GetFLT() {
       uint flt = 0;
       IntPtr p = Marshal.AllocHGlobal(4);
       try {
         SystemParametersInfoW(0x2000 /* SPI_GETFOREGROUNDLOCKTIMEOUT */, 0, p, 0);
         flt = (uint)Marshal.ReadInt32(p);
       } finally { Marshal.FreeHGlobal(p); }
       return flt;
     }
     public static bool SetFLT(uint val) {
       return SystemParametersInfoW(0x2001 /* SPI_SETFOREGROUNDLOCKTIMEOUT */, 0, (IntPtr)val, 2 /* SPIF_SENDCHANGE */);
     }
   }
   '@ ; $res = [SPIHelper]::SetFLT(0) ; 'SetFLT result: {0}; new FLT: {1}' -f $res, [SPIHelper]::GetFLT()

   Note that setting ForegroundLockTimeout to 0 disables Windows' focus-stealing prevention, so this is merely a work-around:

   1. Focus Stealing: Any background application or spawned process can immediately pop to the front and steal keyboard focus while you are typing in another window (rather than flashing on the taskbar).
   2. Misdirected Keystrokes: Typing into an active window (editor, terminal) might accidentally send keystrokes or enter keys into a background window that suddenly takes the foreground.

   For headless machines, CI runners, or dedicated developer/agent VMs, this side effect is harmless; on interactive desktop machines, unexpected pop-ups may interrupt typing.

   @martindroneharmony could you try this work-around and report whether it fixes the token leak for you?
7. martindroneharmony commented on Oct 21, 2026

   

   martindroneharmony

   on Oct 21, 2026

   Last edited by martindroneharmony

   More actions

   @dscho thank you, that was it.

   First a clarification: my earlier wording was misleading. Each measurement was one console window running a bash loop of 1,000 /usr/bin/true starts (for i in $(seq 1000); do /usr/bin/true; done), not 1,000 separate windows. So every start was a fork() followed by exec() inside bash.

   The work-around stops the leak here (Windows 11 26H2, build 10.0.26300, Git for Windows 2.55.0.windows.5). Kernel Token objects per 1,000 starts, 3 rounds each:

   ForegroundLockTimeout at its value before (2147483647):

   • runtime 3.6.10 (installed): +1390, +1138, +955
   • runtime 3.6.11 with your fix: +80, +1064, +1146
   • control, cmd.exe: -11, +98, +63
   • idle 60 s: -12, +379, +149

   ForegroundLockTimeout set to 0 with your snippet (run from a foreground window; from a background process the call has no effect):

   • runtime 3.6.10 (installed): -83, +44, +55 and +10, -34, +28
   • runtime 3.6.11 with your fix: -8, +48, -49
   • control, cmd.exe: -7, +4, +55
   • idle 60 s: -50, +18, +51

   Everything is within the idle noise with the timeout at 0. The bash loops also ran faster (about 50 s instead of 85 s per 1,000 starts).

   One thing that may matter: the live value on this machine was 2147483647, while HKCU\Control Panel\Desktop\ForegroundLockTimeout holds the default 200000, so some program raised it at runtime. I don't know yet which one.

   Happy to test more.

   Update: the reason the value is 2147483647. Others report that on Windows 11 the live value is overwritten at every logon by win32kfull!LoadCPUserPreferences, which reads the registry and then stores 0x7FFFFFFF unconditionally, so the registry setting (200000 here, 0 for them) is ignored: bentoner/windows-token-leak#1 (26200.9550 and 26300.9550, with disassembly; the leak itself is traced there to a PsReferencePrimaryToken without a matching dereference in CForegroundLaunch::_CheckAllowForeground). That matches this machine: registry 200000, live value 2147483647.

   So the work-around has to be re-applied after every logon, by a process that may take the foreground (from a background process the call is refused while the value is 2147483647). A non-elevated scheduled task at logon works here: it set the value from 2147483647 to 0 in a test run. Leaked tokens are only freed by a restart (with Fast Startup, "Shut down" keeps them).