On Windows, Remove-Item fails intermittently with -Recurse and is unexpectedly asynchronous itself

^{Note: cmd.exe's rd /s and .NET's [System.IO.Directory]::Delete() are equally affected: see here and here.}

Note: The problem occurs only on Windows (and there also on Windows PowerShell).

Update:

Starting with Windows 10 version 1909, (at least) build 18363.657 (I don't know that Windows Server version and build that corresponds to; run winver.exe to check your version and build), the DeleteFile Windows API function now exhibits synchronous behavior, which implicitly solves the problems with PowerShell's Remove-Item and .NET's System.IO.File.Delete / System.IO.Directory.Delete (but, curiously, not with cmd.exe's rd /s).

Thus, only older Windows 10 versions and older Windows versions in general are affected.

The Windows API functions DeleteFile() and RemoveDirectory() functions are inherently asynchronous (emphasis added):

The DeleteFile function marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed. Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED."

The RemoveDirectory function marks a directory for deletion on close. Therefore, the directory is not removed until the last handle to the directory is closed."

PowerShell fails to account for this asynchronous behavior, which has two implications:

Problem (a): Trying to delete a nonempty directory (which invariably requires recursive deletion of its content first) may fail - infrequently, but it does happen.
Problem (b): Trying to recreate a successfully deleted directory or a file immediately afterwards may fail - intermittently (easier to provoke than the other problem, but a more exotic use case overall).
- Note: You could argue that it's not PowerShell's responsibility to create a synchronous experience in this case, given that the underlying system API is asynchronous; however, it certainly makes for more robust, predictable scripting if synchronicity is ensured.

As an aside: cmd.exe's rd /s exhibits the same problem, as does .NET Core's [System.IO.Directory]::Delete() (which PowerShell does not use).

Problem (a) is due to using depth-first recursion without accounting for the asynchronous deletion behavior; the problem, along with the solution, is described in this YouTube video (starts at 7:35).

Important:

The repros below use the functions defined at the bottom, which must be defined first.
The tests created temp. dir. $HOME/tmpdir - remove it manually afterwards, if still present.

Steps to reproduce

Setup: Paste and submit the function definitions below at the prompt.

Problem (a):

Assert-ReliableDirRemoval ps

Problem (b):

Assert-SyncDirRemoval ps

Expected behavior

The functions should loop indefinitely (terminate them with Ctrl+C), emitting a . in each iteration.

Actual behavior

Eventually - and that there is no predictable time frame is indicative of the problem - an error will occur, on the order of minutes or even longer.

Problem (a):

Reliable dir-tree-removal test: Repeatedly creating and removing a directory subtree, using ps for removal.
.........Remove-Item : Directory C:\Users\jdoe\tmpDir\sub\sub cannot be removed because it is not empty.

That is, recursive removal of the target dir's content failed due to async timing issues.

Problem (b):

Synchronous dir-removal test: Repeatedly creating and removing a directory, using ps for removal.
.................New-Item : An item with the specified name C:\Users\jdoe\tmpDir already exists.

That is, recreating the target dir failed, because the prior removal hadn't yet completed.

Aux. function definitions:

Functions `Assert-ReliableDirRemoval` and `Assert-SyncDirRemoval`

You can paste the entire code at the prompt in order to define these functions.

function Assert-ReliableDirRemoval {
  param([parameter(Mandatory)] [ValidateSet('cmd', '.net', 'ps')] [string] $method)
  Write-Host "Reliable dir-tree-removal test: Repeatedly creating and removing a directory subtree, using $method for removal."
  # Treat all errors as fatal.
  $ErrorActionPreference = 'Stop'
  Set-Location $HOME  # !! Seemingly, a dir. in the user's home dir. tree reproduces the symptom fastest - $env:TEMP does not.
  # Remove a preexisting directory first, if any.
  Remove-Item -EA Ignore -Recurse tmpDir
  While ($true) {
    Write-Host -NoNewline .
    # Create a subdir. tree.
    $null = New-Item -Type Directory tmpDir, tmpDir/sub, tmpDir/sub/sub, tmpDir/sub/sub/sub
    # Fill each subdir. with 1000 empty test files.
    "tmpDir", "tmpDir/sub", "tmpDir/sub/sub", "tmpDir/sub/sub/sub"| ForEach-Object {
      $dir = $_
      1..1e3 | ForEach-Object { $null > "$dir/$_" }
    }
    # Now remove the entire dir., which invariably involves deleting its contents
    # recursively first.
    switch ($method) {
      'ps'  { Remove-Item -Recurse tmpDir }
      '.net' { [System.IO.Directory]::Delete((Convert-Path tmpDir), $true) }
      'cmd'  { cmd /c rd /s /q tmpDir; if ($LASTEXITCODE) { Throw } }
      # !! If rd /s fails during recursive removal due to async issues, it emits a stderr line, but 
      # !! does NOT report a nonzero exit code. We detect this case below.
    }
    # Does the dir. unexpectedly still exist?
    # This can happen for two reasons:
    #  - The removal of the top-level directory itself has been requested, but isn't complete yet.
    #  - Removing the content of the top-level directory failed quietly.
    while (Test-Path tmpDir) {
      if ($output = Get-ChildItem -EA Ignore -Force -Name tmpDir) { # Still has content?
        Throw "Deletion failed quietly or with exit code 0, tmpDir still has content: $output"
      } else {
        # Top-level directory removal isn't complete yet.
        # Wait for removal to complete, so we can safely recreate the directory in the next iteration.
        # This loop should exit fairly quickly.
        Write-Host -NoNewline !; Start-Sleep -Milliseconds 100
      }      
    }    
  }
}

function Assert-SyncDirRemoval { 
  param([parameter(Mandatory)] [ValidateSet('cmd', '.net', 'ps')] [string] $method)
  Write-Host "Synchronous dir-removal test: Repeatedly creating and removing a directory, using $method for removal."
  # Treat all errors as fatal.
  $ErrorActionPreference = 'Stop'
  Set-Location $HOME  # !! Seemingly, a dir. in the user's home dir. tree reproduces the symptom fastest - $env:TEMP does not.
  While ($true) {
    Write-Host -NoNewline .
    # Remove the test dir., which invariably involves deleting its contents recursively first.
    # Note: This could itself fail intermittently, but with just 10 files and no subdirs. is unlikely to.
    if (Test-Path tmpDir) {
      switch ($method) {
        'ps' { Remove-Item -Recurse tmpDir }
        '.net' { [System.IO.Directory]::Delete((Convert-Path tmpDir), $true) }
         'cmd'  { cmd /c rd /s /q tmpDir; if ($LASTEXITCODE) { Throw } }
         # !! If rd /s fails during recursive removal due to async issues, it emits a stderr line, but 
         # !! does NOT report a nonzero exit code. We detect this case below.
      }
    }
    # (Re-)create the dir. with 10 empty test files.
    # If the previous removal hasn't fully completed yet, this will fail.
    # Note: [System.IO.Directory]::Delete() could have quietly failed to remove the contents of the dir.
    #       due to async timing accidents, but, again, this is unlikely with 1- files and 
    try {
      $null = New-Item -Type Directory tmpDir
    } catch {
      # Handle the case where removal failed
      # quietly due to async vagaries while removing the dir's *content*, which
      # can happen with [System.IO.Directory]::Delete().
      # Note that if removal succeeded but is pending - the scenario we're trying to
      # get to occur - Get-ChildItem reports an access-denied error, which we ignore
      # in favor of reporting the original error from the dir. re-creation attempt.
      if ($output = Get-ChildItem -EA Ignore -Force -Name tmpDir) { # Still has content?
        Write-Warning "Ignoring failed content removal, retrying..."
        # Simply try again
        Start-Sleep -Milliseconds 500; Remove-item tmpDir -Recurse
        continue
      }
      # Re-creation failed due to async removal.
      Throw $_
    }
    1..10 | ForEach-Object { $null > tmpDir/$_ }
  }
}

Environment data

PowerShell Core 6.2.0-preview.1 on Microsoft Windows 10 Pro (64-bit; Version 1803, OS Build: 17134.165)
Windows PowerShell v5.1.17134.228 on Microsoft Windows 10 Pro (64-bit; Version 1803, OS Build: 17134.345)

Is this potentially a problem with when the file system itself decides to flush?

Steve Lee (@SteveL-MSFT)

Seems like it, Steve Lee (@SteveL-MSFT).

I'm still looking into this, but it seems that the WinAPI RemoveDirectory() function is inherently asynchronous, which is why .NET and cmd are equally affected.

From the docs:

"The RemoveDirectory function marks a directory for deletion on close. Therefore, the directory is not removed until the last handle to the directory is closed."

So it may come down to the following choice for us:

Live with the behavior (as cmd and .NET do) and and document it (which cmd and .NET have not done).
As a courtesy - if technically feasible and reasonable - implement the synchronous behavior ourselves.

I'm looking into the latter option, as it seems preferable.

Michael Klement (@mklement0)

Michael Klement (@mklement0) is this something you hit on a regular basis outside of your synthetic test?

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT): As I now realize, the problem is actually more fundamental and more insidious, due to its intermittent, unpredictable nature:

Because file deletion is equally asynchronous, Remove-Item -Recurse itself fails intermittently - see https://serverfault.com/q/199921/176094

And, yes, I have seen such failures in the wild.

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT): Please see the updated initial post: Remove-Item -Recurse is not only unexpectedly asynchronous overall (which is what I originally reported), it fails intermittently, due to inherently flawed design.

I've also seen something similar happen in the wild. Again, it was rather intermittent, but it happened often enough that I've changed some scheduled scripts. One of the ways I worked around it was:

Get-ChildItem $Path -Recurse | Sort-Object -Property FullName -Descending | Remove-Item

The sorting simply ensures that children are passed before parents and leafs before containers. That makes the -Recurse parameter on Remove-Item irrelevant. I don't know if this creates a side effect where the asynchronous issue simply doesn't come up, but it has been the most successful pattern I've found in my limited experience.

And, more rarely, I've done something like this:

Get-ChildItem $Path -Recurse -File | Remove-Item
Get-ChildItem $Path -Directory | ForEach-Object { $_.Delete($true) }

For whatever reason, recursively removing only directories seems to work without any difficulty in my cases.

I should note that in both cases this is usually on network shares, which might make it a separate issue.

Michael Klement (@mklement0)

Michael Klement (@mklement0) I appreciate the research into this, but I don't see a solution here if the native APIs have this behavior. We can't make the cmdlet synchronous as the delete may never complete if something has an open handle to it and the cmdlet will be deadlocked.

Is it possible to handle the case where deletion of a directory fails due to remaining file(s) and have it retry any failing directory deletions after all file deletions have been processed? If that still fails, it can emit a much more friendly and useful error message "cannot delete folders x, y, and z because files A and B could not be removed" or some such.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32) that's a good suggestion since the user explicitly indicated their intent

AikenBM

Thanks, AikenBM: I think your workarounds decrease the likelihood of running into timing issues, by putting more time between deletion of a file and its container, but they may still arise.

Thanks, Rain Sallow (/u/ta11ow) (@vexx32): Retrying would be an improvement, but is still not guaranteed to work in all cases.
The linked video contains a robust solution that doesn't require retrying, summarized in this image:

I don't think we should do 5., because encountering a legitimately locked file should cause the operation to abort with an error message.

Note that %TEMP in the image doesn't literally refer to %TEMP% / $env:TEMP, but any writable location on the same volume as the directory being removed.

The technique relies on moving / renaming apparently being synchronous if performed on the same volume, i.e., the original directory entry is guaranteed to have disappeared on returning from the move/rename.
(Note that files opened with FILE_SHARE_DELETE will live on in their temporary form until their last handle is closed - this is also an improvement over the current behavior, which fails to delete a directory containing such files).

However, there are challenges:

The speaker in the video recommends using the parent directory of the directory being removed.
However, while you typically would expect that directory to be writable too, that's not guaranteed.
I'm unclear on how deleting files / directories from network shares fits into the picture.

I've implemented the above algorithm in a custom PowerShell function named Remove-FileSystemItem in this SO answer, and it passes the two tests above.

It uses .NET methods for better performance, but still won't be a speed demon, but it could be considered a proof of concept for a proper cmdlet solution.

On Unix, where the timing problem doesn't exist, it simply defers to Remove-Item.

Caveats:

As discussed, it uses the parent directory of the target directory to temporarily move the items being deleted to.
It bases the decision on whether to apply the workaround solely on the host OS, so that on Window it is also applied to network drives there, even though there may be no need. Not sure if there's an easy way to infer from the properties of a shared drive whether file deletion on it will be asynchronous or not.

The documentation states that the directory is deleted when the last handle is closed. It does not state that deletion is asynchronous.

Seemingly, a dir. in the user's home dir. tree reproduces the symptom fastest - $env:TEMP does not.

Why would that matter? My theory is that the failures demonstrated by this script are because other programs such as indexing programs very briefly access the file system items in question. This can then indeed delay the deletion until their handle is closed.

I tried to reproduce this. The only way I can reproduce it is if I access the files in question. Even navigating with Windows Explorer into the test directory is enough to reproduce it. But it does not occur on its own for me.

I have seen the talk about file system fragility. So far I have not seen a lot of evidence that deletion is actually asynchronous. The talk does not have any evidence, just a claim.

GSPP

GSPP:

deleted when the last handle is closed. It does not state that deletion is asynchronous.

Only getting deleted when the last handle is closed is being asynchronous, even if that word isn't being used in the docs.

What matters is: by the time the call returns, you cannot know whether deletion has completed yet or not. I call that asynchronous. What would you call it?
Irrespective of what we call it: it's a problem.

My theory is that the failures demonstrated by this script are because other programs such as indexing programs very briefly access the file system items in question

Probably, yes.

the only way I can reproduce if I access the files in question.
So far I have not seen a lot of evidence that deletion is actually asynchronous

The first sentence shows that you have seen the evidence.

Since File Explorer isn't locking files, it shouldn't prevent deletion of a directory (for what should and shouldn't prevent deletion, see this SO answer of mine).
Indexing programs shouldn't prevent reliable deletion of a directory.

In short: the unpredictable behavior of directory removal has been causing problems for years (here's another example), and I think it's time to fix it.

jantari

jantari:

Yes, I meant to imply that my conclusion is that the issue is resolved for [System.IO.File]::DeleteFile/Directory and Remove-Item (it is not resolved for cmd's rmdir, presumably because it doesn't use the DeleteFile Windows API function, which is the one now acting synchronously).

The conclusion is based on what I've stated in my previous comment.
It would be nice to have first-hand confirmation, but, in its absence, the third-party sources and the Assert-ReliableDirRemoval and Assert-SyncDirRemoval tests are the best we have so far.

These tests delete and re-create directory (trees) in an infinite loop, failing only if asynchronous behavior is detected.
On older Windows 10 versions these tests failed for me within minutes at the most, whereas they now run without failure for hours, even when I put additional stress on the file-system by running Get-ChildItem C:\ -Recurse in parallel (on Windows 10 20H2 / 19042.630; the claim that the first version where it started working was 1909 / 18363.657 is from third-party sources).

Saying that something on your laptop doesn't work is worth investigating, but in itself not evidence to the contrary (the failure may be unrelated, for instance - without having a reproducible scenario, we can't know).

under artificial conditions

All tests are by definition artificial - what matters is whether they successfully emulate real-world conditions.

I invite you to:

Study the methodology of the Assert-ReliableDirRemoval and Assert-SyncDirRemoval (see OP) to see if it they fall short with respect to emulating real-world scenarios.
Run the tests yourself (with argument ps), for hours - they should run indefinitely, implying that removal is now synchronous; conversely, a failure indicates lack of synchronicity.
Additionally, based on the failure you observe on your laptop, try to construct a test that reproducibly fails (eventually - the intermittent, unpredictable nature of the behavior is what makes it so insidious), so that others can verify the failure and help investigate.

As there is an open issue (PowerShell/PowerShell#8211) with the -Recu…

Ilya (@iSazonov)

Ilya (@iSazonov) i think we can close this issue

Michael Klement (@mklement0)

Michael Klement (@mklement0) Can we close since now we support only Windows versions >= 10?

Ilya (@iSazonov)

Ilya (@iSazonov), I think so, yes.

I'm still having this issue on Windows Server 2022 (build 20348.4529) with PowerShell 7.5.4. When I run Remove-Item -LiteralPath "\\my-server\code\old_app" -Force -Recurse, it occasionally fails with the following error:

The directory is not empty. : '\my-server\code\old_app\public\css'.

I'm still having this issue on Windows Server 2022 (build 20348.4529) with PowerShell 7.5.4. When I run Remove-Item -LiteralPath "\\my-server\code\old_app" -Force -Recurse, it occasionally fails with the following error:

The directory is not empty. : '\my-server\code\old_app\public\css'.

I am also still seeing this issue on Windows Server 2019 build 17763.

Theodore Brown (@theodorejb)

Theodore Brown (@theodorejb), Joe Delaney (@joedel94) I wonder if you're hitting #27710

Tadas Medišauskas (@Tadas)

Tadas Medišauskas (@Tadas) in this case I am not, the directory I am hitting this with is about 4 total files. I have also replicated this since my last post and am now simply adding logic to ensure all files in the directory are removed before removing the directory itself as a workaround. A bit of a pain to add the extra logic but it works fine.

@theodorejb, @joedel94 I wonder if you're hitting #27710

Navigation Menu

Repository navigation

On Windows, Remove-Item fails intermittently with -Recurse and is unexpectedly asynchronous itself #8211

Steps to reproduce

Expected behavior

Actual behavior

Aux. function definitions:

Functions `Assert-ReliableDirRemoval` and `Assert-SyncDirRemoval`

Environment data

14 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

On Windows, Remove-Item fails intermittently with -Recurse and is unexpectedly asynchronous itself #8211

Description

Steps to reproduce

Expected behavior

Actual behavior

Aux. function definitions:

Functions Assert-ReliableDirRemoval and Assert-SyncDirRemoval

Environment data

Activity

SteveL-MSFT commented on Nov 8, 2018

mklement0 commented on Nov 8, 2018

SteveL-MSFT commented on Nov 8, 2018

mklement0 commented on Nov 8, 2018

mklement0 commented on Nov 19, 2018

AikenBM commented on Nov 21, 2018

SteveL-MSFT commented on Nov 21, 2018

vexx32 commented on Nov 21, 2018

SteveL-MSFT commented on Nov 21, 2018

mklement0 commented on Nov 25, 2018

mklement0 commented on Dec 1, 2018

GSPP commented on Dec 3, 2018

mklement0 commented on Dec 3, 2018

14 remaining items

mklement0 commented on Nov 12, 2020

CarloToso commented on Mar 17, 2023

iSazonov commented on Mar 17, 2023

mklement0 commented on Mar 17, 2023

theodorejb commented on Jan 2, 2026

joedel94 commented on Jul 9, 2026

Tadas commented on Jul 22, 2026

joedel94 commented on Jul 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Functions `Assert-ReliableDirRemoval` and `Assert-SyncDirRemoval`