WIP: Windows: Alternate Data Steams on directories are now accessible. #13795

kyanha · 2020-10-17T03:17:52Z

PR Summary

This PR allows Get-Item -Stream, Get-Content -Stream, Clear-Content -Stream, Set-Content -Stream, Add-Content -Stream, and Remove-Item -Stream to see and address alternate data streams on directories, not merely on files.

Fixes #10570. Fixes #13656.

Supersedes #13650 (squashes all intermediary commits).

PR Context

Issue #10570 has been open for a year. NTFS supports what are called "Alternate Data Streams" on both files and directories (multiple named discrete blobs of data which are associated with a single directory entry). PowerShell currently supports enumeration of these Alternate Data Streams on files, using the '-Stream' parameter to 'Get-Item'. It also supports manipulation of these alternate data streams on files, using the '-Stream' parameter to Set-Content, Add-Content, Clear-Content, and Remove-Item.

Unfortunately, the initial implementation of PowerShell only supported alternate data streams on files, not on directories. This makes an entire facility of the OS's file system invisible, and if an administration team is relying on PowerShell it makes an attractive place for a red team to store data to exfiltrate. (This is not an invitation to destroy the capability to store alternate data streams on directories, as they are useful for many purposes. It is merely a rationale for making their existence visible through PowerShell.)

To create and see an alternate data stream on a directory, use cmd.exe to run the following commands:

> mkdir 10570demo
> cd 1057demo
> echo "This is a file." > 10570demo.txt
> echo "This is an alternate data stream on the file." > 10570demo.txt:datastream
> mkdir bug10570
> echo "This is an alternate data stream on the directory." > bug10570:datastream
> dir /r

The output is something like:

D:\10570demo>dir /r
 Volume in drive D is DATA
 Volume Serial Number is 8FD3-BD69

 Directory of D:\10570demo

09/17/2020  02:59 PM    <DIR>          .
09/17/2020  02:59 PM    <DIR>          ..
09/17/2020  02:58 PM                20 10570demo.txt
                                    50 10570demo.txt:datastream:$DATA
09/17/2020  02:59 PM    <DIR>          bug10570
                                    55 bug10570:datastream:$DATA
               1 File(s)             20 bytes
               3 Dir(s)  88,185,401,344 bytes free

To see the failure of PowerShell being able to see the stream on the file, but not the directory:

> pwsh
PS > Get-Item *


    Directory: D:\10570demo

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
da---           9/17/2020  2:59 PM                bug10570
-a---           9/17/2020  2:58 PM             20 10570demo.txt

PS > Get-Item * -stream *

PSPath        : Microsoft.PowerShell.Core\FileSystem::D:\10570demo\10570demo.txt::$DATA
PSParentPath  : Microsoft.PowerShell.Core\FileSystem::D:\10570demo
PSChildName   : 10570demo.txt::$DATA
PSDrive       : D
PSProvider    : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName      : D:\10570demo\10570demo.txt
Stream        : :$DATA
Length        : 20

PSPath        : Microsoft.PowerShell.Core\FileSystem::D:\10570demo\10570demo.txt:datastream
PSParentPath  : Microsoft.PowerShell.Core\FileSystem::D:\10570demo
PSChildName   : 10570demo.txt:datastream
PSDrive       : D
PSProvider    : Microsoft.PowerShell.Core\FileSystem
PSIsContainer : False
FileName      : D:\10570demo\10570demo.txt
Stream        : datastream
Length        : 50

PS > Get-Item bug10570 -stream *
PS >

Writing the tests revealed that Set-Content internally calls Clear-Content, which is hardcoded to not check for streams on directories. This was raised as an issue in Issue #13656, but I decided to put that into this PR as well.

PR Checklist

PR has a meaningful title
Summarized changes
Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
This PR is ready to merge and is not Work in Progress.
- If the PR is work in progress, please add the prefix WIP: or [ WIP ] to the beginning of the title (the WIP bot will keep its status check at Pending while the prefix is present) and remove the prefix when the PR is ready.
Breaking changes
- I'm assuming bucket 3: current behavior of Get-Item -stream when confronted with a directory was to silently do nothing, and this PR continues to do so in the event a directory doesn't have any alternate data streams. I would like someone to doublecheck, though.
User-facing changes
- Documentation needed
  - Issue filed: Get-Item -stream now works on directories. (PS #13941) MicrosoftDocs/PowerShell-Docs#6648
Testing - New and feature
- Make sure you've added a new test if existing tests do not effectively test the code changed
Tooling
- I have considered the user experience from a tooling perspective and don't believe tooling will be impacted.

kyanha · 2020-10-17T03:48:00Z

All issues raised by CodeFactor are in code that I didn't touch. @anmenaga Please review when you get a moment?

n3rdopolis · 2020-10-20T17:15:59Z

I tested it, it works. One thing though before the compatibility is set in stone, does it make sense that PsIsContainer is $True? It's a member of a container, but I don't think streams that are parts of folders can have any children in NTFS....

kyanha · 2020-10-23T02:25:39Z

@n3rdopolis

I tested it, it works. One thing though before the compatibility is set in stone, does it make sense that PsIsContainer is $True? It's a member of a container, but I don't think streams that are parts of folders can have any children in NTFS....

You are correct, data streams cannot contain children. That's a handy mnemonic to understand why files always have default (unnamed) data streams, and why directories can never have default data streams.

Unfortunately, I can't figure out the path by which the property is set, so I can't figure out if I have enough information available anywhere to communicate to that location that even though the filesystem entry is a directory it's actually a stream on a directory.

If you have any suggestions to figuring out how it works, I am attentive. I'm just a filesystem guy, not a PowerShell internal structure guy, and I've been trying to push this so I can get back to work on what I'm actually supposed to be working on (a Windows filesystem filter driver that uses alternate data streams to store persistent data). :/

kyanha · 2020-10-30T05:46:22Z

Unfortunately, I can't figure out the path by which the property is set, so I can't figure out if I have enough information available anywhere to communicate to that location that even though the filesystem entry is a directory it's actually a stream on a directory.

I did some exploring tonight, and thought I might have figured it out, but alas it didn't work the way I wanted to. You apparently can't use Get-Item drive:\directory:* as a synonym for Get-Item drive:\directory -Stream *, which means that when it passes through private static FileSystemInfo GetFileSystemInfo(string path, out bool isContainer) it doesn't get a stream name. So, even though that seems to be where isContainer is set, it apparently doesn't get enough information for Get-Item to be able to say it's not a container.

I can't figure out where "Stream" is added as a dynamic parameter, so I can't really do anything there either. Ideally, if PSChildName contains a ':', isContainer should be reset to false before it's returned.

kyanha · 2020-10-30T05:56:37Z

I found it! Now, I need to write tests for it to make sure it's correct. Will set the title here to WIP.

kyanha · 2020-10-30T07:08:22Z

I apparently suck at git, and I've gotten my local repository into a state where I can't push it. Will try again on Yet Another Branch[tm].

kyanha · 2020-10-30T07:19:30Z

Superseded by #13941, because apparently some pull I made is trying to update workflow but my OAUTH token doesn't have the "workflow" scope. That PR has all tests written and all commits squashed for review, and addresses @n3rdopolis's concern about $PSIsContainer on the alternate data streams.

Windows: Alternate Data Steams on directories are now accessible

4f93144

kyanha requested a review from anmenaga as a code owner October 17, 2020 03:17

ghost assigned daxian-dbw Oct 17, 2020

kyanha mentioned this pull request Oct 17, 2020

Get-Item directory -stream * now works #10570 #13650

Closed

9 tasks

iSazonov added the CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log label Oct 17, 2020

Alternate data streams can never be containers

f338ff9

kyanha changed the title ~~Windows: Alternate Data Steams on directories are now accessible.~~ WIP: Windows: Alternate Data Steams on directories are now accessible. Oct 30, 2020

kyanha mentioned this pull request Oct 30, 2020

The -Stream parameter now works with directories #13941

Merged

9 tasks

kyanha closed this Oct 30, 2020

kyanha deleted the issue10570v2 branch October 30, 2020 07:23

kyanha restored the issue10570v2 branch December 2, 2020 11:45

kyanha deleted the issue10570v2 branch December 3, 2020 20:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: Windows: Alternate Data Steams on directories are now accessible. #13795

WIP: Windows: Alternate Data Steams on directories are now accessible. #13795

Uh oh!

kyanha commented Oct 17, 2020 •

edited

Loading

Uh oh!

kyanha commented Oct 17, 2020

Uh oh!

n3rdopolis commented Oct 20, 2020

Uh oh!

kyanha commented Oct 23, 2020

Uh oh!

kyanha commented Oct 30, 2020

Uh oh!

kyanha commented Oct 30, 2020

Uh oh!

kyanha commented Oct 30, 2020

Uh oh!

kyanha commented Oct 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WIP: Windows: Alternate Data Steams on directories are now accessible. #13795

WIP: Windows: Alternate Data Steams on directories are now accessible. #13795

Uh oh!

Conversation

kyanha commented Oct 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Context

PR Checklist

Uh oh!

kyanha commented Oct 17, 2020

Uh oh!

n3rdopolis commented Oct 20, 2020

Uh oh!

kyanha commented Oct 23, 2020

Uh oh!

kyanha commented Oct 30, 2020

Uh oh!

kyanha commented Oct 30, 2020

Uh oh!

kyanha commented Oct 30, 2020

Uh oh!

kyanha commented Oct 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kyanha commented Oct 17, 2020 •

edited

Loading