Pasted UTF-8 characters into PowerShell shell are interpreted incorrectly.

I've put the script section first, instead of the description of the issue, because my examples aforementioned utilize values from this script to demonstrate the problem.

Prerequisites

Write a descriptive title.
Make sure you are able to repro it on the latest released version
Search the existing issues.
Refer to the FAQ.
Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Expected behaviour

Originally, this issue was going to be about how PowerShell can't handle certain legal UTF-8 characters in inode paths. However, after creating a test script:

#!/usr/bin/env -S pwsh -File

# Prerequisites
# -------------

	# ~~~pwsh
	# #Requires -PSEdition 'Core'
	# #Requires -Version 7
	# ~~~
	
	Set-Location -LiteralPath "$HOME"

# Example Directory
# -----------------

	# Creates an example directory.
	
	$DirectoryPath = 'TabPrefixed	ExampleDirectory.dir'
	$PrintPath = "$DirectoryPath"
	Write-Output (Show-Markdown -InputObject "The path is ``$PrintPath``.")
	
	New-Item `
		-Path "$HOME" `
		-Name "$DirectoryPath" `
		-ItemType 'directory'

# Example File
# ------------

	# Creates an example file.
	
	$FilePath = 'TabPrefixed	ExampleFile.txt'
	$PrintPath = "$FilePath"
	Write-Output (Show-Markdown -InputObject "The path is ``$PrintPath``.")
	
	New-Item `
		-Path ("$HOME" + [IO.Path]::DirectorySeparatorChar + "$DirectoryPath") `
		-Name "$FilePath" `
		-ItemType 'file' `
		-Value 'Example text.'

# Path Acquisition
# ----------------

	# Lists both.
	
	$FullFilePath = (`
	Get-ChildItem `
		-LiteralPath (
			"$HOME"          + [IO.Path]::DirectorySeparatorChar +
			"$DirectoryPath" + [IO.Path]::DirectorySeparatorChar +
			"$FilePath")
		)[0] | Select-Object -ExpandProperty 'FullName'

# File Content Acquisition
# ------------------------

	# This should fail.
	
	$PrintPath = "$FullFilePath"
	Write-Output (Show-Markdown -InputObject "The path is ``$PrintPath``.")
	
	Get-Content -LiteralPath "$FullFilePath"
	Write-Output $?

I realized that it could. Instead, I've narrowed it to a fault in how PowerShell, via Konsole and VSCode's terminals, interprets certain characters (like - tabs) as control codes.

Steps to reproduce

Try pasting your equivalent to the undermentioned command:

#!/usr/bin/env pwsh
Get-Content -LiteralPath '/home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt'

...into the shell after successfully invoking it.

This value is able to be acquired by the undermentioned command:

#!/usr/bin/env pwsh
(Get-ChildItem -LiteralPath "$DirectoryPath")[0].FullName

Actual behavior

The undermentioned is a command that has been pasted:
```
#!/usr/bin/env pwsh
Get-Content -LiteralPath "/home/RokeJulianLockhart/Documents/pyside6/prefix^Iexample.txt/example.txt"
```
Note within it, the "prefix^Iexample". The original content does not contain that control code.

Alternatively, sometimes, it appears to paste the path correctly, but doesn't recognise the value unless passed in a script:

PS /home/RokeJulianLockhart> (Get-ChildItem -LiteralPath "$DirectoryPath")[0].FullName
/home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt

PS /home/RokeJulianLockhart> Get-Content -LiteralPath '/home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt'                                                                                                                                 
Get-Content: Cannot find path '/home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt' because it does not exist.

PS /home/RokeJulianLockhart> Get-Content -LiteralPath "$FullFilePath"                                                                                  
Example text.

PS /home/RokeJulianLockhart> Write-Output "$FullFilePath"            
/home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt

Error details

```
#!/usr/bin/env pwsh
Get-Error
```

Exception             : 
    Type                 : System.Management.Automation.ItemNotFoundException
    ErrorRecord          : 
        Exception             : 
            Type    : System.Management.Automation.ParentContainsErrorRecordException
            Message : Cannot find path '/home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt' because it does not exist.
            HResult : -2146233087
        TargetObject          : /home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt
        CategoryInfo          : ObjectNotFound: (/home/RokeJulianLoc…    ExampleFile.txt:String) [], ParentContainsErrorRecordException
        FullyQualifiedErrorId : PathNotFound
    ItemName             : /home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt
    SessionStateCategory : Drive
    TargetSite           : 
        Name          : ResolveDriveQualifiedPath
        DeclaringType : [System.Management.Automation.LocationGlobber]
        MemberType    : Method
        Module        : System.Management.Automation.dll
    Message              : Cannot find path '/home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt' because it does not exist.
    Source               : System.Management.Automation
    HResult              : -2146233087
    StackTrace           : 
   at System.Management.Automation.LocationGlobber.ResolveDriveQualifiedPath(String path, CmdletProviderContext context, Boolean allowNonexistingPaths, CmdletProvider& providerInstance)
   at System.Management.Automation.LocationGlobber.GetGlobbedMonadPathsFromMonadPath(String path, Boolean allowNonexistingPaths, CmdletProviderContext context, CmdletProvider& providerInstance)
   at System.Management.Automation.PathIntrinsics.GetResolvedPSPathFromPSPath(String path, CmdletProviderContext context)
   at Microsoft.PowerShell.Commands.ContentCommandBase.ResolvePaths(String[] pathsToResolve, Boolean allowNonexistingPaths, Boolean allowEmptyResult, CmdletProviderContext currentCommandContext)
TargetObject          : /home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt
CategoryInfo          : ObjectNotFound: (/home/RokeJulianLoc…    ExampleFile.txt:String) [Get-Content], ItemNotFoundException
FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetContentCommand
InvocationInfo        : 
    MyCommand        : Get-Content
    ScriptLineNumber : 1
    OffsetInLine     : 1
    HistoryId        : 12
    Line             : Get-Content -LiteralPath '/home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt'
    Statement        : Get-Content -LiteralPath '/home/RokeJulianLockhart/TabPrefixed    ExampleDirectory.dir/TabPrefixed        ExampleFile.txt'
    PositionMessage  : At line:1 char:1
                       + Get-Content -LiteralPath '/home/RokeJulianLockhart/TabPrefixed    Exa …
                       + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    InvocationName   : Get-Content
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : 
      0
      1

Forgot to Export-CLIXML.

Environment data

#!/usr/bin/env pwsh
$PSVersionTable | ConvertTo-YAML

OS: Fedora Linux 40 (KDE Plasma)
SerializationVersion:
  Major: 1
  Minor: 1
  Build: 0
  Revision: 1
  MajorRevision: 0
  MinorRevision: 1
PSRemotingProtocolVersion:
  Major: 2
  Minor: 3
  Build: -1
  Revision: -1
  MajorRevision: -1
  MinorRevision: -1
Platform: Unix
WSManStackVersion:
  Major: 3
  Minor: 0
  Build: -1
  Revision: -1
  MajorRevision: -1
  MinorRevision: -1
GitCommitId: 7.4.3
PSEdition: Core
PSVersion:
  Major: 7
  Minor: 4
  Patch: 3
  PreReleaseLabel: 
  BuildLabel:

Visuals

Screencast_20240710_125007.webm

How it could be represented? As in bash?

237dmitry

#24037 (comment)

237dmitry, I've asked about that at SE SuperUser, and the answer is such that it does appear feasible. Especially, for PowerShell, these utilities don't even need to be able to detect other shells' syntaxes (like ls does). Certainly, I can't think of any other solution this, else I would have filed this as an FR (rather than the BR it is).

Ideally, either -LiteralPath should actually accept verbatim the characters (and never automatically convert pasted paths into partially-escaped ones, like it very occasionally does) or we should automatically escape duplicated paths in every situation. Either would be easily comprehensible to the user.

should actually accept verbatim the characters

Yes, I think so. Maybe with `t or \t replacement.

237dmitry

#24037 (comment)

237dmitry, my point is merely that the user shouldn't need to manually modify the path before pasting it, since they don't need to in scripts.

1 Usually, saving the script as UTF8 encoding with BOM header can solve the problem of file names with special encoding in the script. But this is not compatible with “#!"
2 I usually use mkdir - p 'directory name with utf8 encoding'

kasini3000

#24037 (comment)

kasini3000:

That's a good point - I've been using UTF-8 with a BOM for my PS1 scripts, which hasn't been problematic because I use VSCode's configuration to determine the debugger. Perhaps I didn't notice it in my scripts because of that.
The script I've been using is just LF-delimited UTF-8.

#24037 (comment)

rhubarb-geek-nz, indeed, PSReadLine seems like the better place for this, because it's certainly context-dependent - I think having this behaviour work in -LiteralPath arguments would usually be enough - and because, as you state, there are limitations to a terminal GUI.

~~Can someone transfer this? Otherwise, I'll create a new issue there.~~
Interpret control characters pasted into -LiteralPath arguments as verbatim characters. PSReadLine#4072 (comment)

For context:

The concept of a character-encoding such as UTF-8 doesn't apply to in-memory strings getting pasted.
- Scripts and files without a BOM default to UTF-8 in PowerShell Core; the only reason to use a BOM is for cross-edition scripts that Windows PowerShell too must interpret correctly.
Technically, a tab character is in the ASCII subrange of the Unicode "alphabet", and is a so-called C0 control character.
Pasting into a terminal in most environments is done by simulated typing, unfortunately. In such environments pasting something with a tab causes that tab char. to be interpreted as a request for tab-completion, thereby effectively discarding the tab while possibly altering what is pasted.
- Overcoming this limitation is discussed in the following PSReadLine issues:
  - Right click paste should work mostly like Ctrl+v paste PSReadLine#579
  - Support "Paste Bracketing" PSReadLine#1471 (if implemented, would solve the previous one too).
While the old-style conhost.exe console windows on Windows do support direct pasting, they seemingly discard tab characters.
macOS terminals either quietly convert a pasted tab to a space (Terminal.app) or prompt for whether the tabs should be pasted (iTerm.app), but the pasting then again happens via simulated typing.

I'm unclear on whether PSReadLine even has the option to override these behaviors.

As for tab-completing a filename with embedded tabs (not supported on Windows), as observed on macOS:

bash completes with a literal tab char., properly escaped
PowerShell completes with the tab char. too, but it is rendered as ^I and not escaped, which could be considered a bug (to make it work, the ^I must either be `-escaped, or the whole argument must be quoted; that is, PowerShell itself doesn't recognize the need to quote such a path).

Michael Klement (@mklement0)

#24037 (comment)

Michael Klement (@mklement0), that's a brilliantly comprehensive explanation. Would you consider this issue a duplicate of PowerShell/PSReadLine#1471 (comment), or should I file a separate issue at that repository? Alternatively, considering that you doubt that PSReadLine is able to override this, can you advise whether there might there be merit to leaving this open here?

Additionally, microsoft/terminal#9034 (comment) appears to indicate that this might need to be supported by the terminal too, and I don't know whether VSCode and Konsole support bracketed paste anyway.

Thanks, @RokeJulianLockhart.

PowerShell/PSReadLine#1471 (comment) may be a duplicate - I'm not certain if tabs would be retained as-is as part of a bracketed paste.

I see that you opened a new PSReadLine issue: I suggest updating it with the above information.

It looks like the Windows Terminal PR you linked to was merged quite a while ago, so I assume the feature is already in place.

The concept of a character-encoding such as UTF-8 doesn't apply to in-memory strings getting pasted

When using Get-Clipboard, tab characters are not replaced with spaces.:

$ [int[]][char[]] "a    b"    # pasted from clipboard with Ctrl+V
97
32
32
32
32
98

$ [int[]][char[]]  (Get-Clipboard)
97
9
98

@RokeJulianLockhart As shown in the screenshot below, you copied the text from the terminal output before pasting to Get-Content -LiteralPath. I believe when you copy the terminal output, you already lost the tab characters \t. Can you please double check if that's the case? You can check by pasting to VSCode editor after copying from the terminal output, and see if the tab has been replaced with spaces already.

Dongbo Wang (@daxian-dbw)

Dongbo Wang (@daxian-dbw), while it is true that tab chars. are lost when copying output from VSCode's integrated terminal (they become spaces), the problem exists even if you paste a string with bona fide tab characters.

Yes, I understand. Tab is intentionally replaced with 4 space characters (see the code), and it was a by design change: PowerShell/PSReadLine#144.

I just want to point out that the scenario described by this issue -- copy the terminal output and paste in your command -- won't work even if PSReadLine doesn't alter the Tab character. And I don't think it's worth breaking the current behavior.

Dongbo Wang (@daxian-dbw)

Dongbo Wang (@daxian-dbw):

Good to know about PSReadLine's tabs-to-spaces conversion.

copy the terminal output and paste in your command -- won't work even if PSReadLine doesn't alter the Tab character.

That may be true for VSCode, but it isn't true for other terminal emulators, such as on macOS - on copying, they do preserve literal tab chars. in the selection.

Let me try to summarize:

PSReadLine converts each literal tab char. pasted into 4 spaces by design; Bash's readline library appears to do a similar thing (1 space). Additionally, terminal applications such as iTerm.app on macOS themselves offer modifying the pasted text if it contains tab chars., before passing it to the target shell.
- For pasting code this makes sense, but it prevents pasting file paths that contain embedded tab chars. - though that is definitely an edge case.
PowerShell's tab-completion completes file names with embedded tab chars. as such, but renders them as ^I (which is just a display problem). However, it neglects to quote such file names, which is arguably a bug (but, again, an edge case).

Dongbo Wang (@daxian-dbw)

Dongbo Wang (@daxian-dbw), my sincere apologies for the wait to respond. I've been spending some time diagnosing some related problems, like microsoft/debugpy#1671 (comment), and then migrating wholesale away from tabs in file names. ¹

Irrespective, it certainly does appear to be replaced, and solely by pwsh - not bash:

To demonstrate that I'm not mistakenly utilizing the content pasted to the CLI:

2024-10-22.17-57-45.mp4

stackoverflow.com/revisions/79048760/10 ↩

237dmitry

237dmitry, I'd try Get-Clipboard, but I'm blocked by Pscx/Pscx#103 (comment), and #26832 (comment). Consequently, are you using Windows? If so, I'm surprised you can use tabs in filenames, considering that they're explicitly illegal in filenames created via the Win32 API (as all early UTF-8/ASCII whitespace characters are). Those are merely symbols duplicated into the clipboard - apologies.

Michael Klement (@mklement0)

Michael Klement (@mklement0), thanks for the advice, especially per PowerShell/PSReadLine#4072 (comment). Do you suggest that I merely duplicate the content of #24037 (comment) into the technical implementation section? I ask because I doubt that I am able to accurately rephrase it.

I suggest opening a new PSReadLine issue that succinctly describes the problem; feel free to adapt / copy the text from my comment in the process.

Michael Klement (@mklement0)

Michael Klement (@mklement0), this is a lot to get my head around. Summarily, should I request that PSReadLine not escape tab characters (unless at the start of a line)?

@RokeJulianLockhart, you could ask that PSReadLine paste tabs as-is, though given @daxian-dbw's comment above, I'm not sure such a change will be considered.

If it were to be considered, note that PowerShell/PSReadLine#1471 must be implemented first, because, without paste bracketing, pasting involves simulated typing, so that a tab char. as part of a pasted string would trigger tab-completion
(this is what's currently happening on macOS and Linux).

Michael Klement (@mklement0)

Michael Klement (@mklement0), thanks. Filed at PowerShell/PSReadLine#4447 (comment).

I may have a similar issue. I don't have the time to make a full issue out of it, but the gist is that when the working directory is inside a URN directory on my company network, pasting a move action into Powershell (which I have to do since I can't execute scripts) Powershell does not interpret the paths I've pasted as valid paths.

move '.\bar.ext' '.\foo\bar-rename.ext'

However, if I don't execute the script, I can arrow key to the first path, erase the extension, and hit tab to autocomplete the file, Powershell interprets this fine.

Good tip! Thanks!

Navigation Menu

Repository navigation

Pasted UTF-8 characters into PowerShell shell are interpreted incorrectly. #24037

Prerequisites

Expected behaviour

Steps to reproduce

Actual behavior

Error details

Environment data

Visuals

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Pasted UTF-8 characters into PowerShell shell are interpreted incorrectly. #24037

Description

Prerequisites

Expected behaviour

Steps to reproduce

Actual behavior

Error details

Environment data

Visuals

Activity

237dmitry commented on Jul 10, 2024

beedell-roke commented on Jul 10, 2024

237dmitry commented on Jul 10, 2024

beedell-roke commented on Jul 10, 2024

kasini3000 commented on Jul 10, 2024

beedell-roke commented on Jul 11, 2024

mklement0 commented on Jul 11, 2024

beedell-roke commented on Jul 11, 2024

mklement0 commented on Jul 11, 2024

237dmitry commented on Jul 11, 2024

daxian-dbw commented on Jul 11, 2024

mklement0 commented on Jul 11, 2024

daxian-dbw commented on Jul 12, 2024

mklement0 commented on Jul 12, 2024

FenixxAngel commented on Jul 13, 2024

beedell-roke commented on Jul 14, 2024

beedell-roke commented on Oct 22, 2024

Footnotes

beedell-roke commented on Oct 22, 2024

beedell-roke commented on Oct 22, 2024

mklement0 commented on Oct 22, 2024

beedell-roke commented on Feb 4, 2025

mklement0 commented on Feb 4, 2025

beedell-roke commented on Feb 4, 2025

bonneville-nathan commented on Nov 5, 2026

beedell-roke commented on Nov 5, 2026

bonneville-nathan commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions