Output custom object with headers when no values are present #25096

jshigetomi · 2025-02-26T16:30:35Z

PR Summary

This pull request includes changes to improve the handling of empty and null values in the ConvertFrom-Csv command and updates the corresponding unit tests to verify the new behavior. The most important changes include modifying the Import method in CsvCommands.cs and adding new unit tests in ConvertFrom-Csv.Tests.ps1.

Enhancements to ConvertFrom-Csv command:

src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs: Updated the Import method to skip empty lines only when using Import-CSV, but to keep them when using ConvertFromCSV.

Unit tests updates:

test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertFrom-Csv.Tests.ps1: Added new unit tests to verify that ConvertFrom-Csv handles empty and null values correctly.

PR Context

PR Checklist

…wershell into fixNoValueCSVInput

jshigetomi · 2025-02-26T21:07:26Z

I've adjusted the behavior of ConvertFrom-CSV according to the discussion on issue #24698. Essentially I removed the feature that skips blank lines in the Import method in ImportCsvHelper. It still keeps the last cell null so as to keep this from being a breaking change. #17702

surfingoldelephant · 2025-02-27T01:23:11Z

src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs

+                if (values.Count == 1 && string.IsNullOrEmpty(values[0]) && !(_cmdlet is ConvertFromCsvCommand))
                {
-                    // skip the blank lines
+                    // Skip empty lines when Import-CSV but keep empty lines when ConvertFromCSV


The bug (#24698) affects both ConvertFrom-Csv and Import-Csv; I think the fix should be applied to both commands for consistency. ConvertFrom-Csv was used throughout the original issue for demonstration purposes only.

Do we know if there's a specific reason why the check for values.Count == 1 && string.IsNullOrEmpty(values[0]) was included in the first place?

This comes from Windows PowerShell.

When I disable it for Import-CSV as well, `n's are captured as an extra element. Is this the behavior we want?

Describe "Import-Csv with different newlines" -Tags "CI" { It "Test import-csv with '<name>' newline" -TestCases @( @{ name = "CR"; newline = "`r" } @{ name = "LF"; newline = "`n" } @{ name = "CRLF"; newline = "`r`n" } ) { param($newline) $csvFile = Join-Path $TestDrive -ChildPath $((New-Guid).Guid) $delimiter = ',' "h1,h2,h3$($newline)11,12,13$($newline)21,22,23$($newline)" | Out-File -FilePath $csvFile $returnObject = Import-Csv -Path $csvFile -Delimiter $delimiter $returnObject.Count | Should -Be 2 $returnObject[0].h1 | Should -Be 11 $returnObject[0].h2 | Should -Be 12 $returnObject[0].h3 | Should -Be 13 $returnObject[1].h1 | Should -Be 21 $returnObject[1].h2 | Should -Be 22 $returnObject[1].h3 | Should -Be 23 } }

Is this the behavior we want?

No, it isn't.

This is more about ensuring whatever fix is chosen for #24698 is applied to both ConvertFrom-Csv and Import-Csv. In other words, the example CSV data below should deserialize consistently, irrespective of the source being a file or a string.

"P1" "" "" ""

Can #24698 be fixed for both commands without introducing the newline issue?

I'm thinking it can be deserialized consistently but would result in just one empty string value per header.
For example,

"P1" "" "" "" # Would produce [pscustomobject] @{ P1 = '' } ignoring the other empty cells

@' "P1","P2" "", "", '@ | ConvertFrom-Csv # Would produce [pscustomobject] @{ P1 = ''; P2 = '' } ignoring the other empty cells again

This way we can produce a header only output consistently.

Thoughts?

That's not what I had in mind when I filed the issue. For each non-empty line in input data, an object should be emitted by ConvertFrom-Csv/Import-Csv.

The following (using either command) produces 3 objects:

@' "P1" "a" "b" "c" '@ | ConvertFrom-Csv

Therefore, the following (using either command) should also produce 3 objects:

@' "P1" "" "" "" '@ | ConvertFrom-Csv

Likewise, this produces 2 objects, because the empty line is correctly ignored:

@' "P1" "a" "c" '@ | ConvertFrom-Csv

Therefore, this should also produce 2 objects:

@' "P1" "" "" '@ | ConvertFrom-Csv

iSazonov · 2025-02-28T19:32:35Z

@jshigetomi Root of the issue in ParseNextRecord method. It does not distinguish between an input string without characters and a string with two double quotes "".

PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs

Lines 1609 to 1613 in bfdc498

    
           result.Add(current.ToString()); 
        
           current.Remove(0, current.Length); 
        
           // New line outside quote is end of word and end of record 
        
           break;

Here current.ToString() returns string.Empty for these two cases. We can check current.Length > 0 before call ToString(). I suggest the fix (with removing

PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs

Lines 1390 to 1394 in bfdc498

    
           if (values.Count == 1 && string.IsNullOrEmpty(values[0]) && !(_cmdlet is ConvertFromCsvCommand)) 
        
           { 
        
               // Skip empty lines when Import-CSV but keep empty lines when ConvertFromCSV 
        
               continue; 
        
           }

)

But we get to Zugzwang - sometimes we want to, sometimes we don't. Of course, it would be possible to add more conditions, but this would require a more significant code change in same methods. My suggestion is not to complicate the code. If there are concerns, then make this change an experimental feature.

iSazonov · 2025-03-01T13:46:10Z

@jshigetomi I checked my guess. I had to make a few more changes. You can see the result here:

master...iSazonov:PowerShell:csvblank

jshigetomi · 2025-03-03T19:14:00Z

@iSazonov I'm seeing test failure for Import-CSV

Maybe a warning to the user would be better for no value csvs?

iSazonov · 2025-03-03T19:25:44Z

I don't see errors in my branch.

jshigetomi · 2025-03-03T22:15:48Z

Hmm I'm having some trouble reproducing your results.

master...iSazonov:PowerShell:csvblank#diff-2b77454e7d234e46e6ebdb6c1be4ac0cb9c02b88bbe0e61c9acdfa3f66491e81R1383-R1390

How are you resolving the brackets here?

iSazonov · 2025-03-04T12:33:43Z

@jshigetomi Please look #25120

jshigetomi · 2025-03-06T20:32:52Z

/azp run

azure-pipelines · 2025-03-06T20:33:07Z

Azure Pipelines successfully started running 2 pipeline(s).

jshigetomi · 2025-03-11T18:00:15Z

@surfingoldelephant @iSazonov I added the proposed tests but there seems to be a slight difference in behavior between ConvertFrom-CSV and Import-CSV. Import-CSV is not leaving the last cell $null but placing an empty string.

I think the test cases where the CSV is written to a file then read from the file using Import-CSV is causing the change in behavior. This might be simply a testing limitation. What do you think @iSazonov ?

surfingoldelephant · 2025-03-12T00:22:05Z

@jshigetomi I believe that's because the test file has an empty line at the end (default Set-Content behavior).

You should see the same result with ConvertFrom-Csv if its input also ends with an empty line.

# Import-Csv:
$file = [IO.Path]::GetTempFileName()
@'
"P1","P2"
,
a,
'@ | Set-Content -LiteralPath $file # Includes empty line at end
$result = Import-Csv -LiteralPath $file

$null -eq $result[1].P2 # False


# ConvertFrom-Csv:
$result = @'
"P1","P2"
,
a,

'@  | ConvertFrom-Csv

$null -eq $result[1].P2 # False

Note this is also the existing behavior (without this PR's fix).

If the same CSV data is used without an empty line at the end, both commands should deserialize the last cell as $null.

# Import-Csv:
$file = [IO.Path]::GetTempFileName()
@'
"P1","P2"
,
a,
'@ | Set-Content -LiteralPath $file -NoNewline # No empty line at end
$result = Import-Csv -LiteralPath $file

$null -eq $result[1].P2 # True


# ConvertFrom-Csv:
$result = @'
"P1","P2"
,
a,
'@  | ConvertFrom-Csv

$null -eq $result[1].P2 # True

iSazonov · 2025-03-12T03:19:42Z

@jshigetomi I believe that's because the test file has an empty line at the end (default Set-Content behavior).

I think so too. It would be nice to test both cases with last empty line and without it. We could directly use .Net classes to create such files.

One more test I think for Import-Csv if there is a source file with header only.

surfingoldelephant · 2025-03-12T03:26:19Z

We could directly use .Net classes to create such files.

Providing Set-Content's input is a single, multi-line string (which is the case with all the tests currently), -NoNewLine also avoids a trailing empty line.

See the Set-Content -LiteralPath $file -NoNewline example in my comment above.

jshigetomi · 2025-03-27T18:50:05Z

/azp run

azure-pipelines · 2025-03-27T18:50:17Z

Azure Pipelines successfully started running 1 pipeline(s).

jshigetomi · 2025-03-27T20:26:52Z

@surfingoldelephant @iSazonov Thanks for the pointer for Set-Content. I was not aware of the newline behavior.

I made the ConvertFrom-CSV and Import-CSV tests the same and both behave the same.

iSazonov · 2025-03-28T03:22:28Z

test/powershell/Modules/Microsoft.PowerShell.Utility/Import-Csv.Tests.ps1

+
+Describe "Import-Csv with empty and null values" {
+
+    Context 'Empty CSV Fields' {


Since the PR is about edge cases I think about additional tests:

only header presents without newline

header presents with one empty newline

And more:

empty file (-Header parameter presents)

file with one empty line (-Header parameter presents)

Just to be clear, we're expecting those tests to produce no output (zero custom objects).

Let's also include Import-Csv/ConvertFrom-Csv tests with multiple empty lines.

Header only

@' P1,P2 '@ | ConvertFrom-Csv

Header followed by one empty line

@' P1,P2 '@ | ConvertFrom-Csv

Header followed by multiple empty lines

@' P1,P2 '@ | ConvertFrom-Csv

Empty input and -Header specified

'' | ConvertFrom-Csv -Header P1, P2

One empty line and -Header specified

@' '@ | ConvertFrom-Csv -Header P1, P2

Multiple empty lines and -Header specified

@' '@ | ConvertFrom-Csv -Header P1, P2

Perhaps additional -Header tests to cover the fix:

@' , A1,A2 B1,B2 , '@ | ConvertFrom-Csv -Header P1, P2 # Expected: [pscustomobject] @{ P1 = ''; P2 = '' } [pscustomobject] @{ P1 = 'A1'; P2 = 'A2' } [pscustomobject] @{ P1 = 'B1'; P2 = 'B2' } [pscustomobject] @{ P1 = ''; P2 = $null }

surfingoldelephant

I think we also need to take a look at no detected delimiter scenarios.

# Detected delimiter:

@'
P1,P2,P3
A1,,
B1,,
C1,,
'@ | ConvertFrom-Csv

# Current/expected behavior:
[pscustomobject] @{ P1 = 'A1'; P2 = ''; P3 = '' }
[pscustomobject] @{ P1 = 'B1'; P2 = ''; P3 = '' }
[pscustomobject] @{ P1 = 'C1'; P2 = ''; P3 = $null }

# No detected delimiter:

@'
P1,P2,P3
A1
B1
C1
'@ | ConvertFrom-Csv

# Current/expected behavior:
[pscustomobject] @{ P1 = 'A1'; P2 = $null; P3 = $null }
[pscustomobject] @{ P1 = 'B1'; P2 = $null; P3 = $null }
[pscustomobject] @{ P1 = 'C1'; P2 = $null; P3 = $null }


@'
P1;P2;P3
A1
B1
C1
'@ | ConvertFrom-Csv -Delimiter ';'

# Current/expected behavior:
[pscustomobject] @{ P1 = 'A1'; P2 = $null; P3 = $null }
[pscustomobject] @{ P1 = 'B1'; P2 = $null; P3 = $null }
[pscustomobject] @{ P1 = 'C1'; P2 = $null; P3 = $null }

Notice when there is no (detected) delimiter, the resultant value is always $null. Can we add tests for this?

…wershell into fixNoValueCSVInput

jshigetomi · 2025-08-27T19:44:29Z

/azp run

azure-pipelines · 2025-08-27T19:44:41Z

Azure Pipelines successfully started running 1 pipeline(s).

jshigetomi · 2025-09-29T18:51:06Z

/azp run PowerShell-Windows-Packaging-CI, PowerShell-CI-linux-packaging

azure-pipelines · 2025-09-29T18:51:14Z

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

Justin Chung and others added 9 commits February 26, 2025 10:26

Output custom object with headers when no values are present

df47ac8

Merge branch 'master' into fixNoValueCSVInput

4ccc9ad

Remove if statement that skips blank lines

ba25fa7

Merge branch 'master' into fixNoValueCSVInput

7ef2c8b

Only remove the if statement for ConvertFrom-CSV

83a53ef

Merge branch 'fixNoValueCSVInput' of https://github.com/powershell/po…

b7f491d

…wershell into fixNoValueCSVInput

Add test for testing null values and empty string values

3786536

Add comment about skiping lines

50db112

Delete trailing whitespace

c4950da

PowerShell deleted a comment from azure-pipelines bot Feb 26, 2025

jshigetomi mentioned this pull request Feb 26, 2025

CSV deserialization outputs nothing if there are only a certain number of empty fields #24698

Open

5 tasks

Add more test cases for empty and null values

472c6d0

PowerShell deleted a comment from azure-pipelines bot Feb 26, 2025

Fix test to make it pester 4.10 compatible

bfdc498

surfingoldelephant reviewed Feb 27, 2025

View reviewed changes

Add iSazonov's changes

64e6de8

Justin Chung added 2 commits March 6, 2025 11:33

Remove skip no values check and add skip blank check

8dd4a66

Revert to private feed

6c2c6c7

PowerShell deleted a comment from azure-pipelines bot Mar 6, 2025

microsoft-github-policy-service bot added the Review - Needed The PR is being reviewed label Mar 19, 2025

jshigetomi and others added 2 commits March 27, 2025 13:41

Merge branch 'master' into fixNoValueCSVInput

e1729e6

Add NoNewLine to test and revert test cases to be the same

0ea9e53

jshigetomi requested review from iSazonov and surfingoldelephant March 27, 2025 20:25

iSazonov reviewed Mar 28, 2025

View reviewed changes

surfingoldelephant reviewed Mar 31, 2025

View reviewed changes

Added tests for edge cases and updated old test for Context and typo

6e3fa23

jshigetomi requested review from iSazonov and surfingoldelephant August 4, 2025 18:32

jshigetomi added 6 commits August 4, 2025 14:20

adjust import-csv tests

de52856

Adjust ConvertFrom csv tests

87c5def

fix header param test

1ba4e61

Merge branch 'master' into fixNoValueCSVInput

7a24009

Clean up comments

effd7b2

Merge branch 'fixNoValueCSVInput' of https://github.com/powershell/po…

5bd8b40

…wershell into fixNoValueCSVInput

Merge branch 'master' into fixNoValueCSVInput

db0bbea

jshigetomi added 2 commits September 2, 2025 11:49

Merge branch 'master' into fixNoValueCSVInput

f693967

Merge branch 'master' into fixNoValueCSVInput

0b22c87


		Describe "Import-Csv with empty and null values" {

		Context 'Empty CSV Fields' {

Output custom object with headers when no values are present #25096

Are you sure you want to change the base?

Output custom object with headers when no values are present #25096

Uh oh!

Conversation

jshigetomi commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Context

PR Checklist

Uh oh!

jshigetomi commented Feb 26, 2025

Uh oh!

surfingoldelephant Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

iSazonov Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

jshigetomi Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

surfingoldelephant Feb 28, 2025

Choose a reason for hiding this comment

Uh oh!

jshigetomi Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surfingoldelephant Feb 28, 2025

Choose a reason for hiding this comment

Uh oh!

iSazonov commented Feb 28, 2025

Uh oh!

iSazonov commented Mar 1, 2025

Uh oh!

jshigetomi commented Mar 3, 2025

Uh oh!

iSazonov commented Mar 3, 2025

Uh oh!

jshigetomi commented Mar 3, 2025

Uh oh!

iSazonov commented Mar 4, 2025

Uh oh!

jshigetomi commented Mar 6, 2025

Uh oh!

azure-pipelines bot commented Mar 6, 2025

Uh oh!

jshigetomi commented Mar 11, 2025

Uh oh!

surfingoldelephant commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iSazonov commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

surfingoldelephant commented Mar 12, 2025

Uh oh!

jshigetomi commented Mar 27, 2025

Uh oh!

azure-pipelines bot commented Mar 27, 2025

Uh oh!

jshigetomi commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iSazonov Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

surfingoldelephant Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surfingoldelephant left a comment

Choose a reason for hiding this comment

Uh oh!

jshigetomi commented Aug 27, 2025

Uh oh!

azure-pipelines bot commented Aug 27, 2025

Uh oh!

jshigetomi commented Sep 29, 2025

jshigetomi commented Feb 26, 2025 •

edited

Loading

jshigetomi Feb 28, 2025 •

edited

Loading

surfingoldelephant commented Mar 12, 2025 •

edited

Loading

iSazonov commented Mar 12, 2025 •

edited

Loading

jshigetomi commented Mar 27, 2025 •

edited

Loading

surfingoldelephant Mar 31, 2025 •

edited

Loading