Skip to content

ConvertTo-Json: add parameter AsStream, to implement "streaming" mode - #25597

Open
Paul Hilchey (paulhilchey) wants to merge 5 commits into
PowerShell:masterfrom
paulhilchey:converttojson-asstream
Open

Paul Hilchey (paulhilchey) wants to merge 5 commits into
PowerShell:masterfrom
paulhilchey:converttojson-asstream

Conversation

@paulhilchey

@paulhilchey Paul Hilchey (paulhilchey) commented May 23, 2025 •

Copy link
Copy Markdown

PR Summary

This pull request adds the AsStream parameter to the ConvertTo-Json cmdlet.
When enabled, each input object is converted individually to a Json string.
There will be one output string for each input object.

PR Context

Currently, the ConvertTo-Json cmdlet assembles all input objects into an in-memory
List, then converts that to one Json string for output.
This can be inefficient if there are many input objects.

With the AsStream parameter, each input object is converted to a complete Json string and output
as it is processed. This reduces the memory usage. It can also be substantially
improve throughput when the output stream is being processed by a separate thread
or process.

Note that the ConvertFrom-Json cmdlet will accept a stream of complete Json strings as input and convert them
back to objects as would be expected. The two cmdlets work symetrically in that respect.

The AsStream parameter is analagous to the "-As Stream" feature in the ConvertTo-Xml cmdlet.

PR Checklist

@paulhilchey

Paul Hilchey (paulhilchey) commented May 23, 2025 •

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree company="University of Victoria"

I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@paulhilchey

This comment was marked as duplicate.

@paulhilchey

Copy link
Copy Markdown
Author

Draft for documentation changes are in MicrosoftDocs/PowerShell-Docs#12101

/// the array symbols will occur only if there is more than one input object.
/// </summary>
[Parameter]
[Parameter(ParameterSetName = "Array")]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't this still be used with -AsStream, e.g.

"foo", "bar" | ConvertTo-Json -AsArray -AsStream
# ["foo"]
# ["bar"]

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could. Alternatively, the -AsArray -AsStream combo could output a single Json array, but output as multiple strings such that the data would not all be buffered in memory.

I'm undecided which would be best.

/// If the AsStream property is set to be true, the input objects are converted
/// to JSON individually. There will be one output string for each input object.
/// </summary>
[Parameter(ParameterSetName = "Stream")]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure a new parameter set is actually needed here, the switch itself is what is signalling whether to stream or not. I don't think that -AsArray and -AsStream are mutually exclusive as well as you can stream JSON arrays.

One thing to keep in mind about this pattern, if someone was to set the switch to false it will still go through the parameter set locking you into whatever parameters are exposed in there.

If -AsArray was mutually exclusive (just a hypothetical) then doing "foo" | ConvertTo-Json -AsArray -AsStream:$AsStream would fail if $AsStream was false. You would have to conditionally add the switch which is not ideal.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used the parameter sets to disallow the -AsArray -AsStream combo.
That way it will not be a breaking change when it is decided to make the combo do something.

@microsoft-github-policy-service microsoft-github-policy-service Bot added the Review - Needed The PR is being reviewed label Jun 6, 2025
@iSazonov

Copy link
Copy Markdown
Collaborator

Paul Hilchey (@paulhilchey) Our process requires to have issue approved by Work Group. Please create such issue with a request of the new feature.

@microsoft-github-policy-service microsoft-github-policy-service Bot removed the Review - Needed The PR is being reviewed label Nov 22, 2025
@microsoft-github-policy-service

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Maintainer, please provide feedback and/or mark it as Waiting on Author

@adityapatwardhan

Copy link
Copy Markdown
Member

PowerShell Maintainer's discussed this PR and would love to see -AsStream parameter for Convert-FromJson as well.

@iSazonov Ilya (iSazonov) added the CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log label Dec 4, 2025
@mklement0

Copy link
Copy Markdown
Contributor

Aditya Patwardhan (@adityapatwardhan), I'm glad to hear it, but I suggest calling the parameter -Stream, not -AsStream, following the pattern of Out-String -Stream (no other built-in cmdlets have a similar switch; while ConvertTo-Xml has an -As switch with a Stream argument, I think following Out-String's pattern is preferable, not least because ConvertTo-Xml is little-known and -used).

Copilot AI review requested due to automatic review settings June 21, 2026 05:11
@paulhilchey
Paul Hilchey (paulhilchey) requested a review from a team as a code owner June 21, 2026 05:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request introduces an -AsStream switch to ConvertTo-Json to support “streaming” behavior: each pipeline input object is serialized and written as its own JSON string, instead of buffering all inputs and emitting a single JSON payload at the end.

Changes:

  • Added -AsStream parameter and streaming behavior in ConvertToJsonCommand.ProcessRecord().
  • Refactored JSON conversion context creation to occur once in BeginProcessing() and reused for both streaming and non-streaming modes.
  • Added Pester coverage for streaming output shape and StopProcessing behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommand.cs Adds -AsStream and updates execution flow/context initialization to support per-record JSON output.
test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertTo-Json.Tests.ps1 Adds tests validating streaming output and a StopProcessing scenario for streaming mode.

/// This command converts an object to a Json string representation.
/// </summary>
[Cmdlet(VerbsData.ConvertTo, "Json", HelpUri = "https://go.microsoft.com/fwlink/?LinkID=2096925", RemotingCapability = RemotingCapability.None)]
[Cmdlet(VerbsData.ConvertTo, "Json", HelpUri = "https://go.microsoft.com/fwlink/?LinkID=2096925", RemotingCapability = RemotingCapability.None, DefaultParameterSetName = "Default")]
Comment on lines +185 to +203
$ps = [PowerShell]::Create()
$dataCollection = [System.Management.Automation.PSDataCollection[PSObject]]::new()
$null = $ps.AddScript({
$obj1 = [PSCustomObject]@{Prop = 'value1'}
$obj2 = [PSCustomObject]@{} | Add-Member -MemberType ScriptProperty -Name Prop -Value {
Start-Sleep -Seconds 5
'value2'
} -PassThru
$obj1,$obj2 | ConvertTo-Json -AsStream
throw "Should not have thrown exception"
})
$null = $ps.BeginInvoke($dataCollection,$dataCollection)
Start-Sleep -Milliseconds 1000
$null = $ps.BeginStop($null, $null)
Start-Sleep -Milliseconds 1000
$dataCollection.Count | Should -BeExactly 1
$ps.InvocationStateInfo.State | Should -BeExactly "Stopped"
$ps.Dispose()
}
Comment on lines +169 to +171
$returnObject[0] | Should -Be '{"Name":"Object1","Value":10}'
$returnObject[1] | Should -Be '{"Name":"Object2","Value":20}'
$returnObject[2] | Should -Be '{"Name":"Object3","Value":30}'
@jshigetomi

Justin Chung (jshigetomi) commented Jun 26, 2026 •

Copy link
Copy Markdown
Collaborator

Paul Hilchey (@paulhilchey) Thanks for taking this on! Looks good. Could you please address copilot's review comments?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log Review - Needed The PR is being reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants