Repository navigation
ConvertTo-Json: add parameter AsStream, to implement "streaming" mode - #25597
Paul Hilchey (paulhilchey) wants to merge 5 commits into
Conversation
with corresponding test cases.
|
@microsoft-github-policy-service agree company="University of Victoria" I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer. |
This comment was marked as duplicate.
This comment was marked as duplicate.
|
Draft for documentation changes are in MicrosoftDocs/PowerShell-Docs#12101 |
| /// the array symbols will occur only if there is more than one input object. | ||
| /// </summary> | ||
| [Parameter] | ||
| [Parameter(ParameterSetName = "Array")] |
There was a problem hiding this comment.
Couldn't this still be used with -AsStream, e.g.
"foo", "bar" | ConvertTo-Json -AsArray -AsStream
# ["foo"]
# ["bar"]There was a problem hiding this comment.
It could. Alternatively, the -AsArray -AsStream combo could output a single Json array, but output as multiple strings such that the data would not all be buffered in memory.
I'm undecided which would be best.
| /// If the AsStream property is set to be true, the input objects are converted | ||
| /// to JSON individually. There will be one output string for each input object. | ||
| /// </summary> | ||
| [Parameter(ParameterSetName = "Stream")] |
There was a problem hiding this comment.
I'm not sure a new parameter set is actually needed here, the switch itself is what is signalling whether to stream or not. I don't think that -AsArray and -AsStream are mutually exclusive as well as you can stream JSON arrays.
One thing to keep in mind about this pattern, if someone was to set the switch to false it will still go through the parameter set locking you into whatever parameters are exposed in there.
If -AsArray was mutually exclusive (just a hypothetical) then doing "foo" | ConvertTo-Json -AsArray -AsStream:$AsStream would fail if $AsStream was false. You would have to conditionally add the switch which is not ideal.
There was a problem hiding this comment.
I used the parameter sets to disallow the -AsArray -AsStream combo.
That way it will not be a breaking change when it is decided to make the combo do something.
|
Paul Hilchey (@paulhilchey) Our process requires to have issue approved by Work Group. Please create such issue with a request of the new feature. |
|
This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days. |
|
PowerShell Maintainer's discussed this PR and would love to see |
|
Aditya Patwardhan (@adityapatwardhan), I'm glad to hear it, but I suggest calling the parameter |
There was a problem hiding this comment.
Pull request overview
This pull request introduces an -AsStream switch to ConvertTo-Json to support “streaming” behavior: each pipeline input object is serialized and written as its own JSON string, instead of buffering all inputs and emitting a single JSON payload at the end.
Changes:
- Added
-AsStreamparameter and streaming behavior inConvertToJsonCommand.ProcessRecord(). - Refactored JSON conversion context creation to occur once in
BeginProcessing()and reused for both streaming and non-streaming modes. - Added Pester coverage for streaming output shape and StopProcessing behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommand.cs |
Adds -AsStream and updates execution flow/context initialization to support per-record JSON output. |
test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertTo-Json.Tests.ps1 |
Adds tests validating streaming output and a StopProcessing scenario for streaming mode. |
| /// This command converts an object to a Json string representation. | ||
| /// </summary> | ||
| [Cmdlet(VerbsData.ConvertTo, "Json", HelpUri = "https://go.microsoft.com/fwlink/?LinkID=2096925", RemotingCapability = RemotingCapability.None)] | ||
| [Cmdlet(VerbsData.ConvertTo, "Json", HelpUri = "https://go.microsoft.com/fwlink/?LinkID=2096925", RemotingCapability = RemotingCapability.None, DefaultParameterSetName = "Default")] |
| $ps = [PowerShell]::Create() | ||
| $dataCollection = [System.Management.Automation.PSDataCollection[PSObject]]::new() | ||
| $null = $ps.AddScript({ | ||
| $obj1 = [PSCustomObject]@{Prop = 'value1'} | ||
| $obj2 = [PSCustomObject]@{} | Add-Member -MemberType ScriptProperty -Name Prop -Value { | ||
| Start-Sleep -Seconds 5 | ||
| 'value2' | ||
| } -PassThru | ||
| $obj1,$obj2 | ConvertTo-Json -AsStream | ||
| throw "Should not have thrown exception" | ||
| }) | ||
| $null = $ps.BeginInvoke($dataCollection,$dataCollection) | ||
| Start-Sleep -Milliseconds 1000 | ||
| $null = $ps.BeginStop($null, $null) | ||
| Start-Sleep -Milliseconds 1000 | ||
| $dataCollection.Count | Should -BeExactly 1 | ||
| $ps.InvocationStateInfo.State | Should -BeExactly "Stopped" | ||
| $ps.Dispose() | ||
| } |
| $returnObject[0] | Should -Be '{"Name":"Object1","Value":10}' | ||
| $returnObject[1] | Should -Be '{"Name":"Object2","Value":20}' | ||
| $returnObject[2] | Should -Be '{"Name":"Object3","Value":30}' |
|
Paul Hilchey (@paulhilchey) Thanks for taking this on! Looks good. Could you please address copilot's review comments? |
PR Summary
This pull request adds the AsStream parameter to the ConvertTo-Json cmdlet.
When enabled, each input object is converted individually to a Json string.
There will be one output string for each input object.
PR Context
Currently, the ConvertTo-Json cmdlet assembles all input objects into an in-memory
List, then converts that to one Json string for output.
This can be inefficient if there are many input objects.
With the AsStream parameter, each input object is converted to a complete Json string and output
as it is processed. This reduces the memory usage. It can also be substantially
improve throughput when the output stream is being processed by a separate thread
or process.
Note that the ConvertFrom-Json cmdlet will accept a stream of complete Json strings as input and convert them
back to objects as would be expected. The two cmdlets work symetrically in that respect.
The AsStream parameter is analagous to the "-As Stream" feature in the ConvertTo-Xml cmdlet.
PR Checklist
.h,.cpp,.cs,.ps1and.psm1files have the correct copyright header