Request for help: how to use DotNet HttpClient to async upload large files using multipart/form-data within PowerShell #13721
Description
I was asked in #4129 to open a new issue to discuss my request for help. I originally posted this question on Stackoverflow, and have had no responses. I'm not sure if this is a question for the PowerShell or DotNetCore team, but I'm starting here.
I am in the process of working on updating some PowerShell code that uses System.Net.HttpWebRequest to handle file transfers to and from a web service. The System.IO.FileStream class has a major problem with handling file streams being transferred to a web service using HttpWebRequest and the object (file) is greater than 2GB. It was stated in #4129 this was a breaking change, and will not be fixed by the .NetCore team, and ask the question here. Essentially, one runs into the following exception in PowerShellCore/PowerShell7:
MethodInvocationException: C:\Users\clynch\Documents\WindowsPowerShell\modules\modulename\modulename.psm1:35180
Line |
35180 | $rs.write($readbuffer, 0, $bytesRead)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Exception calling "Write" with "3" argument(s): "Stream was too long."I have the following verified and working PowerShell code I am trying to convert to C# async class method using HttpClient to asynchronously upload files to an Apache/Spring-based REST API service when using PowerShell 5.x, or within PowerShell7/PowerShellCore only when the file is less than 2GB in size:
$Hostname = "somewebserver"
$Method = "POST"
$File = Get-ChildItem C:\Directory\file.iso
$FSOpenMode = [System.IO.FileMode]::Open
$FSRead = [System.IO.FileAccess]::Read
# Read file using FileStream
$fs = [IO.FileStream]::new($File.FullName, $FSOpenMode, $FSRead)
[void]$fs.FlushAsync()
try
{
# Set the URL that will be the upload destination
$url = "https://{0}/upload?uploadfilename={1}" -f $Hostname, $File.Name
$_DispositionContentType = "application/octet-stream"
[System.Net.httpWebRequest]$uploadRequest = [System.Net.WebRequest]::Create($uri)
$uploadRequest.Method = $Method
$boundary = "---------------------------" + [DateTime]::Now.Ticks.ToString("x")
[byte[]]$BoundaryBytes = [System.Text.Encoding]::UTF8.GetBytes("`r`n--" + $boundary + "`r`n");
$disposition = "Content-Disposition: form-data; name=`"file`"; filename=`"{0}`";`r`nContent-Type: {1}`r`n`r`n" -f $File.Name, $_DispositionContentType
[byte[]]$ContentDispBytes = [System.Text.Encoding]::UTF8.GetBytes($disposition);
[byte[]]$EndBoundaryBytes = [System.Text.Encoding]::UTF8.GetBytes("`r`n--" + $boundary + "--`r`n")
$uploadRequest.Timeout = 1200000
$uploadRequest.ContentType = "multipart/form-data; boundary={0}" -f $boundary
$uploadRequest.Headers.Item("auth") = "SessionID"
$uploadRequest.Headers.Item("uploadfilename") = $File.Name
$uploadRequest.AllowWriteStreamBuffering = $true
$uploadRequest.SendChunked = $true
$uploadRequest.ContentLength = $BoundaryBytes.length + $ContentDispBytes.length + $File.Length + $EndBoundaryBytes.Length
$uploadRequest.Headers.Item("ContentLength") = $BoundaryBytes.length + $ContentDispBytes.length + $File.Length + $EndBoundaryBytes.Length
$rs = $uploadRequest.GetRequestStream()
[void]$rs.FlushAsync()
[byte[]]$readbuffer = [byte[]]::new(4096 * 1024)
$rs.write($BoundaryBytes, 0, $BoundaryBytes.Length);
$rs.write($ContentDispBytes, 0, $ContentDispBytes.Length);
# This is used to keep track of the file upload progress.
$numBytesToRead = $fs.Length
[int64]$numBytesRead = 0
$_sw = [System.Diagnostics.Stopwatch]::StartNew()
$_progresssw = [System.Diagnostics.Stopwatch]::StartNew()
while ($bytesRead = $fs.Read($readbuffer, 0, $readbuffer.length))
{
[void]$fs.Flush()
$rs.write($readbuffer, 0, $bytesRead)
[void]$fs.Flush()
[void]$rs.Flush()
# Keep track of where we are at clearduring the read operation
$_numBytesRead += $bytesRead
# Flush the buffer every 200ms and 1MB written
if ($_progresssw.Elapsed.TotalMilliseconds -ge 200 -and $_numBytesRead % 100mb -eq 0)
{
[void]$rs.flush()
}
# Use the Write-Progress cmd-let to show the progress of uploading the file.
[Int]$_percent = [math]::floor(($_numBytesRead / $fs.Length) * 100)
# Elapsed time to calculat throughput
[Int]$_elapsed = $_sw.ElapsedMilliseconds / 1000
if ($_elapsed -ne 0 )
{
[single]$_transferrate = [Math]::Round(($_numBytesRead / $_elapsed) / 1mb)
}
else
{
[single]$_transferrate = 0.0
}
$status = "({0:0}MB of {1:0}MB transferred @ {2}MB/s) Completed {3}%" -f ($_numBytesRead / 1MB), ($numBytesToRead / 1MB), $_transferrate, $_percent
# Handle how poorly Write-Progress adds latency to the file transfer process by only refreshing the progress at a specifc interval
if ($_progresssw.Elapsed.TotalMilliseconds -ge 500)
{
if ($_numBytesRead % 1mb -eq 0)
{
Write-Progress -activity "Upload File" -status ("Uploading '{0}'" -f $File.Name) -CurrentOperation $status -PercentComplete $_percent
}
}
}
$fs.close()
# Write the endboundary to the file's binary upload stream
$rs.write($EndBoundaryBytes, 0, $EndBoundaryBytes.Length)
$rs.close()
$_sw.stop()
$_sw.Reset()
Write-Progress -activity "Upload File" -status ("Uploading '{0}'" -f $File.Name) -Complete
}
catch [System.Exception]
{
if ($fs)
{
$fs.close()
}
if ($_sw.IsRunning)
{
$_sw.Stop()
$_sw.Reset()
}
# Dispose if still exist
if ($rs)
{
$rs.close()
}
Throw $_
}
try
{
# Indicate we are waiting for the API to process the uploaded file in the write progress display
Write-Progress -activity "Upload File" -status ("Uploading '{0}'" -f $File.Name) -CurrentOperation "Waiting for completion response from appliance." -percentComplete $_percent
# Get the response from the API on the progress of identifying the file
[Net.httpWebResponse]$WebResponse = $uploadRequest.getResponse()
$uploadResponseStream = $WebResponse.GetResponseStream()
# Read the response & convert to JSON
$reader = [System.IO.StreamReader]::new($uploadResponseStream)
$responseJson = $reader.ReadToEnd()
$uploadResponse = ConvertFrom-Json $responseJson
$uploadResponseStream.Close()
$uploadRequest = $Null
# Finalize write progress display
Write-Progress -activity "Upload File" -CurrentOperation "Uploading $Filename " -Completed
}
catch [Net.WebException]
{
if ($null -ne $_.Exception.Response)
{
Try
{
# Need to see if Response is not empty
$sr = [IO.StreamReader]::new($_.Exception.Response.GetResponseStream())
}
Catch
{
$PSCmdlet.ThrowTerminatingError($_)
}
$errorObject = $sr.readtoEnd() | ConvertFrom-Json
# dispose if still exist
if ($rs)
{
$rs.close()
}
if ($fs)
{
$fs.close()
}
$sr.close()
# This New-ErrorRecord is an internal function that generates a new System.Management.Automation.ErrorRecord object to then throw
$ErrorRecord = New-ErrorRecord HPEOneview.Appliance.UploadFileException $errorObject.ErrorCode InvalidResult 'Upload-File' -Message $errorObject.Message -InnerException $_.Exception
Throw $ErrorRecord
}
else
{
Throw $_
}
}The following working C# code is the result of some research I have done to try to understand how HttpClient works with multipart/form-data uploads:
public class HttpClientUpload : IDisposable
{
private readonly string _uploadUrl;
private readonly string _sourceFilePath;
private readonly string _authToken;
// THIS IS ONLY HERE FOR TESTING
private static HttpClientHandler _handler = new HttpClientHandler() { ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => { return true; } };
private HttpClient _httpClient = new HttpClient(_handler)
{
Timeout = TimeSpan.FromDays(1)
};
public delegate void ProgressChangedHandler(long? totalFileSize, long totalBytesDownloaded, double? progressPercentage);
public event ProgressChangedHandler ProgressChanged;
public HttpClientUpload(string uploadUrl, string sourceFilePath, string authToken)
{
_uploadUrl = uploadUrl;
_sourceFilePath = sourceFilePath;
_authToken = authToken;
}
// Borrowed code from https://stackoverflow.com/questions/16416601/c-sharp-httpclient-4-5-multipart-form-data-upload
public void UploadAsync()
{
var Path = _sourceFilePath;
var Appliance = _uploadUrl;
var AuthToken = _authToken;
var m_CancellationSource = new CancellationTokenSource();
var token = m_CancellationSource.Token;
var fileInfo = new System.IO.FileInfo(Path);
var uri = $"https://{Appliance}/rest/firmware-bundles?uploadfilename={fileInfo.Name}";
// If the file extension is CRL, we need to use a different disposition/mime type declaration.
var DispositionContentType = fileInfo.Extension == "crl" ? "application/pkix-crl" : "application/octet-stream";
_httpClient.DefaultRequestHeaders.Add("User-Agent", $"Custom User Agent ({Environment.OSVersion.ToString()})");
_httpClient.DefaultRequestHeaders.Add("X-API-Version", "1800");
_httpClient.DefaultRequestHeaders.Add("auth", AuthToken);
_httpClient.DefaultRequestHeaders.Add("uploadfilename", fileInfo.Name);
_httpClient.DefaultRequestHeaders.Add("accept-language", "en_US");
_httpClient.DefaultRequestHeaders.Add("accept-encoding", "gzip, deflate");
using (var content = new MultipartFormDataContent("---------------------------" + DateTime.Now.Ticks.ToString("x")))
{
content.Headers.Add("Content-Disposition", $"form-data; name=\"file\"; filename=\"{fileInfo.Name}\"");
content.Headers.ContentLength = fileInfo.Length;
FileStream fs = File.OpenRead(Path);
var streamContent = new StreamContent(fs);
streamContent.Headers.Add("Content-Type", $"{DispositionContentType}");
streamContent.Headers.Add("Content-Disposition", $"form-data; name=\"file\"; filename=\"{fileInfo.Name}\"");
content.Add(streamContent, "file", fileInfo.FullName);
Task<HttpResponseMessage> message = _httpClient.PostAsync(uri, content);
var input = message.Result.Content.ReadAsStringAsync();
Console.WriteLine(input.Result);
Console.WriteLine("Press enter to continue...");
Console.Read();
}
}
public void Stop()
{
_httpClient.CancelPendingRequests();
throw new OperationCanceledException("File upload cancelled.");
}
public void Dispose()
{
_httpClient?.Dispose();
}
}The code above can be invoked from PowerShell as such, using a compiled DotNet class library that is imported into the users PowerShell runspace or using Add-Type:
$Hostname = "somewebserver"
$AuthToken = "SessionID"
$FileName = 'C:\directory\somefile.iso'
$Task = ([HttpClientUpload]::new($Hostname, $FileName, $AuthToken)).UploadAsync()The file appears to be transferred to the target, but the API endpoint generates an error. Not an HTTP error, but a JSON response indicating that this webservice couldn't parse the file. I get this in our API application log file:
caught apache common fileuploadexception ex=Processing of multipart/form-data request failed. Stream ended unexpectedly while uploading file somefile.iso
I know that this is caused by the fact that the request from the client must append a byte array of an endboundary. Because without it, I would get the very same message with the original PowerShell code I currently use if I omit the following line(s):
# Write the endboundary to the file's binary upload stream
$rs.write($EndBoundaryBytes, 0, $EndBoundaryBytes.Length)So, the TL;DR to my post. My questions are:
- How does one ensure that a multipart/form-data request that needs to include a binary stream of a file contains the start and end boundary for the
HttpClientrequest? I have found no Microsoft documentation that outlines the requirements to properly construct anHttpClientobject with the necessary IO stream to send this multipart/form-data request to a web service. - Any guidance on how to then chunk the file stream in order to report buffer transfer progress? I intend to read this from PowerShell after the async task has begun, in order to use Write-Progress back to the user.
PowerShell environments:
# Windows 10
Name Value
---- -----
PSVersion 7.0.3
PSEdition Core
GitCommitId 7.0.3
OS Microsoft Windows 10.0.19041
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0# Ubuntu 18.04 WSL2 container
PS /mnt/c/Users/clynch> $PSVersiontable
Name Value
---- -----
PSVersion 7.0.1
PSEdition Core
GitCommitId 7.0.1
OS Linux 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58…
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0