[Edited by @daxian-dbw]

PR #3281 adds support to ShellExecute in powershell core when it's running on Windows full SKUs. With that fix, Start-Process is able to run MSI in powershell core.

However, it's not a complete solution. It calls ShellExecuteEx directly in MTA thread instead of from STA thread as suggested in MSDN. Therefore, the current solution may not work for some shell extensions that require COM.

In .NET Core, managed threads are all eagerly initialized with MTA mode, so to call ShellExecuteEx from a STA thread, we need to create a native thread using CreateThread function and initialize COM with STA on that thread. In #3281, we are calling ShellExecuteEx directly on MTA thread, and it works for things like opening a folder in File Explorer, opening a file with the application that is associated with its extension in shell, opening URL in web browser and etc, but it's not guaranteed to work in all ShellExecution scenarios.

Will keep this issue open to track the "invoke-on-STA-thread" work.

PS:4> Start-Process .\PowerShell_6.0.0-alpha.17-win10-win2016-x64.msi -PassThru

 NPM(K)    PM(M)      WS(M)     CPU(s)     Id  SI ProcessName
 ------    -----      -----     ------     --  -- -----------
      9     6.15       7.55       0.03  20092   3 msiexec

PS:5> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.0.0-alpha
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
PSEdition                      Core
GitCommitId                    v6.0.0-alpha.16-54-gc86a287726fe366895723dfcc7739f93038615ea
CLRVersion
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
SerializationVersion           1.1.0.1
BuildVersion                   3.0.0.0