@bergmeister can you push a commit with [Feature] at this beginning on it. This will cause the CI to run like our nightly build which runs more tests. I want to validate that none of those extra tests depend on another test being run from the tests you have split out.

OK, it seems, when using [feature], there are only around 800 tests in the 2nd build matrix but there should be nearly 2000, I need to look into that, maybe I have missed one condition in the test function. But otherwise it's fine

@bergmeister thanks for investigating the results

@TravisEz13 I found the reason for the missing unelevated Pester feature tests and I fixed it. I consider the PR ready for review again now, I just pushed the last simplification commit.
One important question that might be a blocker: In Invoke-AppVeyorAfterTest, code coverage results get uploaded for daily/feature builds. Is this a problem now with tests (and results) being spread across 2 build jobs? Maybe in the case of an actual daily build (i.e. not a feature commit, the Test-DailyBuild function does not disambiguate them) we need to run all tests in one job then.

@bergmeister The last feature build was still 678 tests short... I'm trying to take a quick look why...
Feature build with 7975 tests
Your feature build with 7296 test

With the Feature tag, the elevated tests should be 633, while only 114 were run with your PR.

Update:
Looks like one of the machines in the matrix made the wrong decision about Test-DailyBuild.

Update-2:
My best guess is that AppVeyor is only to populating $env:APPVEYOR_REPO_COMMIT_MESSAGE on the first machine in the matrix.

@TravisEz13 I pushed another [feature] commit and this time the build results here were fine (i.e. had roughly the expected number of tests with 7974).
Your analysis about $env:APPVEYOR_REPO_COMMIT_MESSAGE is probably correct but I can only speculate why it was not set in the 2nd build job of the build that you referenced. Maybe it was because it was triggered by a push that included 3 commits (a merge commit to pull upstream changes in, a commit from myself and then the empty feature commit at last), which might have confused AppVeyor... I tried something similar again below (ie. git pull upstream master; git commit -m '[feature]' --allow-empty; git push to check if that might have been the reason but it worked again, therefore I can only conclude we must have had a flaky build for unknown reasons.
What do you think about my question about code coverage of the daily build?

@bergmeister Code coverage does not use the AppVeyor workflow so this change should not be an issue.

@bergmeister I was discussing the impact of FailFast with other team members. @JamesWTruher Mentioned that we would like to always have logs of tests results (even if it is that tests were canceled.) FailFast defeats this and we don't get logs to any canceled tests. Could you disable FailFast? We don't view this as a critical part of this optimization.

@TravisEz13 OK. Sounds good. This PR already has FailFast removed, I am thinking the same way, especially now with tests running in both build jobs.

@bergmeister sorry I messed that.

@bergmeister
Travis-CI has a similar feature https://docs.travis-ci.com/user/customizing-the-build/#Build-Matrix
We are considering refactoring Travis-CI in a similar way and enabling the feature tests by default on both systems. This would eliminate the most of the time gains you have given us but give us better coverage in the CI system. This is just an FYI in case you are interested in contributing.

@bergmeister And thanks for this work.

@TravisEz13 You're welcome, I quite enjoy working with CI systems and tweaking them. I also like it now to be able to get complete results now again in 15 minutes, which is close to how it was when PowerShell was in Alpha state.
I have not worked with Travis yet myself and having AppVeyor fast is sufficient for me because I am now more interested on Windows as the platform and rarely have PRs that break non-Windows platforms. Therefore don't expect a PR from my side at the moment.

If you think about always running feature tests, then I think it would be the best to do that in an optional 3rd build job that could be implemented in an easier fashion once AppVeyor implements conditional build matrices here: appveyor/ci#2299

It would be interesting if you guys could setup a Travis CI build for PSSA to get coverage for Mac (AppVeyor now officially supports Ubuntu images and I can tell from experience that they work very well, are very fast and the AppVeyor support is awesome, just a few days ago they fixed a bug in production for me in less than an hour or me raising it). You could therefore consider moving the Linux tests to AppVeyor.

@bergmeister

If you think about always running feature tests, then I think it would be the best to do that in an optional 3rd build job once AppVeyor implements conditional build matrices here: appveyor/ci#2299

I don't think we would wait for conditional matrices. We would just add another job with no condition that would run all the time.

AppVeyor now officially supports Ubuntu images and I can tell from experience that they work very well ...

I have discussed with other team members moving the PowerShell Core Linux tests from Travis-CI to AppVeyor. I agree that they seem faster and I prefer the AppVeyor feature set and support over Travis-CI. If we were to move to AppVeyor, we would probably run both for some time to prove that they are reliable. Currently, I have the PowerShell-Docker tests all running in AppVeyor as testing on Linux should be equivalent of testing on macOS for what we are delivering in that repo.

@TravisEz13 Yes, just running the feature tests in a 3rd job would be OK for me as well.
Just as an aside/idea, one could also consider dogfooding VSTS hosted images (that support all 3 platforms) and VSTS currently has a preview feature of offering a public read-only view. What would be great about VSTS is that one only needs to build once and can then kick off multiple release definitions with the artifacts. VSTS supports YAML already for builds and soon for release definitions as well.

@bergmeister We have considered VSTS. I have already reserved powershell.visualstudio.com. One issue I'm aware of, VSTS Hosted Linux uses an Ubuntu container not a VM and our tests currently have issues with containers. There is a PR to get this partially working #6953. I doubt the author will do more work. I'm fine as long as the build works, we tag tests somehow as not working with docker and file an issue to fix the tests (hopefully with some sort of breakout into smaller pieces.)

I plan to move PowerShell-Docker to VSTS when I get the issues worked out (the CI builds aren't running even though the webhook fires.)