Skip to content

Telemetry improvements for tracking experimental feature optout #18762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
daxian-dbw merged 10 commits into from
Mar 8, 2023
Merged

Telemetry improvements for tracking experimental feature optout #18762

daxian-dbw merged 10 commits into from
Mar 8, 2023

Conversation

@JamesWTruher
Copy link
Collaborator

@JamesWTruher JamesWTruher commented Dec 9, 2022 •
edited
Loading

PR Summary

Add new telemetry types to track experimental feature optout better.
Add a GetExperimentalFeatureUseData to provide a consistent way to track feature and details. Add telemetry to PSNativeCommandUseErrorActionPreference so we can track the actual values in use.
This also provides better tracking for experimental features and now tracks 2 states:

  • Enabled
  • Disabled

Also, we will now collect RuntimeInformation.RuntimeIdentifier which allows better clarity on our Docker use since OSDescription is information about the Host OS rather than the guest.

PR Context

We don't currently have an easy way of determining which experimental features are not enabled.
We also don't have any way of determining whether a feature is actually being used. This PR
provides actual setting of PSNativeCommandUseErrorActionPreference (true/false/unset) to help us determine how much this feature is being used.

PR Checklist

@ghost ghost assigned daxian-dbw Dec 9, 2022
@ghost ghost added the Review - Needed The PR is being reviewed label Dec 17, 2022
@ghost
Copy link

ghost commented Dec 17, 2022

This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days.
Maintainer, please provide feedback and/or mark it as Waiting on Author

@JamesWTruher JamesWTruher force-pushed the TelemetryForExperimentalFeature branch from 7d6d178 to 4e9d6eb Compare January 4, 2023 19:43
@JamesWTruher JamesWTruher force-pushed the TelemetryForExperimentalFeature branch 3 times, most recently from c228739 to 85e0c62 Compare January 25, 2023 21:06
@JamesWTruher JamesWTruher force-pushed the TelemetryForExperimentalFeature branch from b604093 to ff61724 Compare February 8, 2023 19:22
@JamesWTruher JamesWTruher changed the title WIP: Telemetry improvements for tracking experimental feature optout Telemetry improvements for tracking experimental feature optout Feb 9, 2023
daxian-dbw
daxian-dbw reviewed Feb 28, 2023
View reviewed changes
JamesWTruher and others added 10 commits February 28, 2023 16:11
@JamesWTruher
Add new telemetry types to track experimental feature optout better.
d7cd5f8 
Add a GetExperimentalFeatureUseData to provide a consistent way to track feature and details.
Add telemetry to PSNativeCommandUseErrorActionPreference so we can track the actual values in use.
@JamesWTruher
Fix up CodeFactor issues.
780487d
@JamesWTruher
Round 2 CodeFactor issues.
d185606
@JamesWTruher
Turn off developer mode for telemetry sends.
ec96808
@JamesWTruher
Add runtime identifier to startup telemetry.
a2ff9cf
@JamesWTruher
Add internal method to report on experimental feature usage.
304cdcc 
We will now report on ExperimentalFeature enabling _and_ disabling.
Add usage data for experimental feature PSNativeCommandUseErrorActionPreference.
Report the number of invalid experimental features were requested to be enabled.
@JamesWTruher
Remove telemetry for invalid experimental features.
d9b310d
@JamesWTruher
Fix up CodeFactor issues.
1793d37
@JamesWTruher @daxian-dbw
Update src/System.Management.Automation/engine/ExperimentalFeature/Ex…
9f98820 
…perimentalFeature.cs

Co-authored-by: Dongbo Wang <dongbow@microsoft.com>
@JamesWTruher
Remove extraneous counter no longer used.
7290f83 
Improve logic in collecting disabled features.
@JamesWTruher JamesWTruher force-pushed the TelemetryForExperimentalFeature branch from 6a830d9 to 7290f83 Compare March 1, 2023 00:12
@pull-request-quantifier-deprecated
Copy link

pull-request-quantifier-deprecated bot commented Mar 1, 2023

This PR has 36 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

Quantification details 

Label      : Extra Small
Size       : +32 -4
Percentile : 14.4%

Total files changed: 3

Change summary by file extension:
.cs : +32 -4

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

  • Fast and predictable releases to production:
    • Optimal size changes are more likely to be reviewed faster with fewer
      iterations.
    • Similarity in low PR complexity drives similar review times.
  • Review quality is likely higher as complexity is lower:
    • Bugs are more likely to be detected.
    • Code inconsistencies are more likely to be detected.
  • Knowledge sharing is improved within the participants:
    • Small portions can be assimilated better.
  • Better engineering practices are exercised:
    • Solving big problems by dividing them in well contained, smaller problems.
    • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

  • Use the PullRequestQuantifier to quantify your PR accurately
    • Create a context profile for your repo using the context generator
    • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
    • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
    • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
  • Change your engineering behaviors
    • For PRs that fall outside of the desired spectrum, review the details and check if:
      • Your PR could be split in smaller, self-contained PRs instead
      • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

  • One line was added: +1 -0
  • One line was deleted: +0 -1
  • One line was modified: +1 -1 (git diff doesn't know about modified, it will
    interpret that line like one addition plus one deletion)
  • Change percentiles: Change characteristics (addition, deletion, modification)
    of this PR in relation to all other PRs within the repository.

Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

1 similar comment
@pull-request-quantifier-deprecated
Copy link

pull-request-quantifier-deprecated bot commented Mar 1, 2023

This PR has 36 quantified lines of changes. In general, a change size of upto 200 lines is ideal for the best PR experience!

Quantification details 

Label      : Extra Small
Size       : +32 -4
Percentile : 14.4%

Total files changed: 3

Change summary by file extension:
.cs : +32 -4

Change counts above are quantified counts, based on the PullRequestQuantifier customizations.

Why proper sizing of changes matters

Optimal pull request sizes drive a better predictable PR flow as they strike a
balance between between PR complexity and PR review overhead. PRs within the
optimal size (typical small, or medium sized PRs) mean:

  • Fast and predictable releases to production:
    • Optimal size changes are more likely to be reviewed faster with fewer
      iterations.
    • Similarity in low PR complexity drives similar review times.
  • Review quality is likely higher as complexity is lower:
    • Bugs are more likely to be detected.
    • Code inconsistencies are more likely to be detected.
  • Knowledge sharing is improved within the participants:
    • Small portions can be assimilated better.
  • Better engineering practices are exercised:
    • Solving big problems by dividing them in well contained, smaller problems.
    • Exercising separation of concerns within the code changes.

What can I do to optimize my changes

  • Use the PullRequestQuantifier to quantify your PR accurately
    • Create a context profile for your repo using the context generator
    • Exclude files that are not necessary to be reviewed or do not increase the review complexity. Example: Autogenerated code, docs, project IDE setting files, binaries, etc. Check out the Excluded section from your prquantifier.yaml context profile.
    • Understand your typical change complexity, drive towards the desired complexity by adjusting the label mapping in your prquantifier.yaml context profile.
    • Only use the labels that matter to you, see context specification to customize your prquantifier.yaml context profile.
  • Change your engineering behaviors
    • For PRs that fall outside of the desired spectrum, review the details and check if:
      • Your PR could be split in smaller, self-contained PRs instead
      • Your PR only solves one particular issue. (For example, don't refactor and code new features in the same PR).

How to interpret the change counts in git diff output

  • One line was added: +1 -0
  • One line was deleted: +0 -1
  • One line was modified: +1 -1 (git diff doesn't know about modified, it will
    interpret that line like one addition plus one deletion)
  • Change percentiles: Change characteristics (addition, deletion, modification)
    of this PR in relation to all other PRs within the repository.

Was this comment helpful? 👍  :ok_hand:  :thumbsdown: (Email)
Customize PullRequestQuantifier for this repository.

SteveL-MSFT
SteveL-MSFT approved these changes Mar 8, 2023
View reviewed changes
Copy link
Member

@SteveL-MSFT SteveL-MSFT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@daxian-dbw daxian-dbw merged commit 14d7bfe into PowerShell:master Mar 8, 2023
@ghost ghost removed the Review - Needed The PR is being reviewed label Mar 8, 2023
@daxian-dbw daxian-dbw added Review - Needed The PR is being reviewed CL-Engine Indicates that a PR should be marked as an engine change in the Change Log CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log and removed CL-Engine Indicates that a PR should be marked as an engine change in the Change Log Review - Needed The PR is being reviewed labels Mar 8, 2023
@ghost
Copy link

ghost commented Mar 14, 2023

🎉v7.4.0-preview.2 has been released which incorporates this pull request.:tada:

Handy links:

@StevenBucher98 StevenBucher98 mentioned this pull request Mar 15, 2023
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@daxian-dbw daxian-dbw daxian-dbw approved these changes

@SteveL-MSFT SteveL-MSFT SteveL-MSFT approved these changes

Assignees

@daxian-dbw daxian-dbw

Labels

CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log Extra Small

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JamesWTruher @daxian-dbw @SteveL-MSFT