DSC for PowerShell 7 #265

SteveL-MSFT · 2020-11-09T17:55:21Z

Thanks to @gaelcolas, @joeyaiello, @DCtheGeek, @mgreenegit for initial feedback

Co-authored-by: Joey Aiello <joeyaiello@users.noreply.github.com> Co-authored-by: David Coulter <David.Coulter@microsoft.com> Co-authored-by: Michael Greene <mgreenegit@users.noreply.github.com>

kgraves83 · 2020-11-10T03:30:13Z

I’m confused about the need for the JSON configuration file. If we are no longer using the LCM and just calling resources directly with Invoke-DscResource, is there still a need for a configuration file at all (including MOF or JSON)?

kvprasoon · 2020-11-10T05:19:20Z

That would be required for "Bring Your Own Agent" scenario when any one creates their agent to parse and apply the JSON based configuration,

iSazonov · 2020-11-10T06:52:11Z

I’m confused about the need for the JSON configuration file.

I also. The RFC looks like more a history story than a design document or roadmap. All key points are mentioned lightly.

vexx32 · 2020-11-10T12:47:48Z

Agreed. Also, several concerns were raised in the issue on PowerShell/PowerShell which I don't see being addressed at all, or even mentioned, in the RFC.

There's not enough detail here nor enough consideration for a complete implementation to have a good starting point.

JamesWTruher · 2020-11-10T19:00:35Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+
+### Invoke-DscResource
+
+About a year ago, we worked with


Although i appreciate the shout-out to Gael, I'm not sure this paragraph belongs here. Perhaps an acknowledgement section in the back of the RFC would be more appropriate.
I think what we should have here is a description of the behavior that we will be supplying.

The point here was to connect it back to the Invoke-DscResource work

JamesWTruher · 2020-11-10T19:02:30Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+
+### Bring your own agent (BYOA)
+
+A significant change from existing DSC for Windows PowerShell is there will be no


I'm still not clear on what I need to do here. Do we have documentation to which we can refer for enabling this.

This is confusing to me as well. If I'm understanding correctly this would be the new workflow:

Write out a configuration in PowerShell (using the Configure keyword), same as previous versions.

Export the configuration to JSON.

Use a custom script or third-party tool to parse the JSON and then run the DSC resource modules using Invoke-DscResource.

I think step 3 is where my disconnect is. Could we have a built in command that would would parse the JSON file and run the DSC resources? This would at least give us the base functionality, which we could then extend if needed.

AFAIK since there is no schema, I think it wont be an easy thig to provide a built-in cmdlet for that.

Well there doesn’t need to be a strict schema. The configuration file is basically just key value pairs. All DSC resources have get, test, and set methods you use to compare with the values in the config. That’s what the original LCM did. Granted it wouldn’t be very hard to write a script to reproduce that functionality, but I feel like by not having this built in, it will prevent a lot of non-programmers from using DSC. I mean the original point of DSC was so non-programmers could easily build up configurations. If you really strip things down to basics, we don’t even really need DSC resources. We could just write PowerShell scripts that apply the settings we want directly.

yep, I agree

A small clarification about ...

If I'm understanding correctly this would be the new workflow:
Write out a configuration in PowerShell (using the Configure keyword), same as previous versions.
Export the configuration to JSON.
Use a custom script or third-party tool to parse the JSON and then run the DSC resource modules using Invoke-DscResource.

This is one option; that is if the user decides to go through "configuration ps1" -> JSON (which used to be MOF) -> third-party tool to parse the JSON and then do whatever it needs route.

Another (faster) option is to just call Invoke-DscResource directly - see examples in this link.

I understand. My main point was that one of the original benefits of DSC was that people without a lot of programming or scripting experience could still write configurations. Having to manually call the resources with Invoke-DscResource, or build a script to replace the functionality of the LCM, just seems like a bit of a step back. I like the idea of having it open for third-parties to build on, but it feels like we are loosing some base functionality in the process. Anyway, that’s just my 2 cents.

I kinda agree with you, but we lost that already from 5.1.
This is actually "bringing back" functionality, so I see it as a good (re)start.
I hope that in due time we'll get closer to what we can do with 5.1, whether it's with a third party agent, from the community or why not using Guest Config...

Hey,

I like the discussion so far. But I must agree with @kgraves83 a lot. By removing the LCM completely, it will be hard for people to manage DSC. I know in the RFC it's talking about: integrate with existing configuration agents like Ansible, Chef, Puppet, Azure Policy Guest Configuration, and other popular configuration solutions. But I don't think that is a good solution for companies, who use the old DSC. Because you can just run Start-DSCConfiguration on any machine in a well-configured environment and the party can start.

With the new solution, you must first find the current json configurations for the machine. Then you must take a look into the configuration and install all required modules on the target machine. (normally DSC would pull them from the pullserver) And then run them on each target machine. And the tool to manage all of this is handcrafted by an admin. Oh and don't forget the monitoring feature. That must be also done manually with the new solution if I understand it right.

I mean it's a good starting point to implement first the logic of "how a configuration is parsed and can be used". But this RFC shouldn't forget the management of many configurations for a big DSC environment. The popular configuration solutions had for me two downsides:

Ansibel and Chef needs a Linux Host

Azure Policy Guest Configuration works only if the company uses azure

So in a completely Windows based environment that is disconnected, there is no solution. I really want to see something like a basic tooling for those use cases. I think the idea of something like a LCM as agent isn't a bad idea. I think DSC need something like that to work well. The downsides of not having it are to big. Maybe this is the wrong RFC for that discussion , but I think that is important.

I really wish to see something like LCM 2.0 or something like that.

Worth noting here: From a CM vendor perspective (/me waves in Puppet developer) while we could support calling a config script from within the CM tool, they already have a DSL (or yaml def in the case of Ansible) where users are defining their full configuration info - even telling DSC to handle the JSON blob of configs would require writing some DSL - we're probably going to continue to support and recommend our integration which is built on top of Invoke-DscResource.

For Puppet specifically, we've been doing a lot of work over the last year to build a strong translation layer between the API surface of DSC Resources and Puppet so you can consume DSC Resources just like any other Puppet resource - with the full suite of support sugar from our VSCode extension, compile time validation, etc.

I think that experience is going to be better for our users than a new Puppet resource which allows you to specify a JSON blob (as a herestring or path to the blob) and is only able to do change reporting based on the entire blob or (assuming property-by-property reporting gets implemented at the same time as these other changes) needs custom logic to surface DSC's internal change reporting.

I suspect but do not know that the situation for other CM vendors is similar from an integration perspective.

The more important need for us is to make sure we can fully map a DSC Resource's surface from Get-DscResource since we won't be able to map it via inspecting the CIM instances anymore (especially whatever we're calling nested CIM instances now that they won't be CIM instances).

JamesWTruher · 2020-11-10T19:05:22Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+still needed to learn MOF and also keep their MOF schema in sync with their script
+implementation.
+
+Today, when you execute a


sort of burying the lead here. I think it will be better to lead with the JSON config and use the earlier sentences as rational for the change.

JamesWTruher · 2020-11-10T19:05:25Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+Since there is no default agent to understand the JSON configuration file, it will
+be up to users or 3rd party agents to understand the configuration file.
+
+Moving from WMI/OMI also means no longer supporting C++ resources nor Python based resources.


i assume that someone could build their own python based resources if they also build an agent for them.

No, as use of DSC Resources is via the *-DscResource cmdlets, those only understand PowerShell classes, so unless they transcompile to a PowerShell class, then other languages are not currently planned to be supported

JamesWTruher · 2020-11-10T19:05:48Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+dependency on WMI/OMI. Instead, we will move towards a JSON configuration file format.
+
+Since there is no default agent to understand the JSON configuration file, it will
+be up to users or 3rd party agents to understand the configuration file.


how? Do we have a pointer for folks to follow to construct this?

JamesWTruher · 2020-11-10T19:08:44Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+- The comment at the start is dropped as that information is available within `ConfigurationDocument`
+
+### No longer require MOF schema by switching to improved class-based resources
+


this is a pretty strong argument to drop the MOF equivalent above.

JamesWTruher · 2020-11-10T19:09:37Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+on a .psm1 module file containing traditionally authored Get/Set/Test-TargetResource
+functions. This would also aid in migrating the more than 1000 existing resources
+for Windows.
+


totally agree - is this sort of an open switch or do you have ideas/suggestions for this?

JamesWTruher · 2020-11-10T19:09:57Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+The current
+[PSDesiredStateConfiguration](https://docs.microsoft.com/powershell/module/psdesiredstateconfiguration/)
+module is not Open Source. The intent is to clean up the code and prepare the GitHub
+repository to be made public as an Open Source project.


JamesWTruher · 2020-11-10T19:11:02Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+To enable DSC for PowerShell 7 to evolve independently to PowerShell 7 itself, the
+code in the engine that handles the DSC configuration keyword will be separated
+as a plugin and made to be part of the PSDesiredStateConfiguration module.
+


although not really needed here - is there an RFC for enabling custom keywords?

JamesWTruher · 2020-11-10T19:12:36Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+[documentation and examples](https://dsccommunity.org/blog/class-based-dsc-resources/).
+
+### Tooling for parsing JSON configuration
+


as an aside, we need to be sure that a schema for the JSON is published or otherwise available for a good authoring experience.

I see an opportunity for a tool or VSCode plugin to generate a schema on the fly from available resources. Based on the example json earlier in the RFC, the schema will end up very generic and not able to offer more than basic structural validation. An auto generated schema can offer a much stronger validation and auto complete for resource properties.

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

daxian-dbw · 2020-11-10T20:18:12Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+Since there is no default agent to understand the JSON configuration file, it will
+be up to users or 3rd party agents to understand the configuration file.
+
+Moving from WMI/OMI also means no longer supporting C++ resources nor Python based resources.


I remember the File/Directory resource is written in C++ (WMI provider), so it's no longer supported with the new DSC, right? If so, the example below may need to change as it's still using File resource.

That is correct, the File resource was just used as an example of configuration file format

daxian-dbw · 2020-11-10T20:20:01Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+Important differences:
+
+- CIM class name prefixes are removed as they are not needed
+- Instances are represented as an array of the type name


It's not very clear to me what this mean as I cannot map this statement to the above demo.

SteveL-MSFT · 2020-11-10T20:25:41Z

@vexx32 can you be specific on what isn't covered? Looking at the comments in that issue, I believe they are answered with the exception of Pester testing which is not in scope for this specific RFC.

SteveL-MSFT · 2020-11-10T20:38:29Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+    "Author": "slee",
+    "GenerationHost": "SLEE-DESKTOP"
+  },
+  "LogResource": [


Fix ordering of instances to match mof example

SteveL-MSFT · 2020-11-10T20:41:05Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+
+To enable DSC for PowerShell 7 to evolve independently to PowerShell 7 itself, the
+code in the engine that handles the DSC configuration keyword will be separated
+as a plugin and made to be part of the PSDesiredStateConfiguration module.


Link to subsystem RFC once published

SteveL-MSFT · 2020-11-10T20:42:41Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+Unlike DSC for Windows PowerShell, DSC for PowerShell 7 is only intended to provide
+a platform for building or integrating with configuration solutions. For PowerShell 7,
+the scope of DSC support is limited to the `configuration` keyword, authoring,
+compilation, and using DSC resources via the PSDesiredStateConfiguration module.


Make it explicit here it's only PowerShell class based DSC resources

SteveL-MSFT · 2020-11-10T20:43:10Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+Unlike DSC for Windows PowerShell, DSC for PowerShell 7 is only intended to provide
+a platform for building or integrating with configuration solutions. For PowerShell 7,
+the scope of DSC support is limited to the `configuration` keyword, authoring,
+compilation, and using DSC resources via the PSDesiredStateConfiguration module.


Make explicit that compilation results in JSON instead of MOF

SteveL-MSFT · 2020-11-10T20:54:18Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+      "SourceInfo": "C:\\\\Users\\\\slee\\\\test\\\\file.dsc.ps1::6::9::File",
+      "ModuleName": "PSDesiredStateConfiguration",
+      "Type": "Directory",
+      "SourcePath": "\\\\\\\\PullServer\\\\DemoSource"


Rename PullServer to some other computer name to avoid confusion with pull server

ThomasNieto · 2020-11-11T01:32:24Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+C/C++ code and compiling for specific Linux distros and processor architectures.
+Writing PowerShell script DSC resources currently still requires a matching schema
+file making it more complex. Instead, the current plan is to
+[only support PowerShell class-based DSC resources](https://github.com/PowerShell/PowerShell/issues/13731)


Why is there no C# class-based DSC resource?

It's certainly something that can be supported later if there is a need for it. Initially, we're targeting making it simpler and PowerShell class-based DSC resource development model already exists while we'd have to create one for C#.

Please correct me if I'm wrong but there really shouldn't be much additional work required supporting PowerShell and C# class-based resources. If the PowerShell class is being imported as a dynamic assembly, DSC can use the same logic to identify C# and PowerShell DSC resources.

You are right, theoretically there really shouldn't be much additional work to support C# resources, however things get complex when we get into details because the existing code is rather difficult for historical reasons.
For example, for resource enumeration (Get-DscResource) current code does Not actually import PowerShell class as a dynamic assembly. It sends psm1 with class to PS parser, retrieves AST and then traverses it looking for DSC attributes (e.g. DscResourceAttribute) and constructing internal resource metadata descriptions that are later (and Not in a straightforward way) returned as Get-DscResource results. The class module is actually imported (in the usual PS sense) only at Invoke-DscResource stage.

If it is difficult to add enhancements to the existing code base, then maybe its time to look at redesigning/refactoring while these major DSC changes are being proposed. As for Get-DscResource not importing the C# assembly until Invoke-DscResource is called, I believe that could be solved with a MetadataLoadContext as described in PowerShell/PowerShell#6653.

That is good info about MetadataLoadContext ; thank you.

gaelcolas · 2020-11-12T15:20:50Z

@vexx32 said:

There's not enough detail here nor enough consideration for a complete implementation to have a good starting point.

Actually, because we're starting for PS7, it's a good starting point IMO.

It does not discuss a Configuration Management solution, but gives the foundation (direction) of a framework that needs to be complemented by other more specific RFCs and/or prototypes (such as the compiled JSON document, DSC Class resources improvements and so on...).

This RFC at least states the ground work needed to make DSC for PS7 a solid foundation other tools and solutions can leverage, while also making decisions of what not to spend time on, at least initially.

@kgraves83 said:

is there still a need for a configuration file at all (including MOF or JSON)
Yes. DSC is also a DSL, and that is useful to build a DAG (directed Acyclic Graph) of resources to apply. That's also when we use composites.
Also, for now, Guest Config uses confg MOFs on top of DSC Resources: https://docs.microsoft.com/en-us/azure/governance/policy/how-to/guest-configuration-create#step-by-step-creating-a-custom-guest-configuration-audit-policy-for-windows

Even if we don't have an Agent to handle such configuration, I think it's important not to lose sight of the DSL and what the compilation offers.
I'm not interested in the output per se, whether JSON or MOF, as I'd prefer to have access to an in-memory version, but I assume that'd be extra work, and maybe not priority.

KevinMarquette · 2020-11-17T17:50:39Z

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

+      "Ensure": "Present",
+      "Recurse": true,
+      "ModuleVersion": "1.0",
+      "DestinationPath": "C:\\\\Users\\\\Public\\\\Documents\\\\DSCDemo\\\\DemoDestination",


I don't think the backslashes need to be double escaped here. once for JSON encoding, but that is all it should need

KevinMarquette · 2020-11-17T18:37:45Z

We need to discuss an example with complex objects and how they will be represented in the JSON. Configuring SSL with XWebSite is the common scenario that jumps out to me.

xWebSite NewWebsite
        {
            Ensure          = "Present"
            Name            = $WebSiteName
            State           = "Started"
            PhysicalPath    = $DestinationPath
            DependsOn       = "[File]WebContent"
            BindingInfo     = MSFT_xWebBindingInformation
            {
                Protocol              = 'https'
                Port                  = '443'
                CertificateStoreName  = 'MY'
                CertificateThumbprint = 'BB84DE3EC423DDDE90C08AB3C5A828692089493C'
                HostName              = $WebSiteName
                IPAddress             = '*'
                SSLFlags              = '1'
            }
        }

Co-authored-by: Dongbo Wang <dongbow@microsoft.com>

JamesWTruher · 2022-05-04T22:25:58Z

@PowerShell/powershell-committee
this should be moved to https://github.com/powershell/psdesiredstateconfiguration
@theJasonHelmick

initial draft (#7)

2bd4550

Co-authored-by: Joey Aiello <joeyaiello@users.noreply.github.com> Co-authored-by: David Coulter <David.Coulter@microsoft.com> Co-authored-by: Michael Greene <mgreenegit@users.noreply.github.com>

JamesWTruher reviewed Nov 10, 2020

View reviewed changes

daxian-dbw reviewed Nov 10, 2020

View reviewed changes

2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md Outdated Show resolved Hide resolved

daxian-dbw reviewed Nov 10, 2020

View reviewed changes

SteveL-MSFT commented Nov 10, 2020

View reviewed changes

ThomasNieto reviewed Nov 11, 2020

View reviewed changes

KevinMarquette reviewed Nov 17, 2020

View reviewed changes

anmenaga mentioned this pull request Jan 11, 2021

Cross-platform updates to DSC code in SMA DLL PowerShell/PowerShell#13399

Merged

13 tasks

Update 2-Draft-Accepted/RFCxxxx-DSC-for-PowerShell7.md

44c3123

Co-authored-by: Dongbo Wang <dongbow@microsoft.com>

JamesWTruher closed this May 4, 2022


		### Bring your own agent (BYOA)

		A significant change from existing DSC for Windows PowerShell is there will be no

		- The comment at the start is dropped as that information is available within `ConfigurationDocument`

		### No longer require MOF schema by switching to improved class-based resources

		[documentation and examples](https://dsccommunity.org/blog/class-based-dsc-resources/).

		### Tooling for parsing JSON configuration

DSC for PowerShell 7 #265

DSC for PowerShell 7 #265

Uh oh!

Conversation

SteveL-MSFT commented Nov 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kgraves83 commented Nov 10, 2020

Uh oh!

kvprasoon commented Nov 10, 2020

Uh oh!

iSazonov commented Nov 10, 2020

Uh oh!

vexx32 commented Nov 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kvprasoon Nov 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SteveL-MSFT commented Nov 10, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SteveL-MSFT commented Nov 9, 2020 •

edited

Loading

kvprasoon Nov 11, 2020 •

edited

Loading

anmenaga Nov 11, 2020 •

edited

Loading