Test-Json has false positives when using anyof and allof statements in JSON schema

Prerequisites

Write a descriptive title.
Make sure you are able to repro it on the latest released version
Search the existing issues.
Refer to the FAQ.
Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

When i am using anyOf or allOf statements in my JSON schema I got false positives when the output is "False"
The issue appears on 7.4. it worked fine on 7.2

Schema

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Landingzone",
    "description": "Landingzone",
    "type": "object",
    "required": [
        "virtualNetwork"
    ],
    "additionalProperties": false,
    "properties": {
        "virtualNetwork": {
            "type": "array",
            "markdownDescription": "Wiki pagina: <br> [shit](https://dev.azure.com/CloudCompetenceCenter/Cloud%20CC/_wiki/wikis/Cloud-CC.wiki/2067/OTAP-JSON-Manual?anchor=virtual-networks) sdf ethe ",
            "description": "https://dev.azure.com/CloudCompetenceCenter/Cloud%20CC/_wiki/wikis/Cloud-CC.wiki/2067/OTAP-JSON-Manual?anchor=virtual-networks",
            "items": {
                "type": "object",
                "additionalProperties": false,
                "properties": {
                    "securityRules": {
                      "type": "array",
                        "items": {
                          "$ref": "#/$defs/securityRulesDetail"
                        }
                    }
                }
            }
        }
    },
    "$defs": {
        "securityRulesDetail": {
            "type": "object",
            "required": [
                "name",
                "port"
            ],
            "additionalProperties": false,
            "properties": {
                "name": {
                    "type": "string",
                    "minLength": 1,
                    "$comment": "No spaces or special characters",
                    "pattern": "^([a-zA-Z0-9-\\._])+$"
                },
                "port": {
                    "type": "array",
                    "items": {
                        "oneOf": [
                            {
                                "type": "integer",
                                "minimum": 0,
                                "maximum": 66535
                            },
                            {
                                "type": "string",
                                "$comment": "range between 0 and 66535 or is *",
                                "pattern": "^(0|[1-9][0-9]{0,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5]|\\*)(-([1-9][0-9]{0,3}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5]))?$"
                            }
                        ]
                    }
                }
            }
        }
    }
}

json

{
  "virtualNetwork": [
    {
      "securityRules": [
        {
          "name": "TestRule01/",
          "port": [
            443
          ]
        },
        {
          "name": "TestRule02",
          "port": [
            443
          ]
        },
        {
          "name": "TestRule03",
          "port": [
            443
          ]
        },
        {
          "name": "TestRule04",
          "port": [
            444
          ]
        },
        {
          "name": "TestRule05",
          "port": [
            443
          ]
        },
        {
          "name": "TestRule06",
          "port": [
            443
          ]
        },
        {
          "name": "Testrule07",
          "port": [
            8080
          ]
        },
        {
          "name": "Testrule08",
          "port": []
        },
        {
          "name": "Testrule09",
          "port": [
            5432
          ]
        }
      ]
    }
  ]
}

Expected behavior

`Test-Json -Schema $schema -Json $data`

Test-Json: The JSON is not valid with the schema: The string value is not a match for the indicated regular expression at '/virtualNetwork/0/securityRules/0/name'

Actual behavior

`Test-Json -Schema $schema -Json $data`

Test-Json: The JSON is not valid with the schema: The string value is not a match for the indicated regular expression at '/virtualNetwork/0/securityRules/0/name'
Test-Json: The JSON is not valid with the schema: Value is "integer" but should be "string" at '/virtualNetwork/0/securityRules/0/port/0'
Test-Json: The JSON is not valid with the schema: Value is "integer" but should be "string" at '/virtualNetwork/0/securityRules/1/port/0'
Test-Json: The JSON is not valid with the schema: Value is "integer" but should be "string" at '/virtualNetwork/0/securityRules/2/port/0'
Test-Json: The JSON is not valid with the schema: Value is "integer" but should be "string" at '/virtualNetwork/0/securityRules/3/port/0'
Test-Json: The JSON is not valid with the schema: Value is "integer" but should be "string" at '/virtualNetwork/0/securityRules/4/port/0'
Test-Json: The JSON is not valid with the schema: Value is "integer" but should be "string" at '/virtualNetwork/0/securityRules/5/port/0'
Test-Json: The JSON is not valid with the schema: Value is "integer" but should be "string" at '/virtualNetwork/0/securityRules/6/port/0'
Test-Json: The JSON is not valid with the schema: Value is "integer" but should be "string" at '/virtualNetwork/0/securityRules/8/port/0'

Error details

Exception             : 
    Type    : System.Exception
    Message : The JSON is not valid with the schema: Value is "integer" but should be "string" at '/virtualNetwork/0/securityRules/8/port/0'
    HResult : -2146233088
CategoryInfo          : InvalidData: (:) [Test-Json], Exception
FullyQualifiedErrorId : InvalidJsonAgainstSchemaDetailed,Microsoft.PowerShell.Commands.TestJsonCommand
InvocationInfo        : 
    MyCommand        : Test-Json
    ScriptLineNumber : 1
    OffsetInLine     : 1
    HistoryId        : 147
    Line             : Test-Json -Schema $schema -Json $data
    Statement        : Test-Json -Schema $schema -Json $data
    PositionMessage  : At line:1 char:1
                       + Test-Json -Schema $schema -Json $data
                       + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    InvocationName   : Test-Json
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo :

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.0
PSEdition                      Core
GitCommitId                    7.4.0
OS                             Darwin 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:25 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6030
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

Dylan Prins (@Dylan-Prins)

Dylan Prins (@Dylan-Prins) the problem is with how Powershell is displaying the output. (I argued with them about this.)

Please have a read through this post to understand better how to interpret output. You can also take your schema and instance to https://json-everything.net/json-schema to get different output formats that may make more sense to you.

json-everything.net
Extended JSON support in .Net built on top of the System.Text.Json namespace

In the output, you'll see several occurrences of something like

{
  "valid": true,
  "evaluationPath": "/properties/virtualNetwork/items/properties/securityRules/items/$ref/properties/port/items",
  "schemaLocation": "https://json-everything.net/e3e22c00ac#/$defs/securityRulesDetail/properties/port/items",
  "instanceLocation": "/virtualNetwork/0/securityRules/6/port/0",
  "details": [
    {
      "valid": true,
      "evaluationPath": "/properties/virtualNetwork/items/properties/securityRules/items/$ref/properties/port/items/oneOf/0",
      "schemaLocation": "https://json-everything.net/e3e22c00ac#/$defs/securityRulesDetail/properties/port/items/oneOf/0",
      "instanceLocation": "/virtualNetwork/0/securityRules/6/port/0"
    },
    {
      "valid": false,
      "evaluationPath": "/properties/virtualNetwork/items/properties/securityRules/items/$ref/properties/port/items/oneOf/1",
      "schemaLocation": "https://json-everything.net/e3e22c00ac#/$defs/securityRulesDetail/properties/port/items/oneOf/1",
      "instanceLocation": "/virtualNetwork/0/securityRules/6/port/0",
      "errors": {
        "type": "Value is \"number\" but should be \"string\""
      }
    }
  ]
}

The top level here is the subschema that contains the oneOf, and the details are the subschemas in the oneOf. You can see that one passes and one fails, making the subschema with the oneOf pass.

The problem with the Powershell output is that they wanted it to just output all of the errors, even if they weren't relevant, which these errors aren't.

I am also running into this issue and it's making the output from my module almost unusable when there is a schema problem.

I ran into this too, and I made a test app to figure out how to fix it. The trick is to ask for OutputFormat.Hierarchical errors, and then don't walk down the children if the parent's IsValid is true.

                    EvaluationResults evaluationResults = _jschema.Evaluate(parsedJson, new EvaluationOptions { OutputFormat = OutputFormat.Hierarchical });
                    result = evaluationResults.IsValid;
                    if (!result)
                    {
                        HandleValidationErrors(evaluationResults);
                    }

        private void HandleValidationErrors(EvaluationResults evaluationResult)
        {
            if (evaluationResults.IsValid)
            {
                return;
            }
            
            if (evaluationResult.HasErrors)
            {
              foreach (var error in evaluationResult.Errors!)
              {
                  Exception exception = new(string.Format(TestJsonCmdletStrings.InvalidJsonAgainstSchemaDetailed, error.Value, evaluationResult.InstanceLocation));
                  ErrorRecord errorRecord = new(exception, "InvalidJsonAgainstSchemaDetailed", ErrorCategory.InvalidData, null);
                  WriteError(errorRecord);
              }
            }
            
            foreach (var nestedResult in evaluationResults.Details)
            {
                HandleValidationErrors(nestedResult);
            }
        }

Steve Otteson (@sotteson1)

Steve Otteson (@sotteson1) I brought this up when I was making this change, and the team opted to keep the existing API over making it actually useable.

Greg Dennis (@gregsdennis)

Greg Dennis (@gregsdennis) I was going to make a PR to fix this. Do they not want it fixed?

That was the directive then. Maybe you'll have better luck.

Fix Test-Json false positive errors when using oneOf or anyOf in sche…

Navigation Menu

Repository navigation

Test-Json has false positives when using anyof and allof statements in JSON schema #21471

Prerequisites

Steps to reproduce

Schema

json

Expected behavior

Actual behavior

Error details

Environment data

Visuals

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Test-Json has false positives when using anyof and allof statements in JSON schema #21471

Description

Prerequisites

Steps to reproduce

Schema

json

Expected behavior

Actual behavior

Error details

Environment data

Visuals

Activity

gregsdennis commented on Apr 14, 2024

gregsdennis commented on Apr 14, 2024

JohannesMoersch commented on Jun 12, 2024

sotteson1 commented on Nov 12, 2024

gregsdennis commented on Nov 12, 2024

sotteson1 commented on Nov 12, 2024

gregsdennis commented on Nov 12, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions